Point cloud registration method and apparatus thereof

Abstract

Disclosed are a point cloud registration method and an apparatus thereof. The point cloud registration method comprises the steps of: (a) sampling an input point cloud, then generating each of patches for sampled interest points, then calculating a local coordinate system using distribution information about each of the patches, and then aligning each of the patches; (b) extracting a rotation-invariant feature value after generating a circular kernel by arranging a plurality of kernel points forming a cylindrical shape for each point of each of the aligned patches; (c) generating rotation-invariant descriptors by applying the extracted rotation-invariant feature value to an artificial intelligence model; and (d) performing registration by comparing the similarities between the generated rotation-invariant descriptors and selecting a descriptor having a low similarity as a core descriptor.

Description

Point cloud alignment method and device thereof

[0001] The present invention relates to a point cloud alignment method and device therefor.

[0002]

[0003] 3D sensors cannot capture a complete, real-world image of the surrounding environment in one shot due to limitations in the field of view or obstructions. Therefore, to obtain a single, complete set of surrounding terrain data, point cloud data captured in multiple segments must be registered.

[0004] Alignment is the process of calculating a transformation matrix to align the source point cloud to the target point cloud when there are two point clouds that are not aligned because they were taken from different angles or positions, and applying the transformation matrix to the source point cloud to align it to the target point cloud.

[0005] To calculate the transformation matrix, a descriptor containing surrounding terrain information is created for each point in the point cloud, and the descriptors between the two point clouds to be aligned are compared and analyzed to find corresponding points with similar descriptor values, and then the transformation matrix is ​​calculated based on the corresponding points.

[0006] Therefore, the descriptor values ​​of points that correspond to each other must be similar, and the descriptor values ​​of points that do not correspond to each other must be different in order to find the correct corresponding points and calculate the correct transformation matrix.

[0007] Recent advances in deep learning have led to research on generating point cloud descriptors using artificial intelligence models. However, when utilizing these models, 1) point clouds or preprocessed point clouds are used as inputs for the AI ​​models to generate descriptors. However, if point clouds with position and rotation changes are used, the output descriptor values ​​may vary. 2) Descriptors generated from repetitive or monotonous features within a point cloud (e.g., walls or floors within a house) are likely to be similar or featureless, hindering the search for correspondences between point clouds.

[0008]

[0009] An object of the present invention is to provide a point cloud alignment method and device therefor.

[0010] Another object of the present invention is to provide a method and device for matching point clouds, in which descriptor values ​​do not change significantly and are constant even when the point cloud is rotated, by generating rotation-invariant descriptors using local coordinate system-based alignment and rotation-invariant feature values.

[0011] Another object of the present invention is to provide a point cloud matching method and device that can compensate for the instability of a local coordinate system and improve the expressiveness of rotationally invariant feature values ​​by using a cylindrical kernel.

[0012] Another object of the present invention is to provide a point cloud matching method and device capable of improving the accuracy of matching by excluding featureless, monotonous points during point cloud matching.

[0013]

[0014] According to one aspect of the present invention, a method for matching point clouds is provided.

[0015] According to one embodiment of the present invention, a point cloud matching method can be provided, including the steps of: (a) sampling an input point cloud, generating patches for each sampled point of interest, calculating a local coordinate system using distribution information of each patch, and then aligning each patch; (b) arranging a plurality of kernel points forming a cylindrical shape for each point of each aligned patch to generate a circular kernel, and then extracting rotationally invariant features; (c) applying the extracted rotationally invariant features to an artificial intelligence model to generate rotationally invariant descriptors; and (d) comparing similarities between the generated rotationally invariant descriptors, selecting descriptors with low similarity as core descriptors, and performing matching.

[0016] The step (b) above may further include a step of calculating a representative neighboring point by calculating a weighted-average position using the calculated distance as a weight after calculating the distance to neighboring points within the patch for each of the plurality of kernel points.

[0017] The step (b) above can extract, as rotation-invariant feature values, the angle between the normal vector of the center point of the kernel and the vector from the center point to the representative neighboring point, the relative distance from the center point to the representative neighboring point, the relative distance from the kernel point to the representative neighboring point, and the relative distance from another kernel point adjacent to the kernel point to the representative neighboring point.

[0018] The step (d) above may include: a step of calculating a cosine similarity between the generated rotation-invariant descriptor and another rotation-invariant descriptor to generate a similarity matrix; a step of calculating an L2 norm for each row of the similarity matrix to calculate a similarity strength; a step of selecting a lower T% having a low similarity strength as a core descriptor; and a step of performing alignment with another point cloud using the selected core descriptor.

[0019]

[0020] According to another aspect of the present invention, a point cloud alignment device is provided.

[0021] According to one embodiment of the present invention, a point cloud matching device may be provided, including: a patch matching unit that samples an input point cloud, generates patches for each sampled point of interest, calculates a local coordinate system using distribution information of each patch, and then aligns each patch; a circular kernel generating unit that generates a circular kernel by arranging a plurality of kernel points forming a cylindrical shape for each point of each aligned patch; a feature extracting unit that extracts rotation-invariant features for each patch using each kernel point of the circular kernel; an encoding unit that applies the extracted rotation-invariant features to an artificial intelligence model to generate rotation-invariant descriptors; and a matching unit that compares similarities between the generated rotation-invariant descriptors, selects descriptors with low similarity as core descriptors, and performs matching.

[0022] The above circular kernel generation unit can further calculate representative neighboring points by calculating distances to neighboring points within the patch for each of the plurality of kernel points and then calculating a weighted-average position using the calculated distances as weights.

[0023] The above feature extraction unit can extract rotationally invariant feature values ​​representing the relationship between each point of each patch and a representative neighboring point using each kernel point of the generated circular kernel.

[0024] The rotation invariant feature values ​​include an angle between a normal vector of the center point of the kernel and a vector from the center point to the representative neighboring point, a relative distance from the center point to the representative neighboring point, a relative distance from the kernel point to the representative neighboring point, and a relative distance from another kernel point adjacent to the kernel point to the representative neighboring point.

[0025] The above matching unit calculates the cosine similarity between the generated rotation-invariant descriptor and another rotation-invariant descriptor to generate a similarity matrix, calculates the L2 norm for each row of the similarity matrix to calculate the similarity strength, selects the lower T% with the low similarity strength as a core descriptor, and then performs matching with another point cloud using the selected core descriptor.

[0026]

[0027] By providing a point cloud alignment method and device according to one embodiment of the present invention, a rotation-invariant descriptor can be generated using local coordinate system-based alignment and rotation-invariant feature values, thereby generating a descriptor that is constant even with respect to rotation of a point cloud.

[0028] In addition, the present invention can compensate for the instability of the local coordinate system and improve the expressive power of rotationally invariant features by using a cylindrical kernel.

[0029] In addition, the present invention can improve the accuracy of matching by excluding monotonous points without features when matching point clouds.

[0030]

[0031] Figure 1 is a flowchart illustrating a point cloud matching method according to one embodiment of the present invention.

[0032] FIG. 2 is a diagram illustrating a process for generating a rotation-invariant descriptor for a point cloud according to one embodiment of the present invention.

[0033] FIG. 3 is a diagram illustrating the configuration of an artificial intelligence model according to one embodiment of the present invention.

[0034] FIG. 4 is a diagram illustrating a matching process according to one embodiment of the present invention.

[0035] Figures 5 and 6 are drawings showing alignment results according to one embodiment of the present invention.

[0036] FIG. 7 is a diagram illustrating a configuration of a point cloud alignment device according to one embodiment of the present invention.

[0037]

[0038] As used herein, singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "consist of" or "include" should not be construed to necessarily include all components or steps described in the specification, and should be construed to mean that some of the components or steps may not be included, or that additional components or steps may be included. In addition, terms such as "part" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software, or by a combination of hardware and software.

[0039] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0040]

[0041] FIG. 1 is a flowchart illustrating a point cloud matching method according to one embodiment of the present invention, FIG. 2 is a diagram illustrating a process of generating a rotation-invariant descriptor for a point cloud according to one embodiment of the present invention, FIG. 3 is a diagram illustrating a configuration of an artificial intelligence model according to one embodiment of the present invention, FIG. 4 is a diagram illustrating a matching process according to one embodiment of the present invention, and FIGS. 5 and 6 are diagrams illustrating matching results according to one embodiment of the present invention.

[0042] In step 110, the point cloud alignment device (100) receives a point cloud as input.

[0043] In step 115, the point cloud matching device (100) samples the input point cloud.

[0044] In step 120, the point cloud alignment device (100) calculates a local coordinate system from adjacent points (patches) of each sampled point of interest, and aligns the adjacent points (patches) using the calculated local coordinate system.

[0045] Let me explain this in more detail.

[0046] point cloud Given this, the point cloud matching device (100) selects interest points for matching. The interest points may be selected randomly and uniformly. In one embodiment of the present invention, the expression "sampling a point cloud" should be understood as a process of randomly and uniformly selecting interest points from the point cloud.

[0047] The point cloud alignment device (100) is configured to match each point of interest ( ) for a set of adjacent points ( ) is extracted as a patch. Here, r represents the patch radius. Next, the point cloud matching device (100) estimates the local reference frame (LRF) of the patch using the 3DsmoothNet method of neighboring points within the patch. Next, the point cloud matching device (100) can calculate three orthogonal vectors for the local coordinate system as the local coordinate system. The three orthogonal vectors for the local reference frame may be an eigenvector corresponding to the smallest eigenvalue of the patch covariance matrix, a weighted sum of vectors from the point of interest to the neighboring point, and an orthogonal vector of the previous two vectors.

[0048] The point cloud matching device (100) estimates a rotation matrix (R) from three orthogonal vectors, and uses the estimated rotation matrix (R) to create a patch ( ) to sort the sorted patches ( ) can be obtained.

[0049] For example, aligned patches ( ) can be obtained as in mathematical expression 1.

[0050]

[0051] In step 125, the point cloud matching device (100) generates a cylindrical circular kernel for each point of the aligned patch. For example, as shown in FIG. 2, the point cloud matching device (100) may generate a circular kernel by arranging additional K points (referred to as kernel points) that form a cylindrical shape for each point of the aligned patch. Next, the point cloud matching device (100) calculates the distance from the surrounding points (neighboring points) within the patch for each kernel point, and calculates the weighted-average position of the neighboring points using the calculated distance as a weight, thereby calculating a representative neighboring point corresponding to each kernel point.

[0052] In step 130, the point cloud matching device (100) calculates rotationally invariant feature values ​​that express the relationship with representative neighboring points using each kernel point of the circular kernel for each point.

[0053] The point cloud matching device (100) can calculate four feature values ​​that express the relationship between each kernel point of a circular kernel and its representative neighboring points. The point cloud matching device (100) can estimate four rotationally invariant feature values, such as the relative distance and angle between each kernel point and its representative neighboring points. The rotationally invariant feature values ​​do not change with rotation changes.

[0054] For example, the angle between the normal vector of each point and the vector from each point to the representative neighboring point, the relative distance from the point to the representative neighboring point, the relative distance from the kernel point to the representative neighboring point, and the relative distance from another kernel point located next to the kernel point to the representative neighboring point can be extracted as rotation-invariant features.

[0055] In step 135, the point cloud matching device (100) generates a rotation-invariant descriptor by applying the extracted rotation-invariant feature values ​​to an artificial intelligence model and encoding them.

[0056] In one embodiment of the present invention, a circular convolution method such as mathematical expression 2 can be applied to efficiently encode rotation-invariant feature values.

[0057]

[0058] Here, represents the feature value extraction function, represents the convolution function. Is Indicates the *th kernel point, and represents the clockwise and counterclockwise adjacent kernels of the kth kernel point. and The order of the kernel points is reversed when they are under the tangent plane. Each kernel point can be encoded symmetrically along the tangent plane to reduce the load on the deep learning network by encoding it with its neighboring kernel points in a rotationally invariant manner and to handle the problem of misestimated LRF (e.g., normal axis sign non-uniqueness). The extracted features are input to four circular convolutions, and the features are compressed by calculating the maximum response along the column (N x 512 to 512) to generate a single descriptor from N encoded features, and a fully connected layer can be applied to generate the final descriptor.

[0059] Network parameters can be trained using the most difficult contrastive loss to narrow the distance between corresponding descriptors (relevant descriptors) while maximizing the irrelevant descriptors.

[0060] That is, the point cloud matching device (100) can use the extracted feature values ​​(P x K x 4) as input to an artificial intelligence model to generate a descriptor containing the topographic information of the patch. That is, N descriptors can be generated from N (natural number) patches.

[0061] In step 140, the point cloud matching device (100) calculates the similarity between the generated descriptors of each point cloud and then uses this to select a key descriptor.

[0062] This will be explained in more detail with reference to Fig. 4.

[0063] For example, the cosine similarity between the descriptor sets (P, Q) of each point cloud P and point cloud Q can be used to calculate the similarity matrix, and then the norm value can be calculated for each row of the similarity matrix to calculate the similarity strength value. Next, the point cloud matching device (100) can select the lower T (natural number) % with low similarity strength values ​​as core descriptors. In this way, by selecting the lower T% with low similarity strength values ​​as core descriptors, the points of monotonous terrain can be excluded and then the matching can be performed. Monotonous terrain such as the floor has high similarity, and thus the similarity strength value is high. Therefore, there is an advantage of increasing the matching accuracy by removing monotonous terrain with high similarity using the similarity strength value and then performing the matching using only the points with features.

[0064] As another example, the point cloud matching device (100) may also derive a dissimilarity matrix by calculating the similarity between rotation-invariant descriptors for two point clouds as in Equation 3.

[0065]

[0066] Here, and represents the ith and jth descriptors of the point cloud *. The point cloud matching device (100) calculates the norm value for each row of the dissimilarity matrix to calculate the dissimilarity strength with other points.

[0067] If this is expressed as a mathematical formula, it is as shown in Mathematical Formula 4.

[0068]

[0069] Here, Is Represents the L2-norm value obtained using the i-th row of * represents the number of descriptors. A higher value indicates that the ith descriptor is different from many other descriptors.

[0070] Therefore, the point cloud matching device (100) has a lower dissimilarity intensity. After excluding descriptors with a dissimilarity intensity lower than %, we can select core descriptors. That is, descriptors with high similarity intensity are monotonous features (e.g., floors), so after excluding them, only distinctive points remain.

[0071] In step 145, the point cloud matching device (100) performs matching of two point clouds using selected key descriptors.

[0072] Figure 5 shows the results of alignment using the 3DMatch data set, which captures an indoor environment using a 3D sensor, and Figure 6 shows the results of alignment using the ETH (outdoor) data set. As shown in Figure 5, consistent performance is observed even under rotational deformation.

[0073] In Figure 6, the dotted box indicates a distinctive terrain feature among the overlapping areas between the two point clouds, and the red arrow indicates a matching error. As shown in Figure 6, the matching result according to one embodiment of the present invention is superior to that of conventional methods.

[0074]

[0075] FIG. 7 is a diagram illustrating the configuration of a point cloud alignment device according to one embodiment of the present invention.

[0076] Referring to FIG. 7, a point cloud matching device (100) according to one embodiment of the present invention is configured to include a patch alignment unit (710), a circular kernel generation unit (720), a feature extraction unit (730), an encoding unit (740), a matching unit (750), a memory (760), and a processor (770).

[0077] The patch alignment unit (710) samples the input point cloud, creates patches for each sampled point of interest, calculates a local coordinate system using the distribution information of each patch, and then aligns each patch.

[0078] The circular kernel generation unit (720) is a means for generating a circular kernel by arranging a plurality of kernel points forming a cylindrical shape at each point of each aligned patch.

[0079] The feature extraction unit (730) is a means for extracting rotationally invariant feature values ​​for each patch using each kernel point of the circular kernel.

[0080] The encoding unit (740) is a means for generating rotation-invariant descriptors by applying the extracted rotation-invariant feature values ​​to an artificial intelligence model.

[0081] The matching unit (750) is a means for performing matching by comparing the similarity between generated rotation-invariant descriptors and selecting descriptors with low similarity as core descriptors.

[0082] The memory (760) stores a program code for performing a point cloud matching method according to one embodiment of the present invention.

[0083] The processor (770) is a means for controlling internal components (e.g., patch alignment unit (710), circular kernel generation unit (720), feature extraction unit (730), encoding unit (740), alignment unit (750), etc.) of the point cloud matching device (100) according to one embodiment of the present invention. In addition, since the detailed functions of the patch alignment unit (710), circular kernel generation unit (720), feature extraction unit (730), and encoding unit (740) are the same as those described in FIG. 1, redundant descriptions will be omitted.

[0084]

[0085] The device and method according to the embodiment of the present invention may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the computer-readable medium may be those specially designed and configured for the present invention or may be those known and usable by those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0086] The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0087] The present invention has been described above, focusing on specific embodiments thereof. Those skilled in the art will appreciate that the present invention can be implemented in modified forms without departing from its essential characteristics. Therefore, the disclosed embodiments should be considered illustrative rather than limiting. The scope of the present invention is set forth in the claims, not the foregoing description, and all differences within the scope equivalent thereto should be construed as being encompassed by the present invention.

Claims

1. (a) A step of sampling an input point cloud, generating patches for each sampled point of interest, calculating a local coordinate system using distribution information of each patch, and then aligning each patch; (b) a step of generating a circular kernel by arranging a plurality of kernel points forming a cylindrical shape for each point of each of the above-mentioned aligned patches, and then extracting rotationally invariant feature values; (c) a step of applying the extracted rotation-invariant feature values ​​to an artificial intelligence model to generate rotation-invariant descriptors; and (d) A point cloud matching method including a step of comparing the similarity between the generated rotation-invariant descriptors and selecting a descriptor with a low similarity as a core descriptor to perform matching.

2. In paragraph 1, Step (b) above, A point cloud matching method further comprising a step of calculating a representative neighboring point by calculating a weighted-average position using the calculated distance as a weight after calculating the distance to neighboring points within the patch for each of the plurality of kernel points.

3. In paragraph 2, Step (b) above, A point cloud matching method characterized in that the angle between the normal vector of the center point of the kernel and the vector from the center point to the representative neighboring point, the relative distance from the center point to the representative neighboring point, the relative distance from the kernel point to the representative neighboring point, and the relative distance from another kernel point adjacent to the kernel point to the representative neighboring point are each extracted as rotation-invariant feature values.

4. In paragraph 1, Step (d) above, A step of calculating the cosine similarity between the generated rotation invariant descriptor and another rotation invariant descriptor and generating a similarity matrix, respectively; A step of calculating the L2 norm for each row of the similarity matrix and calculating the similarity strength respectively; A step of selecting the lower T% (T is a natural number) with low similarity strength as the core descriptor; and A point cloud alignment method comprising the step of performing alignment with another point cloud using the selected key descriptors.

5. A recording medium recording a program code for performing a method according to any one of clauses 1 to 4.

6. A patch alignment unit that samples the input point cloud, creates patches for each sampled point of interest, calculates a local coordinate system using the distribution information of each patch, and then aligns each patch; A circular kernel generation unit that generates a circular kernel by arranging a plurality of kernel points forming a cylindrical shape at each point of each of the above-mentioned aligned patches; A feature extraction unit that extracts rotationally invariant feature values ​​for each patch using each kernel point of the circular kernel; An encoding unit that applies the extracted rotation-invariant feature values ​​to an artificial intelligence model to generate rotation-invariant descriptors; and A point cloud matching device including a matching unit that compares the similarity between the generated rotation-invariant descriptors and selects a descriptor with a low similarity as a core descriptor to perform matching.

7. In paragraph 6, The above circular kernel generation unit is, A point cloud matching device characterized in that it calculates a representative neighboring point by calculating the distance to neighboring points within the patch for each of the plurality of kernel points and then calculating a weighted-average position using the calculated distance as a weight.

8. In paragraph 7, The above feature extraction unit, A point cloud matching device characterized in that it extracts rotationally invariant feature values ​​representing the relationship between each point of each patch and a representative neighboring point using each kernel point of the above-mentioned generated circular kernel.

9. In paragraph 8, The above rotation invariant feature values ​​are, A point cloud matching device characterized in that it includes an angle between a normal vector of the center point of the kernel and a vector from the center point to the representative neighboring point, a relative distance from the center point to the representative neighboring point, a relative distance from the kernel point to the representative neighboring point, and a relative distance from another kernel point adjacent to the kernel point to the representative neighboring point.

10. In paragraph 6, The above matching part is, A point cloud matching device characterized in that the device calculates the cosine similarity between the generated rotation-invariant descriptor and another rotation-invariant descriptor to generate a similarity matrix, calculates the L2 norm for each row of the similarity matrix to calculate the similarity strength, selects the lower T% with the low similarity strength as a core descriptor, and then performs matching with another point cloud using the selected core descriptor.

Images