Method for matching point cloud data generated by multiple lidar devices and computing device using the same

Abstract

The present invention relates to a method for aligning point cloud data generated by a plurality of LiDAR devices and a computing device using the same. A method for aligning point cloud data generated by a plurality of LiDAR devices according to one embodiment of the present invention may include: receiving point cloud data for the same target area from a plurality of LiDAR devices, setting at least one of the plurality of point cloud data as target point cloud data and the remainder as source point cloud data; converting the target point cloud data and the source point cloud data into two-dimensional depth images, respectively, to generate a target image corresponding to the target point cloud data and a source image corresponding to the source point cloud data; performing two-dimensional image matching between the target image and the source image to extract corresponding matching points, converting the matching points into three-dimensional points within the target point cloud data and the source point cloud data to generate a point pair matching relationship between the converted three-dimensional points; and obtaining a transformation relationship between the target point cloud data and the source point cloud data based on the point pair matching relationship, and aligning the source point cloud data and the target point cloud data using the transformation relationship.

Description

Technology Field

[0001] The present invention relates to a matching method, etc., capable of matching point cloud data generated by a plurality of LiDAR devices. Background Technology

[0003] In industrial sites such as manufacturing and construction plants, workers are often involved in activities in the same space as hazardous equipment due to process procedures. In such environments, unexpected safety accidents occur, and the number of fatal accidents claiming the lives of workers is on the rise every year.

[0004] As such, to prevent safety accidents in industrial sites, 3D spatial sensing technology without blind spots is essential for workers working in spaces such as complex machinery and equipment.

[0005] For 3D spatial detection, Light Detection and Ranging (LiDAR) devices can be utilized. Generally, solutions with a structure that processes only point cloud data from a single LiDAR device have been installed and utilized in multiple sets for each zone. However, when data information generated by multiple LiDAR devices is aligned and processed, it is possible to provide event detection services of improved quality.

[0006] However, the Iterative Closest Point (ICP) algorithm, a common point cloud data matching method, has problems such as exhibiting very low matching quality or requiring a very long processing time depending on the situation due to the high influence of data noise and initial settings. Prior art literature

[0008] Korean Patent Publication No. 10-2019-0127624 The problem to be solved

[0009] The present invention aims to provide a method for aligning point cloud data generated by a plurality of LiDAR devices and a computing device utilizing the same, which can quickly and accurately align each point cloud data generated by a plurality of LiDAR devices.

[0010] The present invention aims to provide a method for aligning point cloud data generated from multiple LiDAR devices and a computing device utilizing the same, which can expand the physical detection area by aligning multiple point cloud data and diversify the installation locations of LiDAR devices so as to eliminate blind spots.

[0011] The present invention aims to provide a method for aligning point cloud data generated by a plurality of LiDAR devices, capable of integrating, managing, and processing each point cloud data generated by a plurality of LiDAR devices, and a computing device utilizing the same. means of solving the problem

[0013] A method for aligning point cloud data generated from a plurality of LiDAR devices according to an embodiment of the present invention may include the following steps: utilizing a computing device, receiving point cloud data for the same target area from a plurality of LiDAR devices, setting at least one of the plurality of point cloud data as a target point cloud data and the remainder as source point cloud data; converting the target point cloud data and the source point cloud data into two-dimensional depth images to generate a target image corresponding to the target point cloud data and a source image corresponding to the source point cloud data; performing two-dimensional image matching between the target image and the source image to extract corresponding matching points, converting the matching points into three-dimensional points within the target point cloud data and the source point cloud data to generate a point pair matching relationship between the converted three-dimensional points; and obtaining a transformation relationship between the target point cloud data and the source point cloud data based on the point pair matching relationship, and aligning the source point cloud data and the target point cloud data using the transformation relationship.

[0014] Here, the point cloud data may include three-dimensional coordinate values ​​for each three-dimensional point within the target area where the lidar device irradiated the scan signal, and the received signal strength of the scan signal.

[0015] Here, the setting step may further include the step of applying a clustering filter of the setting shape to the point cloud data to obtain the density between three-dimensional points within the point cloud data included in the clustering filter, and clustering the point cloud data into interest groups based on the density.

[0016] Here, the step of generating the source image may involve replacing pixel values ​​with 0 for the remaining areas other than the interest group included in the target image and the source image.

[0017] Here, the step of generating the point pair matching relationship may be to extract the matching points using the Scale-Invariant Feature Transform (SIFT) between the target image and the source image.

[0018] Here, the matching step may represent the transformation relationship using a rotation matrix and a translation matrix representing the amount of rotation and translation for matching the source point cloud data to the target point cloud data.

[0019] Here, the matching step may involve obtaining the rotation matrix and translation matrix using Singular Value Decomposition (SVD).

[0020] Here, the matching step may involve calculating the matching error between the source point cloud data transformed based on the rotation matrix and translation matrix and the target point cloud data, and updating the rotation matrix and translation matrix so that the matching error is less than a threshold value.

[0021] Here, the matching step may involve generating weights based on the signal strength of each lidar device for the three-dimensional point and calculating the matching error by reflecting the weights.

[0022] According to one embodiment of the present invention, a computer program stored in a medium can be implemented to perform a matching pressure method for point cloud data generated by a plurality of lidar devices described above in combination with hardware.

[0023] A computing device according to one embodiment of the present invention may perform the following steps: aligning point cloud data generated by a plurality of LiDAR devices, wherein the processor receives point cloud data for the same target area from a plurality of LiDAR devices, sets at least one of the plurality of point cloud data as target point cloud data, and sets the remainder as source point cloud data; converts the target point cloud data and the source point cloud data into two-dimensional depth images, respectively, to generate a target image corresponding to the target point cloud data and a source image corresponding to the source point cloud data; performs two-dimensional image matching between the target image and the source image to extract corresponding matching points, converts the matching points into three-dimensional points within the target point cloud data and the source point cloud data, and generates a point pair matching relationship between the converted three-dimensional points; and obtains a transformation relationship between the target point cloud data and the source point cloud data based on the point pair matching relationship, and aligns the source point cloud data and the target point cloud data using the transformation relationship.

[0024] Here, the point cloud data may include three-dimensional coordinate values ​​for each three-dimensional point within the target area where the lidar device irradiated the scan signal, and the received signal strength of the scan signal.

[0025] Here, the above setting may further include applying a clustering filter of the setting shape to the point cloud data, calculating the density between three-dimensional points within the point cloud data included in the clustering filter, and clustering the point cloud data into interest groups based on the density.

[0026] Here, generating the source image may involve replacing pixel values ​​with 0 for the remaining areas other than the interest group included in the target image and the source image.

[0027] Here, generating the above point pair matching relationship may involve using the Scale-Invariant Feature Transform (SIFT) between the target image and the source image.

[0028] Here, the matching described above may be represented by using a rotation matrix and a translation matrix representing the amount of rotation and translation to match the source point cloud data with the target point cloud data, thereby representing the transformation relationship.

[0029] Here, the aforementioned matching may involve obtaining the rotation matrix and translation matrix using Singular Value Decomposition (SVD).

[0030] Here, the matching may involve calculating the matching error between the source point cloud data transformed based on the rotation matrix and translation matrix and the target point cloud data, and updating the rotation matrix and translation matrix so that the matching error is less than a threshold value.

[0031] Here, the matching process may involve generating weights based on the signal strength of each lidar device for the three-dimensional point and calculating the matching error by reflecting the weights.

[0032] In addition, the means for solving the above-mentioned problem do not enumerate all the features of the present invention. Various features of the present invention and the advantages and effects derived therefrom can be understood in more detail by referring to the specific embodiments below. Effects of the invention

[0034] According to a method for aligning point cloud data generated by a plurality of LiDAR devices and a computing device utilizing the same, in accordance with an embodiment of the present invention, it is possible to align each point cloud data generated by a plurality of LiDAR devices quickly and accurately. Furthermore, since a plurality of point cloud data can be aligned, it is possible to expand the physical detection area and diversify the installation locations of the LiDAR devices so as to eliminate blind spots.

[0035] According to a method for aligning point cloud data generated by a plurality of LiDAR devices and a computing device utilizing the same according to an embodiment of the present invention, it is possible to reduce human casualties among workers by preventing safety accidents such as entrapment and falling that may occur in industrial sites in advance.

[0036] According to the method for aligning point cloud data generated by a plurality of LiDAR devices and the computing device utilizing the same according to one embodiment of the present invention, an integrated control monitoring service can be implemented for a manufacturing plant, thereby making it possible to effectively reduce the operating costs of personnel responsible for safety monitoring in a large-scale factory.

[0037] According to a method for aligning point cloud data generated by a plurality of LiDAR devices and a computing device using the same according to an embodiment of the present invention, point cloud data generated by a plurality of LiDAR devices can be aligned into one, thereby enabling the provision of event detection services, such as object recognition of improved quality based on more information.

[0038] However, the effects that can be achieved by the method for matching point cloud data generated by a plurality of LiDAR devices according to the embodiments of the present invention and the computing device using the same are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below. Brief explanation of the drawing

[0040] FIG. 1 is a drawing showing an object recognition system using a plurality of LiDAR devices according to one embodiment of the present invention. FIG. 2 is a drawing for explaining a point cloud data matching device according to an embodiment of the present invention. FIG. 3 is a diagram showing point cloud data according to one embodiment of the present invention. FIG. 4 is a diagram illustrating density-based clustering according to one embodiment of the present invention. FIG. 5 is a diagram illustrating density-based clustering according to another embodiment of the present invention. FIG. 6 is a drawing showing a depth image according to one embodiment of the present invention. FIG. 7 is a drawing showing a plurality of lidar devices according to one embodiment of the present invention. FIG. 8 is a diagram showing image matching between depth images according to one embodiment of the present invention. FIG. 9 is a diagram showing the matching between three-dimensional points in point cloud data according to one embodiment of the present invention. FIG. 10 is a drawing showing a point cloud data matching device according to another embodiment of the present invention. FIG. 11 is a flowchart illustrating a method for aligning point cloud data generated from a plurality of lidar devices according to an embodiment of the present invention. Specific details for implementing the invention

[0041] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" for components used in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. That is, the term "part" used in this invention refers to a hardware component such as software, FPGA, or ASIC, and the "part" performs certain roles. However, the meaning of "part" is not limited to software or hardware. The "part" may be configured to reside in an addressable storage medium or may be configured to run one or more processors. Accordingly, as an example, a 'part' includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or further separated into additional components and 'parts'.

[0042] In addition, when describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art may obscure the essence of the embodiments disclosed in this specification, such detailed description is omitted. Furthermore, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification, and the technical concept disclosed in this specification is not limited by the attached drawings; it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0044] FIG. 1 is a drawing showing an object recognition system using a plurality of LiDAR devices according to one embodiment of the present invention.

[0045] Referring to FIG. 1, an object recognition system according to one embodiment of the present invention may include a LiDAR device (L1, L2), a point cloud data matching device (100), and an object recognition device (200).

[0046] Hereinafter, an object recognition system according to an embodiment of the present invention will be described with reference to FIG. 1.

[0048] Lidar devices (L1, L2) can generate three-dimensional point cloud data for a target area (A). Here, the lidar devices (L1, L2) may be implemented based on Time of Flight (TOF), in which case the lidar device (A) can irradiate a scan signal, such as a laser, within the target area (A). Subsequently, the distance to each three-dimensional point within the target area (A) can be measured by utilizing the time required for each scan signal to be reflected back.

[0049] In addition, according to the embodiment, it is also possible to implement the LiDAR device (L1, L2) using an RGB-D sensor. In this case, distance values ​​for each pixel can be additionally output along with RGB color information on a two-dimensional pixel array of height and width. That is, distance information corresponding to each pixel can be obtained simultaneously along with RGB image information for the target area (A).

[0050] LiDAR devices (L1, L2) can measure multiple points within a target area (A) simultaneously or sequentially, thereby generating three-dimensional point cloud data for the target area (A). Various objects may be located within the target area (A), and the point cloud data can be generated by reflecting the shape of each object.

[0051] As illustrated in FIG. 1, multiple LiDAR devices (L1, L2) may be installed over the same target area (A). That is, with only one LiDAR device (L1), there may be blind spots in part of the target area (A) or insufficient point cloud data may not be generated for the target area (A). Therefore, multiple LiDAR devices (L1, L2) may be installed at different positions and angles to perform scanning of the same target area (A) respectively.

[0052] The point cloud data matching device (100) can match point cloud data generated by a plurality of LiDAR devices (L1, L2) to generate a single point cloud data.

[0053] When a plurality of LiDAR devices (L1, L2) are included within an object recognition system, object extraction, etc., can be performed individually based on point cloud data generated by each LiDAR device (L1, L2); however, according to an embodiment, it is also possible to align the plurality of point cloud data to generate a single point cloud data and use it to perform object detection, etc.

[0054] That is, when multiple lidar devices (L1, L2) individually extract objects, it may be difficult to detect objects located at the boundary, and separate hardware must be provided so that individual lidar devices (L1, L2) can perform object detection.

[0055] On the other hand, by using a point cloud data matching device (100), it is possible to match point cloud data generated from multiple LiDAR devices (L1, L2) to generate a single point cloud data. That is, compared to a single structure that independently processes point cloud data generated from each LiDAR device (L1, L2), the physical detection area can be expanded, and blind spots can be eliminated through diversification of installation locations. In addition, since more point cloud data can be integrated for the same target area, it is possible to provide high-quality event detection services.

[0056] Conventionally, the Iterative Closest Point (IPC) algorithm was used to perform alignment between point cloud data. However, since operations had to be performed on all 3D points included in the point cloud data, there was a problem involving a large amount of computation and consequently a long processing time. Furthermore, if noise was included in the point cloud data or if the correspondence between the point cloud data was incorrectly set, the distance error would not converge, leading to problems such as adversely affecting the alignment quality.

[0057] To resolve this, a point cloud data matching device (100) according to one embodiment of the present invention can generate a point pair matching relationship based on visual feature points on a two-dimensional depth image of source point cloud data and target point cloud data, and then match the entire source point cloud data and target point cloud data based on the point pair matching relationship. Through this, it is possible to perform matching for the point cloud data generated by each LiDAR device (L1, L2) quickly and accurately.

[0058] The object recognition device (200) can receive matched point cloud data from the point cloud data matching device (100) and can perform operations such as detecting a preset target object using the matched point cloud data. For example, if the target area (A) is set within a workplace such as a factory, it can perform operations such as setting a worker (1) located within the target area (A) as a target object and detecting it.

[0059] By utilizing this, an integrated control and monitoring service can be established for manufacturing sites with dozens to hundreds of locations, and the operating costs for safety monitoring personnel within large-scale factories can be effectively reduced.

[0060] In addition, since point cloud data from LiDAR devices (L1, L2) that are independently installed and operated within the same site can be integrated and provided as a single unit, it is possible to provide event detection services of improved quality based on more information, and it is possible to integrate the controllers for processing point cloud data from multiple LiDAR devices (L1, L2) into a single unit.

[0062] FIG. 2 is a drawing for explaining a point cloud data matching device according to an embodiment of the present invention.

[0063] Referring to FIG. 2, a point cloud data matching device (100) according to one embodiment of the present invention may include a preprocessing unit (110), a depth image conversion unit (120), a point pair matching unit (130), and a matching unit (140).

[0064] The preprocessing unit (110) can receive point cloud data for the same target area (A) from a plurality of LiDAR devices (L1, L2). Here, the point cloud data may include 3D coordinate values ​​for each 3D point within the target area (A) where the LiDAR devices (L1, L2) have irradiated a scan signal. Additionally, depending on the embodiment, the received signal strength of the scan signal for each 3D point within the point cloud data may also be included. The received signal strength may be information corresponding to the data reliability of the LiDAR devices (L1, L2), and depending on the embodiment, may have a range of [0, 100]. Here, it can be determined that the higher the received signal strength, the higher the data reliability, distance accuracy, etc., of the corresponding point. The data reliability of the LiDAR devices (L1, L2) may vary depending on the surrounding environment, such as direct sunlight, a reflector with high reflectivity, or dust in the air.

[0065] Meanwhile, the target area (A) may be a pre-set Region of Interest (ROI) area among the scannable areas of the LiDAR devices (L1, L2). As illustrated in FIG. 3, a target object (O) may be included within the ROI area (ROI). For example, in the case of a factory, there may be a high-risk area where an accident is likely to occur when a worker is located, and such a high-risk area can be set as the ROI area among the entire scannable area. Additionally, a space requiring concentrated detection as a risk zone in advance, such as a worker's movement path or a specific space where hazardous equipment is installed, can be registered as the ROI area to receive point cloud data. Furthermore, one or more ROI areas can be set in the factory, etc., to pre-define areas where safety accidents may occur, and concentrated monitoring of such areas can be performed.

[0066] The preprocessing unit (110) may set one of the received multiple point cloud data as the target point cloud data and the rest as the source point cloud data. Here, the remaining source point cloud data may be aligned by shifting them based on the target point cloud data. For example, the point cloud data of the lidar device (L1) located above the target area (A) may be set as the target point cloud data, and the point cloud data of the lidar device (L2) located on the side of the target area (A) may be set as the source point cloud data. However, this is not limited thereto, and the target point cloud data and source point cloud data may be set in various ways depending on the embodiment.

[0067] Additionally, the preprocessing unit (110) can apply filtering to separate the area where the target object (O) exists from the area caused by environmental noise among the point cloud data, thereby extracting only the point cloud data corresponding to the target object (O). Specifically, when point cloud data is input, the preprocessing unit (110) can generate multiple interest groups by performing clustering on the target area (A) based on the density of the point cloud data.

[0068] That is, the preprocessing unit (110) can identify each interest group through clustering, and subsequently, the object recognition device (200), etc., can perform object classification, etc. for each identified interest group. Through this, it is possible to perform worker detection optimized for industrial sites utilizing limited computing resources.

[0069] The preprocessing unit (110) can cluster interest groups and delete interest groups that do not meet the conditions, thereby minimizing the load on the classification process which requires a relatively large amount of processing time.

[0070] The preprocessing unit (110) can cluster point cloud data into multiple interest groups. Here, the preprocessing unit (110) can perform clustering based on the density of points of the point cloud data, and can utilize a clustering filter set according to the shape of the object. That is, clustering can be performed based on the density of points of the point cloud data included in each clustering filter.

[0071] In the area where the target object (O) exists within the 3D point cloud data, the 3D points are densely packed within a certain volume of space. On the other hand, in the case of noise, the 3D points may exist with a locally low density. Additionally, 3D points with high reliability regarding distance information have high reception strength, while 3D points with inaccurate distance values ​​or reduced sensitivity due to noise may appear with low reception strength.

[0072] Here, the preprocessing unit (110) can perform clustering as a preliminary operation for matching the received point cloud data, based on the density of the point cloud data and the received signal strength for each 3D point, by grouping point cloud data with a density greater than a certain level into one interest group. Through this, 3D points corresponding to noise can be removed, and interest groups having a shape that geometrically corresponds to the target object (O) can be effectively found.

[0073] Specifically, the preprocessing unit (110) can determine the number of remaining 3D points included in the target space when each 3D point is located at the center of the target space set by the cluster filter. Here, if the number of remaining 3D points is greater than or equal to the set number, the 3D point can be set as a core point, which is a 3D point constituting the interest group.

[0074] Density-based clustering of point cloud data can be obtained through the following formula.

[0075]

[0076] Here, p i is the i-th point included in the point cloud data, p c is the key point, p b is the boundary point, p n is the noise point, ε is the radius, and N min_pts is the minimum number of points. Referring to FIG. 4, the preprocessing unit (110) scans all point cloud data and points p i The number of 3D points within a constant radius ε from is the minimum number of points N. min_pts If so, the corresponding 3D point p i Key point p c It can be set to. Afterwards, key point p c For 3D points located within the radius of, respectively, key point p c Check if it applies to, and key point p c If applicable, each key point p c All of them can be included within a single cluster (G). In this case, the number of 3D points existing within radius ε is the minimum number of points N. min_pts If it is less than or equal to noise point p n It is determined as such, and the number of 3D points existing within radius ε is the minimum number of points N. min_pts Although it is as follows, the relevant 3D point is another key point p c If it is within radius ε, boundary point p b It can be determined as.

[0077] Here, the preprocessing unit (110) can perform adaptive object-oriented density-based clustering by considering the shape of the object to be recognized and the sensing distance between the LiDAR device (L1, L2) and the target area. For example, since workers working in industrial sites are mainly walking or standing in place, an ellipsoidal clustering filter suitable for worker detection can be applied. Subsequently, the point p to be determined i For this, the number of points existing within the radius of the ellipsoidal clustering filter is the minimum number of points N. min_pts It is possible to determine whether there is an abnormality and form a cluster.

[0078] Here, the cluster filter can be set to an ellipsoid, sphere, capsule, polygonal prism, polygonal pyramid, torus, etc., and can also be set to various other shapes. Specifically, the ellipsoid is a filter in the shape of an ellipsoid that is long in the vertical direction and short in the horizontal direction, and the sphere corresponds to a filter in the shape of a sphere with a constant radius. In addition, the capsule shape may have a spherical shape at the top and bottom, and a cylinder shape in the middle connecting the top and bottom. The polygonal prism corresponds to a filter in the shape of a polygonal prism, such as a triangular prism or a square prism, and the polygonal pyramid corresponds to a filter in the shape of a polygonal base and a horn shape at the top, such as a triangular pyramid or a square pyramid. In other words, various types of cluster filters can be applied depending on the shape of the object to be detected.

[0079] Additionally, the preprocessing unit (110) can variably set the size of the target space according to the sensing distance between the LiDAR device (L1, L2) and the target area (A). That is, the size of the target space can be set larger as the sensing distance is closer, and the size of the target space can be set smaller as the sensing distance is farther. The size of the target space and the sensing distance can be set to be inversely proportional. In order to variably configure a clustering filter (ellipsoid) suitable for the target object according to the sensing distance, α d , β d , γ d , N d min_pts The following formula can be applied to the parameter. α d , β d , γ d , N d min_pts As the distance increases, it can have a range corresponding to twice the initial value from the initial value at 0m.

[0080] α d = (log M+1 (d+1) + 1)α0

[0081] β d = (log M+1 (d+1) + 1) β0

[0082] γ d = (log M+1 (d+1) + 1) γ0

[0083] N d min_pts = (log M+1 (d+1) + 1) N 0 min_pts

[0084] Where d: distance from LiDAR to object(0≤d≤M)

[0085] α0, β0, γ0, N 0 min_pts : the value of α, β, γ, N min_pts at 0m

[0086] M: maximum distance of sensing

[0088] Referring to FIG. 5, an embodiment using an ellipsoidal cluster filter is shown, p i In the case of, within the ellipsoid radius Point p that satisfies k The number of is the minimum number of points N d min_pts If it is more than p i It can be determined that this corresponds to the key point.

[0089] However, depending on the embodiment, a 3D point p included in the target space k Instead of the number, the received signal strength at each 3D point can be utilized. That is, each 3D point p at the center of the target space set by the clustering filter i When is located, determine the remaining 3D points included within the target space and the respective received signal strength l k They can be combined. Subsequently, if the sum of the received signal strengths is greater than or equal to a preset value, the corresponding 3D point can be set as a core point, which is a point constituting a group of interest. According to an embodiment, a weight is assigned to each received signal strength, so that the received signal strength l k weighted sum ∑c k Measure and the minimum number of points N d min_pts It can be compared with. Here, ∑c k Ga N d min_pts If it is greater than or equal to, the corresponding 3D point p iThis can be determined as a key point. Since the received signal strength corresponds to the reliability of the 3D coordinate values ​​of each 3D point, a weak received signal strength can be seen as indicating that the reliability of the corresponding 3D coordinate values ​​is low. Therefore, instead of using the number of each 3D point included within the target space, the received signal strengths can be summed so that if the reliability of each 3D point is above a certain value, it can be determined whether the 3D point is a key point.

[0090] Meanwhile, 3D point p of the LiDAR device (L1, L2) k The received signal strength Ik corresponding to can be adjusted to have a range of [0, 1] and can be used as an indicator representing the reliability of the data corresponding to the point. In Fig. 5, p l In the case of points, Since it corresponds to , it corresponds to a noise point.

[0092] The depth image conversion unit (120) can convert target point cloud data and source point cloud data into two-dimensional depth images, respectively. Here, the point cloud data clustered into interest groups in the preprocessing unit (110) can be converted into two-dimensional depth images, respectively, by the depth image conversion unit (120). At this time, the depth image corresponding to the target point cloud data is the target image, and the depth image corresponding to the source point cloud data is the source image.

[0093] According to an embodiment, the depth image conversion unit (120) may replace pixel values ​​with 0 for the remaining areas other than the interest group included in the target image and the source image. That is, referring to FIG. 6, the initial depth image may appear together as FIG. 6(a), and among them, the interest group corresponding to the object (O) may be set by the preprocessing unit (110). In this case, as shown in FIG. 6(b), pixel values ​​may be replaced with 0 for the remaining 3D points excluding the 3D points corresponding to the interest group while generating the depth image. That is, as shown in FIG. 6(b), only the area corresponding to the interest group may remain in the depth image. In this case, since only the area corresponding to the interest group remains in the depth image, it is possible to perform matching with a higher matching accuracy when matching the source image and the target image.

[0094] The point pair matching unit (130) can perform 2D image matching between a target image and a source image to extract corresponding matching points. Additionally, it can convert the matching points into 3D points within the target point cloud data and the source point cloud data, and generate point pair matching relationships between the converted 3D points.

[0095] Here, the point pair matching unit (130) may use SIFT (Scale-Invariant Feature Transform) for pixel-level matching between the target image and the source image. SIFT is an algorithm that extracts image feature points that are invariant to the size, rotation, illumination, noise, etc. of an object. That is, the point pair matching unit (130) can extract SIFT feature points by applying SIFT to each target image and source image, and then compare the SIFT feature points of each target image and source image to match the SIFT feature points with the smallest difference as matching points. The target image and source image correspond to those generated by each LiDAR device (L1, L2) scanning the same target area (A) at different positions or angles. Therefore, by using the SIFT algorithm, corresponding SIFT feature points between the target image and the source image can be found, and the most similar SIFT feature points can be extracted as matching points.

[0096] Subsequently, the point pair matching unit (130) can convert each matching point included in the target image and the source image into a 3D point included in the 3D target point cloud data and the source point cloud data, and generate a point pair matching relationship between the converted 3D points. That is, by generating a point pair matching relationship of 3D points corresponding to the matching points, it is possible to determine the positional relationship between the target point cloud data and the source point cloud data.

[0097] For example, as illustrated in FIG. 7, two lidar devices (L1, L2) can each generate point cloud data for the same target area, and the point cloud data of the first lidar device (L1) can be set as target point cloud data and the point cloud data of the second lidar device (L2) can be set as source point cloud data.

[0098] Subsequently, target and source images can be generated by converting target point cloud data and source point cloud data into depth images, respectively, and image mapping between depth images can be performed as illustrated in FIG. 8. In this case, SIFT feature points are extracted within each depth image, and matching points can be obtained by comparing each SIFT feature point and matching similar SIFT feature points with each other.

[0099] Here, as illustrated in FIG. 9, while converting each target image and source image into three-dimensional point cloud data, a point-pair matching relationship for three-dimensional points corresponding to each matching point can be generated. That is, a two-dimensional matching relationship can be converted into a three-dimensional point-pair matching relationship.

[0101] The matching unit (140) can obtain a transformation relationship between the target point cloud data and the source point cloud data based on the point pair matching relationship, and can match the source point cloud data and the target point cloud data using the transformation relationship. That is, the matching unit (140) can obtain a rotation matrix and a translation matrix based on the point pair matching relationship for 3D points, and perform matching between the source point cloud data and the target point cloud data by applying the ICP algorithm. Here, the rotation matrix R and the translation matrix t may represent the amount of rotation and the amount of translation, respectively, for matching the source point cloud data to the target point cloud data.

[0102] Specifically, the matching unit (140) can set the point pair matching relationship of each 3D point as an initial value to apply the ICP algorithm, and obtain the rotation matrix R and translation matrix t based on the Singular Value Decomposition (SVD) algorithm for the entire source point cloud data and target point cloud data.

[0103] Afterwards, the matching unit (140) can convert source point cloud data based on the rotation matrix R and the translation matrix t, and can obtain the matching error (E) between the converted source point cloud data and the target point cloud data.

[0104] Specifically, x n is source point cloud data, and corresponds to the position obtained by shifting the source point cloud data using the rotation matrix R and the translation matrix t. Therefore, the transformed source point cloud data It can be obtained using .

[0105] In addition, the matching error E can be calculated as follows.

[0106]

[0107] y n is the target point cloud data, and the target point cloud data y n and transformed source point cloud data The matching error E can be calculated by finding the squared difference for all n 3D points and adding them.

[0108] Here, the weight w is used for the matching error E. n ...can be further reflected, and weight w n It can be generated based on the received signal strength of each LiDAR device (L1, L2) for a 3D point. That is, a low weight can be assigned to point pairs with low received signal strength to have a small effect on updating the matching error, and a high weight can be assigned to point pairs with high received signal strength to have a greater effect on updating the matching error. Through this, the influence of noise or 3D points with low measurement reliability among the measured received signal strengths can be filtered to minimize, and accordingly, accurate matching results can be obtained.

[0109] Subsequently, the alignment unit (140) can repeatedly update the rotation matrix R and translation matrix t until the alignment error becomes less than a threshold value. That is, the alignment unit (140) can, the transformed source point cloud data and the target point cloud data Since the smaller the difference between them, the more accurately they are considered to be aligned, the rotation matrix R and translation matrix t can be updated and applied so that the alignment error E falls below a preset threshold. Finally, when the alignment error E converges to below the threshold, the alignment unit (140) can generate and provide an alignment result for the source point cloud data and the target point cloud data.

[0111] FIG. 10 is a block diagram illustrating a computing environment (10) suitable for use in exemplary embodiments. In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

[0112] The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be a point cloud data matching device (100). The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) may enable the computing device (12) to operate according to the exemplary embodiment described above. For example, the processor (14) may execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, and the computer-executable instructions may be configured to enable the computing device (12) to perform operations according to the exemplary embodiment when executed by the processor (14).

[0113] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by a processor (14). In one embodiment, the computer-readable storage medium (16) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other forms of storage media that are accessed by a computing device (12) and capable of storing desired information, or a suitable combination thereof.

[0114] The communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and the computer-readable storage medium (16).

[0115] The computing device (12) may also include one or more input / output interfaces (22) and one or more network communication interfaces (26) that provide interfaces for one or more input / output devices (24). The input / output interfaces (22) and network communication interfaces (26) are connected to a communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) through the input / output interfaces (22). An exemplary input / output device (24) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (24) may be included inside the computing device (12) as a component constituting the computing device (12), or it may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0117] FIG. 11 is a flowchart illustrating a method for aligning point cloud data generated by a plurality of lidar devices according to an embodiment of the present invention. Herein, each step of FIG. 11 may be performed by a point cloud data aligning device according to an embodiment of the present invention. Hereinafter, a method for aligning point cloud data generated by a plurality of lidar devices according to an embodiment of the present invention will be described with reference to FIG. 11.

[0118] The point cloud data matching device can receive point cloud data for the same target area from a plurality of lidar devices (S110). Here, the point cloud data may include 3D coordinate values ​​for each 3D point within the target area where the lidar device scanned the signal, and, depending on the embodiment, may also include the received signal strength of the scan signal for each 3D point within the point cloud data.

[0119] Additionally, the point cloud data matching device may set one of the received multiple point cloud data as the target point cloud data and the rest as the source point cloud data. Here, the target point cloud data may be the reference point cloud data, and the source point cloud data may be the point cloud data that moves to be matched with the target point cloud data.

[0120] Here, when point cloud data is input, the point cloud data matching device can generate interest groups by performing clustering on a target area based on the density of the point cloud data. At this time, the clustering filter can be set according to the shape of the object, and clustering can be performed based on the density of 3D points within the point cloud data included in the clustering filter.

[0121] According to an embodiment, an ellipsoidal cluster filter suitable for detecting workers working in an industrial site may be applied. In this case, the point p to be distinguished i For this, the number of points existing within the radius of the ellipsoidal clustering filter is the minimum number of points N. min_pts Clusters can be formed by determining whether there is an anomaly. Here, the cluster filter can be set to various shapes in addition to ellipsoids, such as spheres, capsules, polygonal prisms, polygonal pyramids, and torus.

[0122] In addition, the point cloud data matching device can variably set the size of the target space according to the sensing distance between the LiDAR device and the target area. That is, the size of the target space can be set larger as the sensing distance decreases, and the size of the target space can be set smaller as the sensing distance increases.

[0123] The point cloud data matching device can convert the target point cloud data and the source point cloud data into two-dimensional depth images, respectively, to generate a target image corresponding to the target point cloud data and a source image corresponding to the source point cloud data (S120). Here, the point cloud data matching device can convert the point cloud data clustered into interest groups into two-dimensional depth images. Additionally, according to the embodiment, the point cloud data matching device can replace pixel values ​​with 0 for the remaining areas other than the interest groups included in the target image and the source image. That is, by ensuring that only the areas corresponding to the interest groups remain in the depth image, it is possible to induce matching with a higher matching accuracy when matching the source image and the target image.

[0124] Subsequently, the point cloud data matching device can extract corresponding matching points by performing 2D image matching between the target image and the source image. In addition, the matching points can be converted into 3D points within the target point cloud data and the source point cloud data to generate point pair matching relationships between the converted 3D points (S130).

[0125] Here, the point cloud data matching device can utilize SIFT (Scale-Invariant Feature Transform) for pixel-level matching between the target image and the source image. That is, the point cloud data matching device can extract SIFT feature points by applying SIFT to each target image and source image, and then compare the SIFT feature points of each target image and source image to match the SIFT feature points with the smallest difference as matching points.

[0126] In addition, the point cloud data matching device can convert each matching point included in the target image and the source image into a 3D point included in the 3D target point cloud data and the source point cloud data, and generate a point pair matching relationship between the converted 3D points. That is, by generating a point pair matching relationship of 3D points corresponding to the matching points, it is possible to determine the positional relationship between the target point cloud data and the source point cloud data.

[0127] Subsequently, the point cloud data matching device can obtain a transformation relationship between the target point cloud data and the source point cloud data based on the point pair matching relationship, and can match the source point cloud data and the target point cloud data using the transformation relationship (S140). That is, the point cloud data matching device can obtain a rotation matrix and a translation matrix based on the point pair matching relationship for 3D points, and perform matching between the source point cloud data and the target point cloud data by applying the ICP algorithm. Here, the rotation matrix R and the translation matrix t may represent the amount of rotation and the amount of translation, respectively, for matching the source point cloud data to the target point cloud data.

[0128] Specifically, to apply the ICP algorithm, the point cloud data matching device sets the point pair matching relationship of each 3D point as an initial value, and for the entire source point cloud data and target point cloud data, it can obtain the rotation matrix R and the translation matrix t based on the Singular Value Decomposition (SVD) algorithm.

[0129] Subsequently, source point cloud data can be transformed based on the rotation matrix R and the translation matrix t, and the matching error between the transformed source point cloud data and the target point cloud data can be calculated. Here, weights may be further applied to the matching error E, and these weights can be generated based on the received signal strength of each LiDAR device for the 3D points. That is, low weights can be assigned to point pairs with low received signal strengths to minimize the influence on updating the matching error, while high weights can be assigned to point pairs with high received signal strengths to have a greater influence on updating the matching error. Through this, the influence of noise or 3D points with low measurement reliability among the measured received signal strengths can be filtered to be minimized, thereby enabling accurate matching results to be obtained.

[0130] Subsequently, the point cloud data matching device can iteratively update the rotation matrix R and translation matrix t until the matching error falls below a threshold. That is, the transformed source point cloud data and target point cloud data Since a smaller difference between them indicates a more accurate match, the rotation matrix R and translation matrix t can be updated and applied so that the match error falls below a preset threshold. Finally, when the match error converges below the threshold, the point cloud data matching device can generate and provide a match result for the source point cloud data and the target point cloud data.

[0132] The present invention described above can be implemented as computer-readable code on a medium on which a program is recorded. The computer-readable medium may be one that continuously stores a program executable by a computer, or temporarily stores it for execution or download. Furthermore, the medium may be various recording or storage means in the form of a single or multiple hardware components, and is not limited to a medium directly connected to a computer system but may also exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software. Accordingly, the above detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the present invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included within the scope of the present invention.

[0134] The present invention is not limited by the aforementioned embodiments and attached drawings. It will be obvious to those skilled in the art that the components according to the present invention can be substituted, modified, and changed within the scope of the technical concept of the present invention without departing from the spirit of the invention. Explanation of the symbols

[0136] 100: Point cloud data matching device 110: Preprocessing unit 120: Depth image conversion unit 130: Point pair matching unit 140: Matching unit 200: Object recognition device

Claims

Claim 1 A method for matching point cloud data generated from a plurality of LiDAR devices using a computing device, comprising: receiving point cloud data for the same target area from a plurality of LiDAR devices, setting at least one of the plurality of point cloud data as target point cloud data and the remainder as source point cloud data; converting the target point cloud data and the source point cloud data into two-dimensional depth images, respectively, to generate a target image, which is a two-dimensional depth image corresponding to the target point cloud data, and a source image, which is a two-dimensional depth image corresponding to the source point cloud data; performing two-dimensional image matching between the target image and the source image to extract corresponding matching points, and converting the matching points into three-dimensional points within the target point cloud data and the source point cloud data, respectively, to generate a point pair matching relationship between the converted three-dimensional points.The method comprises the step of obtaining a transformation relationship between the target point cloud data and the source point cloud data based on the point pair matching relationship, and aligning the source point cloud data and the target point cloud data using the transformation relationship, wherein the point cloud data includes 3D coordinate values ​​for each 3D point within the target area where the LiDAR device irradiated a scan signal, and the received signal strength of the scan signal, and the setting step involves applying a clustering filter of a setting shape to the point cloud data to obtain the density between 3D points within the point cloud data included in the clustering filter, clustering the point cloud data into interest groups based on the density, and setting the point cloud data included in the interest groups as the target point cloud data and the source point cloud data, wherein the clustering filter sets the size of the target space, which is set by the setting shape of the clustering filter, differently according to the sensing distance between the plurality of LiDAR devices and the target area, and obtains the density based on the sum of the received signal strengths of the 3D points included in the target space, generated by a plurality of LiDAR devices Point cloud data alignment method.; Claim 2 delete Claim 3 delete Claim 4 A method for matching point cloud data generated by a plurality of LiDAR devices, wherein the step of generating the source image is to replace pixel values ​​with 0 for the remaining areas other than the interest group included in the target image and the source image. Claim 5 A method for matching point cloud data generated by a plurality of LiDAR devices, wherein the step of generating the point pair matching relationship in claim 1 is to extract the matching points using SIFT (Scale-Invariant Feature Transform) between the target image and the source image. Claim 6 A method for matching point cloud data generated by a plurality of LiDAR devices, wherein the matching step is to represent the transformation relationship using a rotation matrix and a translation matrix representing a rotation amount and a translation amount for matching the source point cloud data to the target point cloud data. Claim 7 A method for matching point cloud data generated from a plurality of LiDAR devices, wherein, in claim 6, the matching step is to obtain the rotation matrix and translation matrix using Singular Value Decomposition (SVD). Claim 8 A method for aligning point cloud data generated by a plurality of LiDAR devices, wherein the aligning step is to calculate an alignment error between source point cloud data transformed based on the rotation matrix and translation matrix and the target point cloud data, and to update the rotation matrix and translation matrix so that the alignment error is less than a threshold value. Claim 9 A method for matching point cloud data generated by a plurality of lidar devices, wherein the matching step in claim 8 is to generate weights based on the signal strength of each lidar device for the three-dimensional point and to obtain the matching error by reflecting the weights. Claim 10 A computer program stored on a medium to perform a method of matching point cloud data generated by a plurality of lidar devices according to any one of claims 1, 4 to 9, in combination with hardware. Claim 11 A computing device comprising a processor and matching point cloud data generated from a plurality of LiDAR devices, wherein the processor, upon receiving point cloud data for the same target area from a plurality of LiDAR devices, sets at least one of the plurality of point cloud data as target point cloud data and sets the remainder as source point cloud data; converting the target point cloud data and the source point cloud data into two-dimensional depth images, respectively, to generate a target image, which is a two-dimensional depth image corresponding to the target point cloud data, and a source image, which is a two-dimensional depth image corresponding to the source point cloud data; performing two-dimensional image matching between the target image and the source image to extract corresponding matching points, and converting the matching points into three-dimensional points within the target point cloud data and the source point cloud data, respectively, to generate point pair matching relationships between the converted three-dimensional points;A computing device comprising: obtaining a transformation relationship between the target point cloud data and the source point cloud data based on the point pair matching relationship, and aligning the source point cloud data and the target point cloud data using the transformation relationship; wherein the point cloud data includes 3D coordinate values ​​for each 3D point within the target area where the LiDAR device irradiated a scan signal, and the received signal strength of the scan signal; wherein the setting involves applying a clustering filter of a set shape to the point cloud data to obtain the density between 3D points within the point cloud data included in the clustering filter, clustering the point cloud data into interest groups based on the density, and setting the point cloud data included in the interest groups as the target point cloud data and the source point cloud data; wherein the clustering filter sets the size of the target space, which is set by the set shape of the clustering filter, differently according to the sensing distance between the plurality of LiDAR devices and the target area, and calculates the density based on the sum of the received signal strengths of the 3D points included in the target space. Claim 12 delete Claim 13 delete Claim 14 A computing device according to claim 11, wherein generating the source image involves replacing pixel values ​​with 0 for the remaining areas other than the interest group included in the target image and the source image. Claim 15 A computing device according to claim 11, wherein generating the point pair matching relationship utilizes the Scale-Invariant Feature Transform (SIFT) between the target image and the source image. Claim 16 A computing device according to claim 11, wherein the matching above represents the transformation relationship using a rotation matrix and a translation matrix representing a rotation amount and a translation amount for matching the source point cloud data to the target point cloud data. Claim 17 A computing device according to claim 16, wherein the matching above is to obtain the rotation matrix and translation matrix using Singular Value Decomposition (SVD). Claim 18 A computing device according to claim 16, wherein the matching is to calculate a matching error between source point cloud data transformed based on the rotation matrix and translation matrix and the target point cloud data, and to update the rotation matrix and translation matrix so that the matching error is less than a threshold value. Claim 19 A computing device according to claim 18, wherein the matching is performed by generating weights based on the signal strengths of each LiDAR device for the three-dimensional point and calculating the matching error by reflecting the weights.

Images