Image point cloud automatic registration method without fixed target

Abstract

The invention discloses an automatic registration method for an image point cloud without a fixed target, and belongs to the field of computer vision. According to the invention, automatic feature matching is carried out by using edge contour information in a 2D image and a 3D point cloud, 3D point cloud edge information is converted into a 2D space by continuously adjusting position postures, feature point pairs are constructed by using neighbor intertraversal and a local matching algorithm, an optimal matching point pair is determined in a mode of first rough matching and then fine matching, and the optimal matching point pair is obtained. And then solving and determining a corresponding relation between the two through nonlinear optimization, and completing automatic registration of the image point cloud. Calibration work does not need to depend on specially arranged calibration objects and environments, a target object does not need to be trained in advance, steps are simple, design is reasonable, implementation is convenient, the using effect is good, good universality is achieved, the method can be effectively suitable for the point cloud data and image automatic registration process of different terrains, and the method is suitable for popularization and application. A simpler and more convenient method is provided for fusion registration of the image and the point cloud, and the work period is effectively shortened.

Description

Technical Field

[0001] The invention relates to the field of computer vision, and in particular to an automatic registration method for image point clouds without fixed targets. Background Art

[0002] In the existing technology, it is usually necessary to use the registration of 3D laser point cloud and monocular vision image, for example, in scenarios such as autonomous driving and drone mapping. At present, the registration of image and 3D point cloud requires manual selection of feature points with the same name for matching or design of specific target calibration objects for identification and positioning. The calibration work depends on the specially arranged specific calibration objects and environment, and the steps are cumbersome and not universal. Summary of the invention

[0003] The present invention provides an automatic registration method for image point clouds without fixed targets. The calibration work does not need to rely on specially arranged calibration objects and environments. The method has simple steps, reasonable design, convenient implementation, good use effect and good versatility.

[0004] The present invention provides the following technical solutions:

[0005] An automatic registration method for image point clouds without fixed targets, the method comprising:

[0006] S10: Acquire 2D image data and 3D point cloud data in the same monitoring area at the same time, and obtain camera calibration parameters;

[0007] S20: extracting edge information of the 3D point cloud data and the 2D image data respectively, and obtaining a point cloud 3D boundary point set and an image 2D boundary point set respectively;

[0008] S30: taking the point cloud 3D boundary point set as input, and performing posture transformation on the point cloud 3D boundary point set according to a preset initial posture;

[0009] S40: Generate a perspective transformation matrix using camera calibration parameters, perform perspective projection transformation on the point cloud 3D boundary point set after the posture transformation, and generate a point cloud 2D boundary point set;

[0010] S50: Find common feature point pairs between the point cloud 2D boundary point set and the image 2D boundary point set, and obtain a list of common feature point pairs;

[0011] Among them, when a point cloud point in the point cloud 2D boundary point set and an image point in the image 2D boundary point set are neighboring points to each other, and the similarity between the local image at the point cloud point and the local image at the image point reaches a set similarity threshold, the point cloud point and the image point are regarded as a common feature point pair;

[0012] S60: Calculating a cost value according to the number of point pairs in the common feature point pair list and the number of 2D boundary point sets of the point cloud;

[0013] S70: adjusting the posture according to the cost value, performing posture transformation on the point cloud 3D boundary point set according to the adjusted posture, and returning to S40 for the next round of iteration; the iteration ends after reaching the set number of iterations, outputting the minimum cost value and its corresponding posture, and executing S80;

[0014] S80: finely adjust the posture corresponding to the minimum cost value, transform the posture of the point cloud 3D boundary point set according to the finely adjusted posture, and return to S40 for the next round of iteration; the iteration ends when the cost value reaches the set convergence, outputs the minimum cost value and its corresponding public feature point pair list, and executes S90;

[0015] S90: Perform nonlinear optimization based on the list of common feature point pairs corresponding to the minimum cost value, and output the final registration transformation relationship.

[0016] Further, S10 includes:

[0017] S11: photographing the monitoring area through a camera to obtain 2D image data;

[0018] S12: Scanning the same monitoring area at the same time by means of a 3D point cloud scanning device to obtain 3D point cloud data;

[0019] S13: Calibrate the camera to obtain camera calibration parameters; wherein the camera calibration parameters include camera focal length parameters and camera optical center parameters.

[0020] Further, the S20 includes:

[0021] S21: performing voxel filtering and downsampling on the 3D point cloud data;

[0022] S22: traversing the 3D point cloud data after voxel filtering and downsampling, and performing stereo block processing according to a preset stereo block space size;

[0023] S23: In each three-dimensional block, plane recognition is performed by random sampling, and the intersection of two planes is obtained according to the intersection of plane normals, and a point cloud 3D boundary point set is constructed according to the intersection;

[0024] S23: In each stereo block, line segment estimation is performed by random sampling method, and the points where the line segments are located are added to the 3D boundary point set of the point cloud;

[0025] S24: Perform edge detection on the 2D image data, and construct an image 2D boundary point set according to the detected contour boundary.

[0026] Further, the S50 includes:

[0027] S51: traverse each point cloud point in the point cloud 2D boundary point set, and perform K nearest neighbor search on the traversed point cloud point and each image point in the image 2D boundary point set, obtain the image points with the minimum distance to the traversed point cloud point, and record them in the list imgPt_list;

[0028] S52: Based on the traversed point cloud point as a reference, a K nearest neighbor search is performed in the point cloud 2D boundary point set to obtain the point cloud point with the minimum distance to the traversed point cloud point, and the point cloud point is recorded in the list lidarPt_list;

[0029] S53: traverse each image point in the list imgPt_list, and reversely perform a K-nearest neighbor search on the point cloud points in the point cloud 2D boundary point set using the traversed image points to determine whether the image point and the point cloud point are neighboring points to each other;

[0030] S54: performing local feature matching on the local images of the area where the list imgPt_list and the list lidarPt_list are located, and calculating the similarity;

[0031] S55: When the image point and the point cloud point are neighboring points and the similarity reaches a set similarity threshold, the point cloud point and the image point are taken as a common feature point pair and added to a common feature point pair list pointPair.

[0032] Furthermore, the calculation formula of the cost value is as follows:

[0033] cost = 1-(P1 / P)

[0034] Among them, P1 is the number of point pairs in the common feature point pair list pointPair, and P is the number of 2D boundary point sets of the point cloud.

[0035] Further, the adjusting the posture according to the cost value includes:

[0036] S71: Calculate the difference between the cost value of this iteration and the cost value of the previous iteration, and determine the adjustment range of the posture according to the difference;

[0037] Wherein, when the difference is smaller, the adjustment range is larger;

[0038] S72: Adjust the posture according to the adjustment range.

[0039] Furthermore, the fine adjustment of the posture corresponding to the minimum cost value includes:

[0040] Based on the adjustment range of the previous iteration, take 1 / N of the adjustment range of the previous iteration to make fine adjustments;

[0041] Wherein N is an integer greater than 1.

[0042] The present invention has the following beneficial effects:

[0043] The present invention provides an automatic registration method for image point clouds without fixed targets, which is a universal calibration algorithm. The calibration work does not need to rely on specially arranged calibration objects and environments, and does not need to train the target object in advance. It uses the edge contour information in the 2D image and the 3D point cloud to perform automatic feature matching, and the 3D point cloud edge information is transformed into the 2D space by continuously adjusting the position and posture, and the nearest neighbor mutual traversal and local matching algorithms are used to construct feature point pairs, and the optimal matching point pairs are determined by first coarse matching and then fine matching, and then the corresponding relationship between the two is determined by nonlinear optimization, so as to complete the automatic registration of the image point cloud. The present invention has simple steps, reasonable design, convenient implementation, good use effect, good versatility, and can be effectively applied to the automatic registration process of point cloud data and images of different terrains, providing a more simple and easy method for the fusion registration of images and point clouds, and effectively shortening the working cycle. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of the method for automatic registration of image point clouds without fixed targets of the present invention;

[0045] Figure 2 This is the effect diagram of the point cloud 3D boundary point set effect;

[0046] Figure 3 This is the effect diagram of the 2D boundary point set of the image;

[0047] Figure 4 This is the effect picture after image point cloud fusion. DETAILED DESCRIPTION

[0048] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0049] The embodiment of the present invention provides an automatic registration method for image point clouds without fixed targets, such as Figure 1 As shown, the method includes:

[0050] S10: Acquire 2D image data and 3D point cloud data in the same monitoring area at the same time, and obtain camera calibration parameters.

[0051] In one of the examples, this step is implemented as follows:

[0052] S11: photographing the monitored area through a camera to obtain 2D image data.

[0053] S12: Scan the same monitoring area at the same time using a 3D point cloud scanning device to obtain 3D point cloud data.

[0054] The 3D point cloud scanning device can be a laser radar, which uses the laser radar to scan data at fixed points; the 3D point cloud scanning device can also be a drone, which uses the drone to scan data.

[0055] S13: Calibrate the camera to obtain camera calibration parameters; wherein the camera calibration parameters include camera focal length parameters and camera optical center parameters.

[0056] For example, the camera can be calibrated by Zhang Zhengyou calibration method to obtain the camera internal parameters, where fx and fy are the camera focal length parameters, and cx and cy are the camera optical center parameters.

[0057] S20: extract edge information of the 3D point cloud data and the 2D image data respectively, and obtain a point cloud 3D boundary point set and an image 2D boundary point set respectively.

[0058] Specifically, S20 includes:

[0059] S21: Perform voxel filtering and downsampling on the 3D point cloud data.

[0060] For example, the voxel size of the voxel filter downsampling can be set to 0.02m.

[0061] S22: traverse the 3D point cloud data after voxel filtering and downsampling, and perform stereo block processing according to a preset stereo block space size.

[0062] For example, the size of the three-dimensional block space may be 1.5m*1.5m*1.5m.

[0063] S23: In each three-dimensional block, plane recognition is performed by random sampling method, and the information of each plane is recorded. The intersection of two planes is obtained according to the intersection of plane normals, and the point cloud 3D boundary point set is constructed according to the intersection.

[0064] S23: In each stereo block, line segment estimation is performed by random sampling method, and the points where the line segments are located are added to the point cloud 3D boundary point set. The effect diagram of the point cloud 3D boundary point set is shown in Figure 2 shown.

[0065] S24: Perform edge detection on the 2D image data, and construct a 2D boundary point set of the image according to the detected contour boundary.

[0066] Edge detection can use the Canny algorithm to obtain the contour boundary. Before edge detection, the 2D image data can also be preprocessed, such as Gaussian filtering and noise reduction. Image 2D boundary point set Figure 3 shown.

[0067] S30: Taking the point cloud 3D boundary point set as input, and performing posture transformation on the point cloud 3D boundary point set according to a preset initial posture.

[0068] S40: Generate a perspective transformation matrix using camera calibration parameters, perform perspective projection transformation on the 3D boundary point set of the point cloud after the posture transformation, and generate a 2D boundary point set of the point cloud.

[0069] S50: Find common feature point pairs between the point cloud 2D boundary point set and the image 2D boundary point set, and obtain a common feature point pair list.

[0070] Among them, when a point cloud point in the point cloud 2D boundary point set and an image point in the image 2D boundary point set are neighboring points to each other, and the similarity between the local image at the point cloud point and the local image at the image point reaches a set similarity threshold, the point cloud point and the image point are regarded as a common feature point pair.

[0071] As an implementation method, this step includes:

[0072] S51: Taking the point cloud 2D boundary point set as the benchmark, traverse each point cloud point in the point cloud 2D boundary point set, and perform K nearest neighbor search on the traversed point cloud points and each image point in the image 2D boundary point set. According to the preset k value, obtain the image point with the first k minimum distance to the traversed point cloud point, and record it in the list imgPt_list.

[0073] S52: Based on the traversed point cloud points, a K nearest neighbor search is performed in the point cloud 2D boundary point set to obtain the point cloud points with the minimum distance to the traversed point cloud points, and the points are recorded in the list lidarPt_list.

[0074] S53: Based on imgPt_list, traverse each image point in the list imgPt_list, and reversely perform K nearest neighbor search on the point cloud points in the point cloud 2D boundary point set for the traversed image points to determine whether the image points and the point cloud points are neighboring points to each other, so as to confirm whether the same points are correlated with each other.

[0075] S54: Perform local feature matching on the local images of the area where the list imgPt_list and the list lidarPt_list are located, and calculate the similarity.

[0076] S55: When the image point and the point cloud point are neighboring points and the similarity reaches a set similarity threshold, the point cloud point and the image point are taken as a common feature point pair and added to the common feature point pair list pointPair.

[0077] The feature mutual matching algorithm designed by the present invention finds k points within a certain distance range as feature points. If the point cloud and image point set are neighboring points to each other and the local image feature matching approximation of the area where the feature point is located meets the expectations, it is considered to be a public feature point. A double confirmation mechanism is used to ensure the position consistency of the same features of the two data.

[0078] S60: Calculate a cost value according to the number of point pairs in the common feature point pair list and the number of 2D boundary point sets of the point cloud.

[0079] After the point cloud 2D boundary point set is completed, the cost value cost is calculated. The calculation formula is as follows:

[0080] cost = 1-(P1 / P)

[0081] Among them, P1 is the number of point pairs in the common feature point pair list pointPair, and P is the number of 2D boundary point sets of the point cloud.

[0082] S70: Adjust the posture according to the cost value, transform the posture of the point cloud 3D boundary point set according to the adjusted posture, and return to S40 for the next round of iteration; end the iteration after reaching the set number of iterations, output the minimum cost value and its corresponding posture, and execute S80.

[0083] The process of adjusting the posture is as follows:

[0084] S71: Calculate the difference between the cost value of this iteration and the cost value of the previous iteration, and determine the adjustment range of the posture according to the difference.

[0085] Among them, the smaller the difference is, the larger the adjustment range is.

[0086] S72: Adjust the posture according to the adjustment range.

[0087] The present invention guides the adjustment range of the posture according to the cost value, and the adjustment here is a rough adjustment. If the cost value is slightly different from the cost value of the previous iteration, the posture adjustment range is increased, and if the difference is large, the posture adjustment range is reduced. The posture is updated and the next round of iteration is entered until the iteration ends, and the minimum cost value and its corresponding posture are output.

[0088] S80: fine-tune the posture corresponding to the minimum cost value, transform the posture of the point cloud 3D boundary point set according to the finely adjusted posture, and return to S40 for the next round of iteration; end the iteration until the cost value reaches the set convergence, output the minimum cost value and its corresponding public feature point pair list, and execute S90.

[0089] According to the convergence of cost, the present invention makes a more refined adjustment based on the posture adjustment amplitude of the previous iteration. For example, 1 / N of the adjustment amplitude of the previous iteration is taken for refined adjustment; where N is an integer greater than 1, and can be preset as N=10. Then, based on the refined adjustment amplitude, it is iterated according to the above process to output the final common feature point pair.

[0090] S90: Perform nonlinear optimization based on the list of common feature point pairs corresponding to the minimum cost value, and output the final registration transformation relationship.

[0091] This step is used to accurately estimate and optimize the corresponding point set outputted in the end. Here, according to the number of point sets, the Ceres nonlinear least square method can be used to construct equations, solve the optimal matrix, and output the final numerically optimized registration transformation relationship. According to the registration transformation relationship, the image point cloud fusion can be realized, and the fusion effect is shown in the figure below. Figure 4 shown.

[0092] The present invention provides an automatic registration method for image point clouds without fixed targets, which is a universal calibration algorithm. The calibration work does not need to rely on specially arranged calibration objects and environments, and does not need to train the target object in advance. It uses the edge contour information in the 2D image and the 3D point cloud to perform automatic feature matching, and the 3D point cloud edge information is transformed into the 2D space by continuously adjusting the position and posture, and the nearest neighbor mutual traversal and local matching algorithms are used to construct feature point pairs, and the optimal matching point pairs are determined by first coarse matching and then fine matching, and then the corresponding relationship between the two is determined by nonlinear optimization, so as to complete the automatic registration of the image point cloud. The present invention has simple steps, reasonable design, convenient implementation, good use effect, good versatility, and can be effectively applied to the automatic registration process of point cloud data and images of different terrains, providing a more simple and easy method for the fusion registration of images and point clouds, and effectively shortening the working cycle.

[0093] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An automatic registration method for image point clouds without fixed targets, characterized in that: The method comprises: S10: Acquire 2D image data and 3D point cloud data in the same monitoring area at the same time, and obtain camera calibration parameters; S20: extracting edge information of the 3D point cloud data and the 2D image data respectively, and obtaining a point cloud 3D boundary point set and an image 2D boundary point set respectively; S30: taking the point cloud 3D boundary point set as input, and performing posture transformation on the point cloud 3D boundary point set according to a preset initial posture; S40: Generate a perspective transformation matrix using camera calibration parameters, perform perspective projection transformation on the point cloud 3D boundary point set after the posture transformation, and generate a point cloud 2D boundary point set; S50: Find common feature point pairs between the point cloud 2D boundary point set and the image 2D boundary point set, and obtain a list of common feature point pairs; Among them, when a point cloud point in the point cloud 2D boundary point set and an image point in the image 2D boundary point set are neighboring points to each other, and the similarity between the local image at the point cloud point and the local image at the image point reaches a set similarity threshold, the point cloud point and the image point are regarded as a common feature point pair; S60: Calculating a cost value according to the number of point pairs in the common feature point pair list and the number of 2D boundary point sets of the point cloud; S70: adjusting the posture according to the cost value, performing posture transformation on the point cloud 3D boundary point set according to the adjusted posture, and returning to S40 for the next round of iteration; the iteration ends after reaching the set number of iterations, outputting the minimum cost value and its corresponding posture, and executing S80; S80: finely adjust the posture corresponding to the minimum cost value, transform the posture of the point cloud 3D boundary point set according to the finely adjusted posture, and return to S40 for the next round of iteration; the iteration ends when the cost value reaches the set convergence, outputs the minimum cost value and its corresponding public feature point pair list, and executes S90; S90: Perform nonlinear optimization based on the list of common feature point pairs corresponding to the minimum cost value, and output the final registration transformation relationship.

2. The method for automatic registration of image point clouds without fixed targets according to claim 1, characterized in that: The S10 includes: S11: photographing the monitoring area through a camera to obtain 2D image data; S12: Scanning the same monitoring area at the same time by means of a 3D point cloud scanning device to obtain 3D point cloud data; S13: Calibrate the camera to obtain camera calibration parameters; wherein the camera calibration parameters include camera focal length parameters and camera optical center parameters.

3. The method for automatic registration of image point clouds without fixed targets according to claim 2, characterized in that: The S20 includes: S21: performing voxel filtering and downsampling on the 3D point cloud data; S22: traversing the 3D point cloud data after voxel filtering and downsampling, and performing stereo block processing according to a preset stereo block space size; S23: In each three-dimensional block, plane recognition is performed by random sampling, and the intersection of two planes is obtained according to the intersection of plane normals, and a point cloud 3D boundary point set is constructed according to the intersection; S23: In each stereo block, line segment estimation is performed by random sampling method, and the points where the line segments are located are added to the 3D boundary point set of the point cloud; S24: Perform edge detection on the 2D image data, and construct an image 2D boundary point set according to the detected contour boundary.

4. The method for automatic registration of image point clouds without fixed targets according to any one of claims 1 to 3, characterized in that: The S50 includes: S51: traverse each point cloud point in the point cloud 2D boundary point set, and perform K nearest neighbor search on the traversed point cloud point and each image point in the image 2D boundary point set, obtain the image points with the minimum distance to the traversed point cloud point, and record them in the list imgPt_list; S52: Based on the traversed point cloud point as a reference, a K nearest neighbor search is performed in the point cloud 2D boundary point set to obtain the point cloud point with the minimum distance to the traversed point cloud point, and the point cloud point is recorded in the list lidarPt_list; S53: traverse each image point in the list imgPt_list, and reversely perform a K-nearest neighbor search on the point cloud points in the point cloud 2D boundary point set using the traversed image points to determine whether the image point and the point cloud point are neighboring points to each other; S54: performing local feature matching on the local images of the area where the list imgPt_list and the list lidarPt_list are located, and calculating the similarity; S55: When the image point and the point cloud point are neighboring points and the similarity reaches a set similarity threshold, the point cloud point and the image point are taken as a common feature point pair and added to a common feature point pair list pointPair.

5. The method for automatic registration of image point clouds without fixed targets according to claim 4, characterized in that: The calculation formula of the cost value is as follows: cost = 1-(P1 / P) Among them, P1 is the number of point pairs in the common feature point pair list pointPair, and P is the number of 2D boundary point sets of the point cloud.

6. The method for automatic registration of image point clouds without fixed targets according to claim 5, characterized in that: The adjusting the posture according to the cost value includes: S71: Calculate the difference between the cost value of this iteration and the cost value of the previous iteration, and determine the adjustment range of the posture according to the difference; Wherein, when the difference is smaller, the adjustment range is larger; S72: Adjust the posture according to the adjustment range.

7. The method for automatic registration of image point clouds without fixed targets according to claim 6, characterized in that: The fine adjustment of the posture corresponding to the minimum cost value includes: Based on the adjustment range of the previous iteration, take 1 / N of the adjustment range of the previous iteration to make fine adjustments; Wherein N is an integer greater than 1.

Images