Infrared temperature image and high-precision point cloud registration method

Abstract

The invention relates to the technical field of matching and fusion of two-dimensional image data and three-dimensional point cloud data, in particular to an infrared temperature image and high-precision point cloud registration method, which comprises the following steps of: acquiring data by using multiple sensors, namely acquiring dense point cloud data by using a laser radar, acquiring image data by using a camera sensor and calibrating; calculating a spatial transformation matrix from image data to spatial coordinates through sensor pose data and feature matching, and further forming sparse point cloud data; then using an optimized point cloud registration method, namely, based on an optimized four-point base algorithm (DA-4PCS), to carry out coarse registration, and then using an optimized nearest neighbor iterative algorithm (RC-ICP) to carry out fine registration; and finally, for the problem that one pixel is matched with multiple point cloud data, providing K neighborhood retrieval based on normal vector constraint to realize expansion from sparse point cloud features to dense point cloud, and finally obtaining a two-dimensional and three-dimensional data registration fusion result.

Description

Technical Field

[0001] The present invention relates to the technical field of matching and fusing two-dimensional image data and three-dimensional point cloud data, and in particular to a method for matching infrared temperature images with high-precision point clouds. Background Art

[0002] In recent years, with the continuous improvement of LiDAR equipment, the fusion technology of 3D point cloud and visible light image has become a research hotspot. 3D point cloud has rich spatial information and is not easily affected by external light, but the high-precision point cloud obtained by LiDAR lacks real texture and color information. On the contrary, optical images have real texture and color information and high resolution, but they are greatly affected by light during the acquisition process, and spatial distance information cannot be obtained from them. Therefore, quickly and accurately fusing visible light images with 3D point cloud images can greatly give play to the advantages of both. The fusion results of the two can realize the true color expression of the 3D world, and the fusion visualization results are of great significance in many fields.

[0003] In general, the current registration methods are mainly: the first method is to realize the projection of two-dimensional data to three-dimensional data based on spatial transformation. The feature matching in spatial transformation can use the calibration plate during shooting for feature matching, or use the image and the part with features in the space to match; the second method is to use photogrammetry and other methods to project the obtained image data into space to generate color point cloud data, and then use the point cloud registration method to complete the mapping of feature information; the third method is to use deep learning methods to annotate and register image data and point cloud data, and finally complete feature mapping and matching in the potential feature space. These methods have their own characteristics and advantages and disadvantages.

[0004] From the above studies, we can see that there are many types of methods for aligning two-dimensional image data with point clouds, and each of these methods has its own advantages and disadvantages.

[0005] The second method is the current mainstream registration and fusion method. The second method mainly uses a multi-sensor fusion method to fuse the visible light point cloud constructed based on binocular vision and the dense point cloud constructed by laser scanning. This method mainly uses photogrammetry to project image data into spatial coordinates to form a sparse point cloud, and then registers it with the fine point cloud scanned by the lidar. It should be noted that the point cloud generated by the image is usually much sparser than the point cloud scanned by the lidar, so it is necessary to perform point cloud neighborhood retrieval and feature scale restoration, while taking into account both interpretability and the accuracy of registration and fusion.

[0006] However, the applicant found that when the second method of fusing and registering the visible light point cloud constructed based on binocular vision and the dense point cloud constructed by laser scanning was used, the overall registration algorithm ran slowly and had poor robustness, and there was a problem of registering multiple point clouds with one pixel.

[0007] To this end, the present invention will propose an infrared temperature image and high-precision point cloud registration method based on the second method to solve the problems of slow registration speed, low registration accuracy and the existence of one pixel registering multiple point clouds between point clouds. Summary of the invention

[0008] In view of this, the purpose of the present invention is to propose an infrared temperature image and a high-precision point cloud registration method to solve the problems of slow running speed and poor robustness of the overall registration algorithm and the existence of one pixel registering multiple point clouds when the existing visible light point cloud constructed based on binocular vision and the dense point cloud constructed by laser scanning are fused and registered.

[0009] Based on the above purpose, the present invention provides an infrared temperature image and a high-precision point cloud registration method, comprising the following steps:

[0010] S1, use LiDAR to obtain dense point cloud data;

[0011] S2, using sensor calibration to obtain color sparse point cloud data;

[0012] S3, registering the sparse point cloud data and the dense point cloud data;

[0013] In step S3, the registration of the sparse point cloud data and the dense point cloud data includes coarse registration and fine registration;

[0014] S3.1, optimizing the four-point base algorithm, and performing coarse registration based on the optimized four-point base algorithm, i.e., DA-4PCS;

[0015] S3.2, based on the position of the four-point basis of 4PCS coarse registration, a regional correspondence-based ICP algorithm, namely RC-ICP, is proposed for fine registration;

[0016] S4, K-neighborhood retrieval based on normal vector constraints realizes the expansion of sparse point cloud RGB color features to dense point cloud, and obtains dense point cloud after feature registration;

[0017] S5. Obtain the result of two-dimensional and three-dimensional data registration and fusion.

[0018] Preferably, step S2 also includes the following steps:

[0019] S2.1, use the camera sensor to obtain image data and perform accurate calibration;

[0020] S2.2, obtaining the position and attitude information of the sensor;

[0021] S2.3, calculate the spatial transformation matrix based on the sensor posture information and image overlap;

[0022] S2.4. Determine the mapping relationship between pixel points and spatial coordinates to generate a sparse point cloud.

[0023] Preferably, in step S3.1, the optimization of the four-point basis algorithm includes two directions;

[0024] S3.11, one is to reduce the iteration frequency of each stage;

[0025] S3.12. The second is to improve the search algorithm within each iteration.

[0026] Preferably, in S1.11, reducing the iteration frequency of each stage can be achieved by selecting an asymmetric four-point basis. By selecting an asymmetric four-point basis, the problem of doubling the number of candidate bases to be verified due to the symmetric four-point basis can be avoided, thereby fundamentally reducing the number of candidate congruent sets that need to be verified in the CSV stage, thereby reducing the number of iterations required for verification in this stage.

[0027] Preferably, in S3.11, when selecting an asymmetric four-point basis, it is necessary to avoid the symmetric four-point basis, and the steps of determining the symmetric four-point basis are as follows:

[0028] S3.111. Divide the symmetric four-point basis {a, b, c, d} into two categories:

[0029] S3.112, the first type of symmetry, the symmetry axis of the symmetric four-point basis passes through the intersection point, the two line segments are axially symmetrical, and have the characteristics of equal length of the line segments and equal or equal addition of the intersection ratio of the line segments to 1;

[0030] Assuming that the intersection of the two line segments is e, and assuming that the two baselines are divided into four segments s1, s2, s3, and s4 by the intersection e, the formula is expressed as follows:

[0031] {(s i =s j )∪(s i +s j =d1)}∩(d1=d2),(i,j∈1,2,3,4),i≠j;

[0032] S3.113, the second type of symmetry, one of the line segments of the symmetric four-point basis acts as the symmetry axis, the line segment intersection ratio is equal to 0.5 and the line segments are perpendicular to each other;

[0033] Assuming that the intersection of the two line segments is e, and assuming that the two baselines are divided into four segments by the intersection e, the formula is expressed as follows:

[0034] {(si =0.5d1)∪(s i =0.5d2)}∩(d1⊥d2), (i,j∈1,2,3,4),i≠j.

[0035] Among them, d1 and d2 are the lengths of the base midline segment.

[0036] Preferably, in S3.12, improving the search algorithm in each iteration can be achieved by dynamically increasing the length of the line segment in the basis;

[0037] The longer the line segment in the basis, the fewer line segments of the same length are extracted in the point cloud. Therefore, increasing the distance of the line segment in the basis can reduce the probability of its appearance, thereby reducing the number of congruent sets that need to be verified in the CSV stage. In addition, using a wider range of four points in the overlapping area to explore the transformation matrix can enhance the robustness of the algorithm.

[0038] Preferably, in S3.12, determining the possible length d0 of the line segment in the overlapping area comprises the following steps:

[0039] S3.121. The possible length d0 of the line segment in the overlapping area is obtained by using the largest common point set (LCP) updated in the iterative process, the LCP value lcp, and the length of the corresponding line segment in the base at that time. To calculate, for a given point cloud P, its diameter is diameterP, and the length d0 is defined as follows:

[0040]

[0041] When selecting a basis, the lengths of the line segments d1 and d2 in the basis must satisfy the following formula:

[0042] (d1≥d0)∪(d2≥d0)

[0043] Among them, α is the scale factor, the value is 0.85, and lcp can represent the current overlap ratio between the two point clouds;

[0044] When lcp is a true value, it means that the current four-point basis is in the point cloud overlapping area. The value represents the maximum distance currently calculated for the line segments in the overlapping area, and is multiplied by the scale factor so that the distance of the line segments can fluctuate within a certain range to find the optimal basis;

[0045] When lcp is less than When , it means that the group basis is not in the overlapping area, and the wrong match obtains a small value of lcp. This will cause the selection of the basis to not fall into the overlapping area, and thus the correct transformation cannot be solved. At this time, d0=lcp*diameterP can calculate the distance reached by the line segment in the overlapping area.

[0046] Preferably, in S3.2, the following steps are included:

[0047] S3.21, the four-point basis forms a shape similar to a quadrilateral, and the RC-ICP algorithm takes each point of the four-point basis as the center and extracts points within a certain distance range and facing the direction of the quadrilateral;

[0048] S3.22, dividing the point cloud into four independent small blocks, and dividing the two point clouds at the same time to obtain four pairs of corresponding partitions;

[0049] S3.23, find the closest point in each pair of partitioned point clouds, put all the point pairs together to solve the best transformation matrix;

[0050] S3.24, take point clouds P and Q as an example, and for the convenience of representation, point clouds P and Q are set to be planar;

[0051] S3.25. For point clouds P and Q, the matching basis of the rough registration result is {a, b, c, d} and {a′, b′, c′, c′}, and line segments ab and cd are line segments with non-coterminal points in the basis of point cloud P. Then the region definition formula of point cloud P at point α is as follows:

[0052]

[0053] In the formula, β is a scaling factor, which is generally taken as 0.5. Similarly, the nearby areas of other points of the basis and the corresponding points of Q are extracted to obtain four pairs of corresponding areas of the point cloud.

[0054] Preferably, in S4, by introducing the information of the normal vector change gradient, the normal vector change trend can be found, which is convenient for increasing the weight factor for the edge local feature, which specifically includes the following steps:

[0055] S4.1. First, solve the bounding box volume V and the number of point clouds Point of the dense point cloud data num , the local density of point cloud data is solved by the following formula;

[0056]

[0057] S4.2. Find the point P to be retrieved in the sparse point cloud i , and find the closest point P j , and calculate its interval bounding box volume V', the formula is as follows:

[0058]

[0059] Among them, l is to P j distance;

[0060] The number of iteration points is calculated based on local features. The formula is as follows:

[0061] Cluster num =density×V';

[0062] For point P i , you need to retrieve its Cluster in the dense point cloud num points, and P i Assign the RGB value to this Cluster num points in a dense point cloud;

[0063] S4.3. Retrieve P i The 4 nearest points in the dense point cloud are selected as candidate points, and the points P i The point with the largest normal vector difference is taken as the cluster point Q′ i , and then cluster point Q′ i As a new retrieval point, search for other unclustered points in the dense point cloud according to the above method until the retrieval is complete. P i The RGB value is assigned to this Cluster num This is for P i The above iterations are continuously repeated to complete such clustering for all points in the sparse point cloud, so as to realize the expansion of sparse point cloud features to dense point cloud features.

[0064] The beneficial effects of the present invention are as follows:

[0065] 1. Selecting an asymmetric four-point basis belongs to the idea of ​​reducing the iteration frequency of each stage. By selecting an asymmetric four-point basis, the problem of doubling the number of candidate bases to be verified due to the symmetric four-point basis can be avoided. The number of candidate congruent sets that need to be verified in the CSV (congruent set verification) stage is reduced from the root, which also reduces the number of iterations required for verification in this stage, thereby reducing the iteration frequency of the entire alignment process in this stage, which is to improve the algorithm speed from the perspective of reducing the number of iterations.

[0066] Second, the strategy of dynamically increasing the length of the line segment in the basis is based on the principle of line segment length correlation. By reasonably adjusting the length of the line segment in the basis, it is more targeted and efficient to find a suitable four-point basis to determine the transformation matrix when verifying the congruence set in the CSV stage. In essence, it changes the strategy for selecting the four-point basis in each iteration (especially when it comes to CSV verification iterations), optimizes the algorithm logic for searching for a suitable four-point basis, and improves the search algorithm in each iteration, thereby improving the speed and robustness of the overall registration algorithm.

[0067] 3. Since the 4PCS algorithm has unique advantages in the point cloud registration process, its rough registration results can not only move the overlapping parts of the two point clouds to a similar position, but also obtain approximately accurate information on the four pairs of nearest points. Based on the position of the four-point basis of the 4PCS rough registration, an ICP algorithm based on regional correspondence, RC-ICP, is proposed. The RC-ICP partitioning method can limit the search range of the nearest point and speed up the speed. The matching point pairs found in different partitions are calculated as a rigid body transformation to avoid mismatching and local optimality of a single partition. Limiting the search range of point pairs can prevent the divergence of the overall offset in multiple iterations.

[0068] Fourth, by proposing a dynamic neighborhood retrieval method for edge feature extraction and introducing the information of the normal vector change gradient, we can find the trend of large normal vector changes and increase the weight factors for local edge features. It can be seen that since this method increases the normal vector constraint, points close to the normal vector of the point to be retrieved are filtered out, which can better characterize the local features of point clouds with complex local structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0070] Figure 1 is an overall flow chart of an embodiment of the present invention;

[0071] Figure 2 Schematic diagram of the planar structure of the first type of symmetrical four-point basis according to an embodiment of the present invention;

[0072] Figure 3 Schematic diagram of the planar structure of the second type of symmetrical four-point basis according to an embodiment of the present invention;

[0073] Figure 4 It is a planar schematic diagram of the partition pairing of point cloud P and point cloud Q according to an embodiment of the present invention;

[0074] Figure 5 A flowchart of a neighborhood retrieval method based on edge feature extraction according to an embodiment of the present invention;

[0075] Figure 6 A point cloud data graph retrieved by k-neighborhood according to an embodiment of the present invention;

[0076] Figure 7 A point cloud data graph retrieved through a spherical neighborhood according to an embodiment of the present invention;

[0077] Figure 8 This is a point cloud data graph of dynamic neighborhood retrieval for edge feature extraction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0078] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0079] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0080] like Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 As shown, the infrared temperature image and high-precision point cloud registration method includes the following steps:

[0081] S1, use LiDAR to obtain dense point cloud data;

[0082] S2, using sensor calibration to obtain color sparse point cloud data;

[0083] Step S2 also includes the following steps:

[0084] S2.1, use the camera sensor to obtain image data and perform accurate calibration;

[0085] S2.2, obtaining the position and attitude information of the sensor;

[0086] S2.3, calculate the spatial transformation matrix based on the sensor posture information and image overlap;

[0087] S2.4. Determine the mapping relationship between pixel points and spatial coordinates to generate a sparse point cloud.

[0088] The purpose of registration is to find the correspondence between the image and the point cloud. It is essentially a calibration process. First, you need to find the corresponding point set in the point cloud and the image. The commonly used features for finding matching point sets are: point, line, and surface features. Different types of point clouds and images select appropriate feature primitives for registration according to the characteristics of their data. In the registration between the LiDAR point cloud and the image, different types of feature primitives will be selected according to the characteristics of the scene where the point cloud is acquired. In the LiDAR point cloud, compared with point features, straight lines are easier to extract and distinguish, and the spatial geometric characteristics between line features are more constrained. At the same time, the mathematical model of line features is more robust than that of point features.

[0089] S3, registering the sparse point cloud data and the dense point cloud data;

[0090] In step S3, the registration of the sparse point cloud data and the dense point cloud data includes coarse registration and fine registration;

[0091] In the rough registration stage, the four-point basis space structure of the 4PCS algorithm was improved, and the DA-4PCS (Dynamic and Asymmetric 4PCS) algorithm was proposed.

[0092] S3.1, optimizing the four-point base algorithm, and performing coarse registration based on the optimized four-point base algorithm, namely DA-4PCS;

[0093] like Figure 2 , Figure 3 As shown in the figure, when searching for the congruent four-point sets of the two types of symmetric four-point bases {a, b, c, d} on the plane, given the symmetry of the four-point structure itself, the four-point set obtained by flipping it along the symmetry axis is still congruent to the original basis. However, for point cloud registration, only one of these two situations is the correct solution. In this way, each candidate four-point combination in the candidate congruent set will have a four-point combination that has been flipped itself, which doubles the number of candidate bases to be verified. This symmetry problem also exists in the generalized four-point basis.

[0094] In the CSV stage, the number of candidate congruence sets to be verified is on average twice that of ordinary asymmetric structures. The operation of checking whether the four-point basis is symmetric can be placed before the point pair extraction. Since the time spent on selecting the basis is almost negligible compared to the total registration time, discarding this set of basis and carrying out the next iteration will not increase the total registration time.

[0095] In step S3.1, the optimization of the four-point basis algorithm includes two directions;

[0096] S3.11, one is to reduce the iteration frequency of each stage;

[0097] In S1.11, reducing the iteration frequency of each stage can be achieved by selecting an asymmetric four-point basis. By selecting an asymmetric four-point basis, the problem of doubling the number of candidate bases to be verified due to the symmetric four-point basis can be avoided, and the number of candidate congruent sets that need to be verified in the CSV stage is fundamentally reduced, thereby reducing the number of iterations required for verification in this stage.

[0098] In S3.11, when selecting an asymmetric four-point basis, it is necessary to avoid the symmetric four-point basis. The steps to determine the symmetric four-point basis are as follows:

[0099] S3.111. Divide the symmetric four-point basis {a, b, c, d} into two categories:

[0100] like Figure 2 As shown, S3.112, the first type of symmetry, the symmetry axis of the symmetric four-point basis passes through the intersection point, the two line segments are axially symmetrical, and have the characteristics of equal line segment lengths and equal or summed intersection ratios of the line segments equal to 1;

[0101] Assuming that the intersection of the two line segments is e, and assuming that the two baselines are divided into four segments s1, s2, s3, and s4 by the intersection e, the formula is expressed as follows:

[0102] {(s i =s j )∪(s i +s j =d1)}∩(d1=d2),(i,j∈1,2,3,4),i≠j;

[0103] like Figure 3 As shown, S3.113, in the second type of symmetry, one of the line segments of the symmetric four-point basis acts as the symmetry axis, satisfying the line segment intersection ratio equal to 0.5 and the line segments are perpendicular to each other;

[0104] Assuming that the intersection of the two line segments is e, and assuming that the two baselines are divided into four segments by the intersection e, the formula is expressed as follows:

[0105] {(s i =0.5d1)∪(s i =0.5d2)}∩(d1⊥d2), (i,j∈1,2,3,4),i≠j.

[0106] Among them, d1 and d2 are the lengths of the base midline segment.

[0107] S3.12. The second is to improve the search algorithm within each iteration.

[0108] In S3.12, the improvement of the search algorithm within each iteration can be achieved by dynamically increasing the length of the line segment in the basis;

[0109] The longer the line segment in the basis, the fewer line segments of the same length are extracted in the point cloud. Therefore, increasing the distance of the line segment in the basis can reduce the probability of its appearance, thereby reducing the number of congruent sets that need to be verified in the CSV stage. In addition, using a wider range of four points in the overlapping area to explore the transformation matrix can enhance the robustness of the algorithm.

[0110] In S3.12, determining the possible length d0 of the line segment in the overlap region comprises the following steps:

[0111] S3.121. The possible length d0 of the line segment in the overlapping area is obtained by using the largest common point set (LCP) updated in the iterative process, the LCP value is lcp, and the length of the corresponding line segment in the base at that time To calculate, for a given point cloud P, its diameter is diameterP, and the length d0 is defined as follows:

[0112]

[0113] When selecting a basis, the lengths of the line segments d1 and d2 in the basis must satisfy the following formula:

[0114] (d1≥d0)∪(d2≥d0)

[0115] Among them, α is the scale factor, the value is 0.85, and lcp can represent the current overlap ratio between the two point clouds;

[0116] When lcp is a true value, it means that the current four-point basis is in the point cloud overlapping area. The value represents the maximum distance currently calculated for the line segments in the overlapping area, and is multiplied by the scale factor so that the distance of the line segments can fluctuate within a certain range to find the optimal basis;

[0117] When lcp is less than When , it means that the group basis is not in the overlapping area, and the wrong match obtains a small value of lcp. This will cause the selection of the basis to not fall into the overlapping area, and thus the correct transformation cannot be solved. At this time, d0=lcp*diameterP can calculate the distance reached by the line segment in the overlapping area.

[0118] The strategy of dynamically increasing the length of the line segment in the basis is based on the principle of line segment length correlation. By reasonably adjusting the length of the line segment in the basis, it is more targeted and efficient to find a suitable four-point basis to determine the transformation matrix when verifying the congruence set in the CSV stage. In essence, it changes the strategy for selecting the four-point basis in each iteration (especially when it comes to CSV verification iterations), optimizes the algorithm logic for searching for a suitable four-point basis, and improves the search algorithm in each iteration, thereby improving the speed and robustness of the overall registration algorithm.

[0119] In the fine registration stage, the ICP fine registration algorithm is improved based on the coarse registration results of the 4PCS algorithm to form the RC-ICP (Region Corresponding ICP) algorithm.

[0120] The 4PCS algorithm has unique advantages in the point cloud registration process. Its rough registration result can not only move the overlapping parts of the two point clouds to similar positions, but also obtain approximately accurate information of the four pairs of nearest points.

[0121] S3.2, based on the position of the four-point basis of 4PCS coarse registration, a regional correspondence-based ICP algorithm, namely RC-ICP, is proposed for fine registration;

[0122] In S3.2, the following steps are included:

[0123] like Figure 4 As shown in S3.21, according to the continuity of the object surface, the two point clouds near the corresponding four points also have overlapping areas, and the four-point base forms a shape similar to a quadrilateral. The RC-ICP algorithm takes each point of the four-point base as the center and extracts points within a certain distance range and facing the direction of the quadrilateral;

[0124] S3.22, dividing the point cloud into four independent small blocks, and dividing the two point clouds at the same time to obtain four pairs of corresponding partitions;

[0125] S3.23, find the closest point in each pair of partitioned point clouds, put all the point pairs together to solve the best transformation matrix;

[0126] S3.24, take point clouds P and Q as an example, and for the convenience of representation, point clouds P and Q are set to be planar;

[0127] S3.25. For point clouds P and Q, the matching basis of the rough registration result is {a, b, c, d} and {a′, b′, c′, c′}, and line segments ab and cd are line segments with non-coterminal points in the basis of point cloud P. Then the region definition formula of point cloud P at point α is as follows:

[0128]

[0129] In the formula, β is a scaling factor, which is generally taken as 0.5. Similarly, the nearby areas of other points of the basis and the corresponding points of Q are extracted to obtain four pairs of corresponding areas of the point cloud.

[0130] The essence of fine registration is to use the four-point basis method of coarse registration to divide the entire point cloud P and point cloud Q into four parts, namely P1, P2, P3, P4, Q1, Q2, Q3, and Q4, according to the point basis and baseline segment, and then align these eight parts separately. The above ab and cd are usually the longest baseline segments, which can be understood as the two groups of line segments with the farthest distance between the point clouds;

[0131] The region can be divided by the formula in S3.25. For example, if there is a baseline ab for point cloud P, then the baseline a′b′ is used for point cloud Q. According to the above formula, any point p in point cloud P that satisfies the above formula (that is, the line segment length of the ap line is β times the length of ab, and the angle is less than 90°) is considered to be in the P1 part. Similarly, p′ that satisfies the above formula in Q is the point in the Q1 part. Then, P1 and Q1 are aligned. Similarly, points in P2, P3, P4, Q2, Q3, and Q4 can be found and then aligned.

[0132] Among them, β is just a scaling factor. In fact, if you want to cover the entire point cloud, β should be However, because the rough registration has been completed before, and in the rough registration, the point cloud registration accuracy in the middle area of ​​the baseline segment is the highest, so β ​​takes a smaller number, 0.5 in the above formula, which can reduce the number of point clouds for subsequent fine registration while ensuring accuracy.

[0133] By partitioning, the search range of the nearest point can be limited and the speed can be increased. The matching point pairs found in different partitions are used together to calculate the rigid body transformation to avoid mismatch and local optimality in a single partition. Limiting the search range of point pairs can prevent the divergence of the overall offset after multiple iterations.

[0134] The registration of sparse point cloud data and dense point cloud data is completed through the above coarse and fine registration. However, the sparse point cloud is a point cloud with both spatial coordinates and RGB color features, while the dense point cloud only contains spatial coordinates. Therefore, the next task is how to accurately align the RGB color of the sparse point cloud to each point in the dense point cloud data.

[0135] The idea is to find a point in the sparse point cloud, perform neighborhood search on the dense point cloud, and uniformly select the RGB color information of the sparse point cloud for data that meets a specific range. The specific feature expansion process is as follows: Figure 5 As shown;

[0136] S4, K-neighborhood retrieval based on normal vector constraints realizes the expansion of sparse point cloud RGB color features to dense point cloud, and obtains dense point cloud after feature registration;

[0137] In S4, by introducing the information of the normal vector change gradient, the normal vector change trend can be found, which is convenient for increasing the weight factor for the local edge features, which specifically includes the following steps:

[0138] S4.1. First, solve the bounding box volume V and the number of point clouds Point of the dense point cloud data num , the local density of point cloud data is solved by the following formula;

[0139]

[0140] S4.2. Find the point P to be retrieved in the sparse point cloud i , and find the closest point P j , and calculate its interval bounding box volume V', the formula is as follows:

[0141]

[0142] Among them, l is to P j distance;

[0143] The number of iteration points is calculated based on local features. The formula is as follows:

[0144] Cluster num =density×V';

[0145] For point P i , you need to retrieve its Cluster in the dense point cloud num points, and P i Assign the RGB value to this Cluster num points in a dense point cloud;

[0146] S4.3. Retrieve P i The 4 nearest points in the dense point cloud are selected as candidate points, and the points P i The point with the largest normal vector difference is taken as the cluster point Q′ i , and then cluster point Q′ i As a new retrieval point, search for other unclustered points in the dense point cloud according to the above method until the retrieval is complete. P i The RGB value is assigned to this Cluster num This is for P i The above iterations are continuously repeated to complete such clustering for all points in the sparse point cloud, so as to realize the expansion of sparse point cloud features to dense point cloud features.

[0147] Assuming that the dense point cloud is Q and the sparse point cloud is P, then it is obvious that the point cloud density of Q is greater than that of P. For example, within a certain spatial range (assuming 10 cubic meters), Q has 100 points and P has 10 points, then the density of Q is 10. Next, P is retrieved and expanded.

[0148] First, randomly select a point p1 in P, retrieve the point p2 closest to p1 in P, and calculate the bounding box volume based on these two points. Assuming that the point cloud is evenly distributed, the size of the bounding box constructed by these two points is 1 cubic meter. num That is 10, that is, for point p1, you need to retrieve its 10 points in the point cloud Q and assign the RGB value of p1 to these 10 points in Q.

[0149] Next, we search in Q. First, we find the four nearest points Q1, Q2, Q3, and Q4 in Q as candidate points based on p1. Then, we calculate the normal vectors from p1 to these four points respectively, and select the point with the largest normal vector difference as the cluster point Q′1. Then, we use Q′1 as the new search point and search for other unclustered points in Q according to the above method until we find Q. 10 ′, then the RGB values ​​of p1 are assigned to these 10 points. This is the clustering for p1, and then all the points in the sparse point cloud P are clustered in this way.

[0150] During the process, we need to pay attention to repeated searches. For example, when searching for p2, there is a point Q1′ that has been clustered by p1. Then we need to calculate the distances pq1 and pq2 from this point Q1′ to p1 and p2. If pq1>pq2, it will be classified into the clustering range of p2. Otherwise, Q1′ will be retained in the clustering range of p1.

[0151] in Figure 6 is k-neighbor search, Figure 7 For ball neighborhood search, Figure 8 This is the dynamic neighborhood retrieval for edge feature extraction proposed by this method. It can be seen that since this method adds normal vector constraints, points close to the normal vector of the point to be retrieved are filtered out, which can better characterize the local features of point clouds with complex local structures. For example, in the insulator point cloud data in the above figure, this method can better retrieve the neighborhood edge points of the point to be retrieved, and better extract the local features of the point cloud data.

[0152] S5. Obtain the result of two-dimensional and three-dimensional data registration and fusion.

[0153] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention is limited to these examples. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0154] The embodiments of the present invention are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. Infrared temperature image and high-precision point cloud registration method, characterized in that: The following steps are involved: S1, use LiDAR to obtain dense point cloud data; S2, using sensor calibration to obtain color sparse point cloud data; S3, registering the sparse point cloud data and the dense point cloud data; In step S3, the registration of the sparse point cloud data and the dense point cloud data includes coarse registration and fine registration; S3.1, optimizing the four-point base algorithm, and performing coarse registration based on the optimized four-point base algorithm, namely DA-4PCS; S3.2, based on the position of the four-point basis of 4PCS coarse registration, a regional correspondence-based ICP algorithm, namely RC-ICP, is proposed for fine registration; S4, K-neighborhood retrieval based on normal vector constraints realizes the expansion of sparse point cloud RGB color features to dense point cloud; S5. Obtain the result of two-dimensional and three-dimensional data registration and fusion.

2. The infrared temperature image and high-precision point cloud registration method according to claim 1 is characterized in that: Step S2 also includes the following steps: S2.1, use the camera sensor to obtain image data and perform accurate calibration; S2.2, obtaining the position and attitude information of the sensor; S2.3, calculate the spatial transformation matrix based on the sensor posture information and image overlap; S2.

4. Determine the mapping relationship between pixel points and spatial coordinates to generate a sparse point cloud.

3. The infrared temperature image and high-precision point cloud registration method according to claim 1, characterized in that: In step S3.1, the optimization of the four-point basis algorithm includes two directions; S3.11, one is to reduce the iteration frequency of each stage; S3.

12. The second is to improve the search algorithm within each iteration.

4. The infrared temperature image and high-precision point cloud registration method according to claim 3 is characterized in that: In S1.11, reducing the iteration frequency of each stage can be achieved by selecting an asymmetric four-point basis. By selecting an asymmetric four-point basis, the problem of doubling the number of candidate bases to be verified due to the symmetric four-point basis can be avoided, and the number of candidate congruent sets that need to be verified in the CSV stage is fundamentally reduced, thereby reducing the number of iterations required for verification in this stage.

5. The infrared temperature image and high-precision point cloud registration method according to claim 4 is characterized in that: In S3.11, when selecting an asymmetric four-point basis, it is necessary to avoid the symmetric four-point basis. The steps to determine the symmetric four-point basis are as follows: S3.

111. Divide the symmetric four-point basis {a, b, c, d} into two categories: S3.112, the first type of symmetry, the symmetry axis of the symmetric four-point basis passes through the intersection point, the two line segments are axially symmetrical, and have the characteristics of equal length of the line segments and equal or equal addition of the intersection ratio of the line segments to 1; Assuming that the intersection of the two line segments is e, and assuming that the two baselines are divided into four segments s1, s2, s3, and s4 by the intersection e, the formula is expressed as follows: {(s i =s j )∪(s i +s j =d1)}∩(d1=d2),(i,j∈1,2,3,4),i≠j; S3.113, the second type of symmetry, one of the line segments of the symmetric four-point basis acts as the symmetry axis, the line segment intersection ratio is equal to 0.5 and the line segments are perpendicular to each other; Assuming that the intersection point of the two line segments is e, and assuming that the two baselines are divided into four segments by the intersection point e, the formula is expressed as follows: {(s i =0.5d1)∪(s i =0.5d2)}∩(d1⊥d2),(i,j∈1,2,3,4),i≠j; Among them, d1 and d2 are the lengths of the base midline segment.

6. The infrared temperature image and high-precision point cloud registration method according to claim 3 is characterized in that: In S3.12, the improvement of the search algorithm within each iteration can be achieved by dynamically increasing the length of the line segment in the basis; The longer the line segment in the basis, the fewer line segments of the same length are extracted in the point cloud. Therefore, increasing the distance of the line segment in the basis can reduce the probability of its appearance, thereby reducing the number of congruent sets that need to be verified in the CSV stage. In addition, using a wider range of four points in the overlapping area to explore the transformation matrix can enhance the robustness of the algorithm.

7. The infrared temperature image and high-precision point cloud registration method according to claim 6, characterized in that: In S3.12, determining the possible length d0 of the line segment in the overlap region comprises the following steps: S3.

121. The possible length d0 of the line segment in the overlapping area is obtained by using the LCP value lcp updated in the iterative process, where LCP is the largest common point set, and the length of the corresponding line segment in the base at that time. To calculate, for a given point cloud P, its diameter is diameterP, and the length d0 is defined as follows: When selecting a basis, the lengths of the line segments d1 and d2 in the basis must satisfy the following formula: (d1≥d0)∪(d2≥d0) Among them, α is the scale factor, the value is 0.85, and lcp can represent the current overlap ratio between the two point clouds; When lcp is a true value, it means that the current four-point basis is in the point cloud overlapping area. The value represents the maximum distance currently calculated for the line segments in the overlapping area, and is multiplied by the scale factor so that the distance of the line segments can fluctuate within a certain range to find the optimal basis; When lcp is less than When , it means that the group basis is not in the overlapping area, and the wrong match results in a small value of lcp. This will cause the selection of the basis to not fall into the overlapping area, and thus the correct transformation cannot be solved. At this time, d0=lcp*diameterP can calculate the distance reached by the line segment in the overlapping area.

8. The infrared temperature image and high-precision point cloud registration method according to claim 1, characterized in that: In S3.2, the following steps are included: S3.21, the four-point basis forms a shape similar to a quadrilateral, and the RC-ICP algorithm takes each point of the four-point basis as the center and extracts points within a certain distance range and facing the direction of the quadrilateral; S3.22, dividing the point cloud into four independent small blocks, and dividing the two point clouds at the same time to obtain four pairs of corresponding partitions; S3.23, find the closest point in each pair of partitioned point clouds, put all the point pairs together to solve the best transformation matrix; S3.24, take point clouds P and Q as an example, and for the convenience of representation, point clouds P and Q are set to be planar; S3.

25. For point clouds P and Q, the matching basis of the rough registration result is {a, b, c, d} and {a′, b′, c′, c′}, and line segments ab and cd are line segments with non-coterminal points in the basis of point cloud P. Then the region definition formula of point cloud P at point α is as follows: In the formula, β is a proportional factor, which is generally taken as 0.

5. Similarly, the nearby areas of other points of the basis and the corresponding points of Q are extracted to obtain four pairs of point cloud corresponding areas, among which ab and cd are the longest baseline segments, which can be essentially understood as the two groups of line segments with the farthest distance between the point clouds. The essence of fine registration is to use the four-point basis method of coarse registration to divide the entire point cloud P and point cloud Q into four parts, namely P1, P2, P3, P4, Q1, Q2, Q3, and Q4, according to the point basis and baseline segment, and then align these eight parts separately; The area can be divided by the formula in S3.

25. For example, if point cloud P has a baseline ab, then point cloud Q is the baseline. According to the above formula, any point p in point cloud P that satisfies the above formula, that is, the length of the line segment connecting ap is β times the length of ab, and the angle is less than 90°, is considered to be in the P1 part. Similarly, the points that satisfy the above formula in Q are the points in the Q1 part, and then P1 and Q1 are aligned. Similarly, points in P2, P3, P4, Q2, Q3, and Q4 can be found and then aligned.

9. The infrared temperature image and high-precision point cloud registration method according to claim 1, characterized in that: In S4, by introducing the information of the normal vector change gradient, the normal vector change trend can be found, which is convenient for increasing the weight factor for the local edge features, which specifically includes the following steps: S4.

1. First, solve the bounding box volume V and the number of point clouds Point of the dense point cloud data num , the local density of point cloud data is solved by the following formula; S4.

2. Find the point P to be retrieved in the sparse point cloud i , and find the closest point P j , and calculate its interval bounding box volume V', the formula is as follows: Among them, l is to P j distance; The number of iteration points is calculated based on local features. The formula is as follows: Cluster num =density×V'; For point P i , you need to retrieve its Cluster in the dense point cloud num points, and P i Assign the RGB value to this Cluster num points in a dense point cloud; S4.

3. Retrieve P i The 4 nearest points in the dense point cloud are selected as candidate points, and the points P i The point with the largest normal vector difference is taken as the cluster point Q′ i , and then cluster point Q′ i As a new retrieval point, search for other unclustered points in the dense point cloud according to the above method until the retrieval is complete. P i The RGB value is assigned to this Cluster num This is for P i The above iterations are continuously repeated to complete such clustering for all points in the sparse point cloud, so as to realize the expansion of sparse point cloud features to dense point cloud features.

Images