Heterogeneous data fusion-oriented homonymy feature point solving method

Abstract

The invention relates to the field of image processing and data fusion, discloses a homonymy feature point solving method for heterogeneous data fusion, and is suitable for heterogeneous data fusion to improve the accuracy and stability of a system. The method comprises the following steps: S1, acquiring scene data containing a dual-light cross marker by using an infrared thermal imager and a structured light camera; s2, preprocessing the acquired infrared image and the three-dimensional point cloud; s3, extracting feature points of the infrared image by using a Harris algorithm; s4, after the feature points are optimized through bilateral threshold filtering, classification is carried out through a K-Means algorithm; s5, processing the three-dimensional point cloud by using an RANSAC (Random Sample Consensus) algorithm and an Alpha Shapes algorithm, extracting edges and fitting straight lines to obtain intersection points; and S6, solving homonymous feature points in combination with intersection points in the infrared and point cloud data. According to the method disclosed by the invention, the accuracy of feature point detection and the stability of the system are improved by utilizing fusion of heterogeneous data.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and data fusion, and in particular to a method for solving feature points of the same name for heterogeneous data fusion. Background Art

[0002] For heterogeneous data fusion from infrared imagers and structured light cameras, solving the same-name feature points is a key task, which involves determining the intrinsic and extrinsic parameters of the camera. The same-name feature points refer to the same points corresponding in different images, which are usually found through feature matching algorithms. Currently commonly used feature point extraction algorithms include SIFT, SURF, ORB, etc. The matching algorithm can use nearest neighbor matching or some more complex methods. Deep learning technology is also widely used to extract and match features. In addition, the problem of solving the internal and external parameters of the camera is usually modeled as an optimization problem, in which the goal is to minimize the reprojection error. Optimization methods include least squares method, Levenberg-Marquardt algorithm, etc.

[0003] When using different sensors or cameras, due to differences in their imaging principles, photosensitive elements, and lens performance, the markers may have different imaging effects on different devices. This may make it difficult to simultaneously image clearly in some cases, thus affecting the accuracy of camera calibration. When designing markers, you can consider using features that adapt to a variety of imaging conditions, such as markers with distinct features in color, contrast, and shape. In addition, there may be noise in the image, as well as mismatches due to occlusion or changes in viewing angle, which may lead to incorrect feature matching, thus affecting the accuracy of calibration. When the camera or scene moves, there may be blur in the image, which will affect the accuracy of feature matching. Distortion of the camera lens is also a common problem and requires additional correction. As the image resolution and data volume increase, the computational complexity of solving camera parameters increases.

[0004] Therefore, the present invention proposes a method for solving feature points with the same name for heterogeneous data fusion to overcome the above shortcomings. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method for solving feature points with the same name for heterogeneous data fusion, which combines the RANSAC, AlphaShapes, and K-Means algorithms. While stabilizing the imaging quality, the accuracy of the image and point cloud is guaranteed. Combined with the "dual-light cross" marker, the method can realize the solution of feature points with the same name in the external parameter calibration of infrared thermal imagers and structured light cameras.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for solving feature points with the same name for heterogeneous data fusion, comprising the following steps:

[0007] Use infrared thermal imagers and structured light cameras to collect data on scenes containing "double light cross" markers;

[0008] Preprocess the collected infrared images and three-dimensional point cloud data;

[0009] Harris algorithm is used to extract feature points from the preprocessed infrared image;

[0010] Use bilateral threshold filtering to optimize the feature points in the infrared image, and then use the K-Means algorithm to classify the optimized feature points;

[0011] The RANSAC algorithm is used to segment the surface of the landmark from the preprocessed 3D point cloud, and the AlphaShapes algorithm is used to extract the edge, and the intersection of the 3D point cloud is obtained by fitting the straight line;

[0012] The intersection points obtained in the infrared image and the three-dimensional point cloud are combined to solve the feature points with the same name.

[0013] Preferably, the pretreatment step comprises:

[0014] Use Gaussian filtering to denoise infrared images, and its mathematical expression is:

[0015]

[0016] Among them, G(i,j) is a one-dimensional Gaussian kernel function, defined as:

[0017]

[0018] Where σ is the standard deviation of the Gaussian distribution;

[0019] Use statistical filtering methods to remove discrete points in 3D point cloud data, including:

[0020] Calculate point P i (x i ,y i ,z i ) within the specified radius of the neighborhood μ i and standard deviation σ i ;

[0021] Remove the items that satisfy z i >μ i +k·σ i or i <μ i -k·σ i where k is a user-defined parameter.

[0022] Preferably, the step of extracting feature points from the preprocessed infrared image using the Harris algorithm includes:

[0023] Harris algorithm is used to calculate the gradient I of each pixel in the infrared image. x and I y ;

[0024] Create the structure matrix M:

[0025]

[0026] Among them, w(i,j) is the Gaussian window function;

[0027] Calculate the response function R of the corner point:

[0028] R=det(M)-k·trace 2 (M)

[0029] Where det(M) is the determinant of the matrix M, trace(M) is the trace of the matrix M, and k is a constant;

[0030] If R is greater than the set threshold, it is determined to be a corner point.

[0031] Preferably, the step of optimizing the feature points in the infrared image using bilateral threshold filtering comprises:

[0032] Use bilateral threshold filtering to optimize the extracted feature points, retain edge information, and filter out noise;

[0033] Use the K-Means algorithm to classify the optimized feature points and calculate the Euclidean distance between any two points:

[0034]

[0035] Compute the center point of each cluster:

[0036]

[0037] Iterative optimization objective function J:

[0038]

[0039] Where K is the number of clusters, C k is the number of data points in the kth cluster, c k is the center point of the kth cluster;

[0040] Use the least squares method to fit a straight line and find the intersection points of the landmarks.

[0041] Preferably, the step of segmenting the surface of the marker from the preprocessed three-dimensional point cloud using the RANSAC algorithm, extracting the edge using the Alpha Shapes algorithm, and obtaining the intersection of the three-dimensional point cloud by fitting the straight line comprises:

[0042] Use the RANSAC algorithm to segment the surface of the landmark and randomly select three points P from the point cloud. 1 (x 1 ,y 1 ,z 1 ), P 2 (x 2 ,y 2 ,z 2 ), P 3 (x 3 ,y 3 ,z 3 ), calculate the coefficients A, B, C, D of the plane equation:

[0043] A=(y 2 -y 1 )(z 3 -z 1 )-(z 2 -z 1 )(y 3 -y 1 )

[0044] B=(x 2 -x 1 )(z 3 -z 1 )-(z 2 -z 1 )(x 3 -x 1 )

[0045] C=(x 2 -x 1 )(y 3 -y 1 )-(y 2 -y 1 )(x 3 -x 1 )

[0046] D=-(Ax 1 +By 1 +Cz 1 )

[0047] Calculate the distance l from each point in the point cloud to the plane, compare it with the tolerance error e, and count the number of internal points n;

[0048] Iteratively select three new points and repeat the above steps until the best plane model is found;

[0049] Use the AlphaShapes algorithm to extract the edge of the landmark. Assume that the radius of the rolling circle is a, process the point P(x,y), search for the point set Q whose distance from P is less than 2a, and calculate the center O of the circle. 1 (x o1 ,y o1 ) and O 2 (x o2 ,y o2 ):

[0050]

[0051] in, S 2 =(xx 1 ) 2 +(yy 1 ) 2 .

[0052] Preferably, the AlphaShapes algorithm further includes determining whether point P is a contour point. If all points in the point set Q are from point P to point O, 1 and O 2 If the distances of are all greater than a, then P is a contour point; otherwise, traverse the remaining points and recalculate the center of the circle.

[0053] Preferably, the step of solving the feature points with the same name by combining the infrared image and the intersection points obtained in the three-dimensional point cloud comprises:

[0054] Perform geometric transformation and registration on feature points of infrared images and 3D point clouds;

[0055] The feature points with the same name are paired according to the matching of the transformed feature points.

[0056] Preferably, the step of solving the same-name feature points by combining the infrared image and the intersection points obtained in the three-dimensional point cloud also includes improving the registration accuracy by an iterative optimization method.

[0057] Preferably, the "double light cross" marker has the following structural features:

[0058] The marker comprises an aluminum base plate, on the surface of which is placed a "cross"-shaped boss, the material of the boss being aluminum;

[0059] Matte black plastic sheets are applied over the top of the bosses and around the aluminum base plate to enhance imaging contrast in the thermal imaging camera while ensuring shape recognizability in the structured light camera.

[0060] The present invention also provides a device for solving feature points of the same name for heterogeneous data fusion, comprising an infrared thermal imager, a structured light camera and a processing unit, wherein the processing unit is configured to execute the method described, and the device comprises: a data acquisition module for acquiring infrared images and three-dimensional point cloud data;

[0061] Data preprocessing module, used to remove noise and discrete points from the collected data;

[0062] A feature point extraction module is used to extract feature points from the preprocessed data;

[0063] Feature point classification module, used to classify and optimize the extracted feature points;

[0064] The feature point solving module is used to combine the feature points of infrared images and three-dimensional point clouds to solve the feature points with the same name.

[0065] The present invention provides a method for solving feature points with the same name for heterogeneous data fusion. It has the following beneficial effects:

[0066] 1. The method of the present invention utilizes the fusion of heterogeneous data to improve the accuracy of feature point detection and the stability of the system. Through precise preprocessing and feature point extraction technology, feature points with the same name can be accurately matched in different sensor data, which is crucial for the calibration and application of multi-sensor systems.

[0067] 2. The present invention designs a "dual-light cross" marker based on the radiation coefficients of different materials and the imaging characteristics of sensors, so that the marker can be clearly imaged in different sensors. In addition, the designed "dual-light cross" marker can also be used for camera calibration in other bands.

[0068] 3. In combination with the characteristics of the "double light cross" marker, the present invention designs an algorithm for solving the same-name feature points, including a method for extracting the same-name feature points of infrared images based on RANSAC and a method for extracting the same-name feature points of three-dimensional point clouds based on AlphaShapes. The proposed method for solving the same-name feature points has improved accuracy compared with the traditional method. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a schematic diagram of the data acquisition system of the present invention;

[0070] Figure 2 It is a schematic diagram of the overall flow of the algorithm of the present invention;

[0071] Figure 3 This is a schematic diagram of the structure of the "cross double light" marker of the present invention;

[0072] Figure 4 It is a schematic diagram of the infrared image feature point solving algorithm of the present invention;

[0073] Figure 5 Schematic diagram of the three-dimensional point cloud feature point solving algorithm of the present invention. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] Please refer to the attached Figure 1 -Attached Figure 5 The embodiment of the present invention provides a method for solving feature points of the same name for heterogeneous data fusion, which is applied to the external parameter calibration of infrared thermal imagers and three-dimensional structured light cameras. The data acquisition system is as follows Figure 1 The infrared thermal imager and structured light camera collect data of the "double light cross" marker, and then transmit it to the industrial computer for data processing. The processing flow is as follows: Figure 2 shown.

[0076] Specifically, the method comprises the following steps:

[0077] S1. Data Collection

[0078] In this step, the infrared thermal imager and structured light camera are used to synchronously collect scene data containing the "double light cross" marker. The double light cross marker is made of materials with different radiation characteristics to ensure clear imaging effects in both infrared and visible light bands. The infrared thermal imager captures images based on the thermal radiation of the object and is suitable for nighttime or dimly lit environments; the structured light camera obtains three-dimensional information about the surface of the object by projecting a specific light pattern and capturing its reflection.

[0079] The design of the "Dual Light Cross" marker ensures that the color or texture of the "Dual Light Cross" marker has enough contrast for both the infrared camera and the structured light camera. In the infrared camera, the contrast depends more on the temperature difference of the object rather than the shape. In the structured light camera, the contrast of the shape is more important. When choosing the material, consider the reflective characteristics of the infrared camera and the structured light camera. Make sure that the material is not too reflective or absorptive under infrared or structured light conditions, so as not to affect the detection of feature points. Introduce more feature points in different orientations and positions of the "Dual Light Cross" marker to increase the stability of the calibration.

[0080] S2. Preprocessing

[0081] The collected data needs to be properly preprocessed. Gaussian filtering is used to remove random noise and enhance image quality for infrared images; statistical filtering methods are used for 3D point cloud data, such as calculating the mean and standard deviation of points in the neighborhood and removing discrete points, which helps to extract feature points and landmark edges more accurately in the future.

[0082] The infrared image preprocessing of the "double light cross" marker can optimize the marker image quality, reduce noise, and highlight the target features, which can be better applied to the subsequent task of solving the same-name feature points. The marker images collected by the thermal imager may contain various types of noise, including thermal noise, electronic noise, etc. Select a filter to preprocess it. The three-dimensional point cloud preprocessing of the "double light cross" marker removes the noise contained in the point cloud and improves the subsequent recognition accuracy of the same-name feature points. When processing three-dimensional point cloud data, noise usually manifests itself in the form of outliers, sampling noise, depth inconsistency, etc. The goal of preprocessing is to eliminate or reduce these noises so that the subsequent three-dimensional point cloud feature point recognition is more accurate and reliable. Use methods based on distance thresholds or statistical distributions to identify and eliminate outliers and improve processing accuracy.

[0083] S3. Feature point extraction

[0084] Harris algorithm is used to extract feature points from the preprocessed infrared image. Harris algorithm is a corner detection algorithm based on the local autocorrelation function of image gradient, which can effectively identify the corner points in the image. These corner points are usually used as feature points because of their uniqueness in the image.

[0085] S4. Feature point optimization and classification

[0086] Bilateral threshold filtering is used to further optimize the feature points in the infrared image. This method can not only reduce noise but also maintain edge clarity. The optimized feature points are then classified using the K-Means algorithm to group the feature points for more accurate processing and analysis.

[0087] S5. 3D point cloud processing

[0088] The RANSAC algorithm is used to segment the surface of the marker from the 3D point cloud. This algorithm can effectively identify and remove outlier data and retain the most representative plane data. Next, the edge of the marker is extracted using the Alpha Shapes algorithm. This algorithm forms a closed boundary based on the 3D point cloud data and then identifies the outer contour. Through these processing steps, the edge and shape of the marker can be accurately located from the point cloud.

[0089] S6. Solving the feature points with the same name

[0090] Finally, the intersection points obtained from the infrared image and the 3D point cloud are combined to solve the feature points with the same name. This step aligns the feature points of the two data sources through geometric transformation and registration technology, so as to find the corresponding points of the same physical location in different sensor data.

[0091] The method of the present invention utilizes the fusion of heterogeneous data to improve the accuracy of feature point detection and the stability of the system. Through precise preprocessing and feature point extraction technology, feature points with the same name can be accurately matched in different sensor data, which is crucial for the calibration and application of multi-sensor systems. In addition, the method can also be applied to a wide range of application scenarios, such as autonomous driving, robot navigation, augmented reality, etc., providing an efficient and reliable technical solution.

[0092] Please refer to the attached Figure 3 In a preferred embodiment of the present invention, when designing a "cross dual-light" marker, by making full use of the radiometric characteristics of different materials in infrared thermal imagers and structured light cameras, it can be seen from the material radiation coefficient table that the material radiation coefficient of the aluminum sheet is relatively small, only 0.05-0.07, while the material radiation coefficient of the plastic sheet is very large, 0.94, and the difference between the two material radiation coefficients is very large. When the markers are placed adjacent to each other, the infrared thermal imager can clearly image, but the structured light camera cannot obtain a three-dimensional point cloud containing feature points of the same name. To solve this problem, a 20mm thick "cross" boss is placed in the center of the aluminum plate, and a black matte plastic sheet is attached to it. To improve the imaging effect, black matte plastic sheets are also attached around the aluminum plate.

[0093] To achieve clear images in both imaging systems simultaneously. This involves selecting materials to ensure that they exhibit good radiometric contrast in both the infrared band and the visible light band. The markers are designed so that they can produce clear and distinguishable images under different sensors. This approach not only improves the overall imaging effect, but also helps to meet the needs of multi-sensor systems to obtain more comprehensive target information.

[0094] In a preferred embodiment of the present invention, in a preferred embodiment of the present invention, the preprocessing step uses two main filtering techniques to optimize the infrared image and three-dimensional point cloud data to facilitate subsequent feature point extraction and analysis.

[0095] Gaussian filtering for infrared image denoising

[0096] The preprocessing of infrared images includes denoising the images using Gaussian filtering to improve image quality and reduce errors in the processing process. Gaussian filtering is a linear smoothing filter that is suitable for eliminating Gaussian noise and can maintain the effect of image edges. The filtering process uses a one-dimensional Gaussian kernel function to process each pixel in the image. The specific mathematical expression is:

[0097]

[0098] Among them, G(i,j) is a one-dimensional Gaussian kernel function, defined as:

[0099]

[0100] Among them, σ is the standard deviation of the Gaussian distribution, and its value is selected according to the actual denoising requirements to achieve the best image smoothing effect.

[0101] Statistical filtering of 3D point clouds to remove discrete points

[0102] For a 3D point cloud, it is necessary to remove the discrete points it contains to improve the accuracy of the solution. Discrete points may be outliers caused by sensor noise, incomplete scanning or other factors. Statistical filtering is used to remove discrete points. By calculating the statistical information of the points in the neighborhood of each point, it is determined whether the point is a discrete point. This method usually uses the mean and standard deviation for outlier detection and removes points that exceed the threshold. This method includes the following steps:

[0103] First, the statistical filtering method is used to remove discrete points in the point cloud. Given a point P in the point cloud i (x i ,y i ,z i ) Statistical filtering first calculates the statistical information of the point in a neighborhood within a specified radius, usually using the mean and standard deviation. The set of points in the neighborhood is denoted by N i First, calculate the mean μ i :

[0104]

[0105] Then calculate the standard deviation σ i :

[0106]

[0107] Among them, P j,z Indicates point P j The z coordinate of |N i | represents the number of points in the neighborhood.

[0108] After calculating the mean and standard deviation, define the threshold T:

[0109] T=μ i ±k·σ i

[0110] Here, k is a user-defined parameter used to adjust the sensitivity of the threshold.

[0111] If zi >T or z i <-T, then P i is a discrete point. Through this process, statistical filtering can effectively identify and remove points with abnormal height changes within a given radius.

[0112] Through these optimized preprocessing steps, the embodiments of the present invention can effectively improve the processing quality of data and provide a more accurate data basis for subsequent feature point matching and solving feature points with the same name. Gaussian filtering ensures the clarity and accuracy of infrared images, while statistical filtering helps remove abnormal points from point cloud data that may interfere with analysis. These preprocessing measures are key steps to achieve efficient and reliable multi-sensor data fusion, and have a significant impact on improving the overall performance and stability of the system.

[0113] In a preferred embodiment of the present invention, the feature point extraction process utilizes the Harris algorithm, which is an algorithm widely used in corner point detection, and is particularly suitable for feature point extraction in infrared images because it can effectively identify the boundaries and corner points generated by temperature differences in the image. The following are the detailed steps and principles of the Harris algorithm in the present invention:

[0114] 1. Calculate the gradient

[0115] First, the gradient I of each pixel is calculated for the preprocessed infrared image. x and I y , the formula is as follows:

[0116]

[0117] Where I is the intensity of the image, I x and I y are the gradients of the image in the x and y directions, respectively.

[0118] This step is the basis for identifying potential corner locations in the image, because corner regions usually have large changes in the gradient direction.

[0119] Harris algorithm is used to calculate the gradient I of each pixel in the infrared image. x and I y ;

[0120] 2. Establish a structural matrix

[0121] Use the calculated gradient values ​​to establish the structure matrix M:

[0122]

[0123] Among them, w(i,j) is a Gaussian window function, which is used to give different weights to pixels at different positions, and the center point usually has the largest weight. This weighting process helps to enhance the influence of local features, making corner detection more accurate.

[0124] 3. Calculate the response function of the corner point

[0125] Calculate the corner response function R of the structure matrix M:

[0126] R=det(M)-k·trace 2 (M)

[0127] Among them, det(M) is the determinant of the matrix M, trace(M) is the trace of the matrix M, k is a constant that affects the sensitivity of the response function. The higher the value of the response function R, the more likely the corresponding pixel is a corner point.

[0128] 4. Corner point determination

[0129] If the value of R is greater than the preset threshold, the corresponding pixel is determined to be a corner point. The setting of this threshold depends on the characteristics of the image and application requirements. A higher threshold means that only the most significant corner points will be detected.

[0130] The use of the Harris algorithm to extract corner points in infrared images makes feature point detection not only accurate but also robust, especially when dealing with scenes with high temperature contrast. In addition, due to the precise calculation of the structure matrix and response function, the algorithm can effectively distinguish between real corner points and noise, thereby improving the accuracy of subsequent data fusion and feature point matching. This method is suitable for a variety of applications that require high-precision feature point detection, such as environmental perception for autonomous vehicles, event detection in security monitoring systems, and other machine vision systems.

[0131] Please refer to the attached Figure 4 In a preferred embodiment of the present invention, the steps of optimizing the feature points in the infrared image using bilateral threshold filtering and then classifying the optimized feature points using the K-Means algorithm include:

[0132] 1. Bilateral threshold filtering optimization

[0133] Apply bilateral threshold filtering to the extracted feature points for optimization. Bilateral threshold filtering is an edge-preserving filtering technique that combines spatial proximity and pixel difference to effectively remove noise while retaining edge information. This step is critical because it not only removes noise from the image, but also retains important edge information of the feature points, which is crucial for subsequent feature point classification and matching.

[0134] 2. K-Means Algorithm Classification

[0135] Use the K-Means algorithm to classify the optimized feature points. K-Means is a widely used clustering algorithm that divides data points into K clusters through an iterative process to minimize the distance between each point and its cluster center. The specific process is as follows:

[0136] Calculate the Euclidean distance. For any two feature points p and q, calculate the Euclidean distance between them:

[0137]

[0138] This distance measures the similarity of feature points in the feature space.

[0139] Compute cluster centers:

[0140]

[0141] Among them, C k is the set of data points of the kth cluster, c k is the mean of these points and represents the center point of the cluster.

[0142] Optimization objective function:

[0143]

[0144] The goal is to minimize the sum of the squares of the distances from each point to its cluster center, thereby determining the optimal center point for each cluster.

[0145] 3. Least squares method to fit straight line and solve intersection points

[0146] After the feature points are classified, the least squares method is used to fit a straight line to each classified cluster to solve the intersection of the landmarks. This step uses mathematical methods to determine the best fit line, thereby accurately calculating the intersection positions of different straight lines. These intersections are the feature points with the same name, which are crucial for further image analysis and data fusion.

[0147] This method not only improves the detection accuracy of feature points, but also optimizes the data processing process. Accurate classification of feature points and precise intersection solution are critical to improving the overall performance and reliability of image processing systems, especially in technical fields that require high-precision positioning and recognition, such as autonomous driving, robot navigation, and security monitoring. In addition, the implementation of this method improves the automation level of feature point processing, reduces the need for manual intervention, and increases the efficiency of the processing process.

[0148] Please refer to the attached Figure 5In a preferred embodiment of the present invention, the RANSAC algorithm and the Alpha Shapes algorithm are used to process the preprocessed three-dimensional point cloud data to accurately segment the surface of the marker and extract its edges. This processing flow effectively utilizes the advantages of these two algorithms to improve the accuracy and efficiency of three-dimensional data processing. The specific steps of this embodiment are described in detail below:

[0149] RANSAC algorithm segmentation of landmark surface

[0150] 1. Random point selection and plane equation calculation:

[0151] Randomly select three points P from the 3D point cloud 1 (x 1 ,y 1 ,z 1 ), P 2 (x 2 ,y 2 ,z 2 ), P 3 (x 3 ,y 3 ,z 3 );

[0152] To calculate the coefficients A, B, C, D of the plane defined by these three points, use the following formulas:

[0153] A=(y 2 -y 1 )(z 3 -z 1 )-(z 2 -z 1 )(y 3 -y 1 )

[0154] B=(x 2 -x 1 )(z 3 -z 1 )-(z 2 -z 1 )(x 3 -x 1 )

[0155] C=(x 2 -x 1 )(y 3 -y 1 )-(y 2 -y 1 )(x 3 -x 1 )

[0156] D=-(Ax 1 +By1 +Cz 1 )

[0157] 2. Interior point calculation and model verification:

[0158] For each point in the point cloud, calculate its vertical distance l to the above plane;

[0159] Compare the distance with the preset tolerance error e to determine which points belong to the inner points;

[0160] This process is repeated to find the plane with the largest set of internal points as the best model through iterative optimization.

[0161] Alpha Shapes algorithm to extract edges

[0162] 1. Edge extraction preparation:

[0163] Process the determined plane, project the point cloud data onto the plane, select a point P(x,y), and find all points whose distance from the point on the 2D plane is less than 2a to form a point set Q.

[0164] 2. Calculate the center of the circle and determine the edge:

[0165] For every point x in the point set Q 1 ,y 1 , calculate the possible center O 1 (x o1 ,y o1 ) and O 2 (x o2 ,y o2 ), use the following formula:

[0166]

[0167] in, S 2 =(xx 1 ) 2 +(yy 1 ) 2 .

[0168] 3. Edge determination:

[0169] The AlphaShapes algorithm is used to determine the edge of the point cloud data based on the center and radius of the circle calculated above.

[0170] This embodiment provides an efficient 3D point cloud processing method combining the RANSAC algorithm and the AlphaShapes algorithm, making it possible to accurately segment and extract the edges of landmarks from complex backgrounds. This method not only enhances the data processing accuracy, but also improves the processing speed, and is applicable to various technical fields that require high-precision 3D point cloud processing, such as machine vision, automated production, robot navigation, and virtual reality.

[0171] In a preferred embodiment of the present invention, the application of the Alpha Shapes algorithm is not limited to extracting the edge of the marker, but further includes a step of determining whether point P is a contour point, thereby enhancing the accuracy of edge detection. The detailed description of this step is as follows:

[0172] Contour point judgment in Alpha Shapes algorithm

[0173] 1. Calculation of the distance between the center of the circle:

[0174] For the identified point set Q, after calculating the center O 1 and O 2 Finally, the distance from each point in the point set to the two circle centers is further measured.

[0175] 2. Contour point recognition:

[0176] If all points in the point set Q are connected to O 1 and O 2 If the distances of the points P(x,y) are all greater than the radius a of the circle, then the initial point P(x,y) is identified as a contour point. This means that point P is outside the closed contour formed by the point set Q, marking an obvious boundary point.

[0177] 3. Iteration process:

[0178] If there are one or more points in the point set Q whose distance does not exceed the radius a, this indicates that P may not be a contour point. In this case, it is necessary to traverse the remaining points in Q and recalculate the center O for each new point as a potential contour point. 1 and O 2 .

[0179] Repeat the above distance detection and circle center recalculation process until a contour point is found that satisfies that the distance of all points is greater than a, or it is confirmed that P is not a contour point.

[0180] This enhanced Alpha Shapes algorithm step makes the edge extraction process more accurate, ensuring that only contour points are correctly identified, thereby reducing errors and uncertainties in subsequent processing. This precise contour detection is particularly important for highly accurate 3D modeling and analysis applications, such as engineering modeling, geographic information systems (GIS), biomedical imaging, and other fields that require precise edge identification.

[0181] In a preferred embodiment of the present invention, the detailed steps of solving the feature points with the same name by combining the infrared image and the intersection points obtained in the three-dimensional point cloud are as follows:

[0182] Geometric transformation and registration

[0183] 1. Geometric transformation:

[0184] The infrared image and 3D point cloud data are geometrically transformed to ensure that the two data are compared and analyzed in the same coordinate system. This step usually includes rotation, translation and scaling operations to align the two data sources so that they have the same scale and orientation.

[0185] 2. Data registration:

[0186] After the geometric transformation is completed, data registration is performed, which is a key process that involves maximizing the correspondence between the feature points of the infrared image and the feature points of the 3D point cloud. The registration algorithm may use feature-based methods such as the Iterative Closest Point (ICP) algorithm, or other advanced image registration techniques to ensure the spatial consistency of the two data.

[0187] Feature point matching and pairing of feature points with the same name

[0188] 1. Feature point matching:

[0189] Using the registered data, feature point matching is performed. This step uses an algorithm to identify and match feature points in the infrared image with corresponding feature points in the 3D point cloud. The matching process usually relies on the geometric properties and relative positions of the feature points, as well as possible texture or color information.

[0190] 2. Pairing of feature points with the same name:

[0191] After successfully matching feature points, further identification is performed on those feature points with the same name that appear in both the infrared image and the 3D point cloud. These feature points are crucial for subsequent analysis, such as object recognition, scene reconstruction, or positioning in navigation systems.

[0192] Through accurate geometric transformation, precise data registration and effective feature point matching, the present invention can achieve high consistency and fusion between different sensor data, thereby improving the accuracy and reliability of the system in practical applications. In addition, the accurate pairing of feature points with the same name provides strong support for multi-sensor fusion, enabling the system to provide more accurate analysis results in complex environments, and is particularly suitable for advanced applications that require precise environmental perception, such as self-driving cars, robotics, and augmented reality.

[0193] In a preferred embodiment of the present invention, the process of solving the same-name feature points by combining the infrared image and the intersection points obtained in the three-dimensional point cloud includes not only geometric transformation and registration, but also the use of iterative optimization methods to further improve the registration accuracy. The detailed steps are as follows:

[0194] Iterative optimization method improves registration accuracy

[0195] 1. Initial registration:

[0196] First, according to the geometric transformation and data registration steps described previously, a preliminary registration is performed to align the infrared image and 3D point cloud data to a unified coordinate system.

[0197] 2. Error measurement and adjustment:

[0198] The initial registration is evaluated using some error metric (e.g., mean squared error, surface distance difference, or other similarity metric). Based on this evaluation, parameters or areas that require adjustment are identified.

[0199] 3. Iterative optimization:

[0200] Apply iterative optimization techniques, such as the iterative closest point (ICP) algorithm, to gradually optimize the transformation parameters (including rotation, translation, and scaling) to minimize the above error metrics. This process may involve multiple iterations, each of which adjusts the parameters based on the results of the previous one, until a preset accuracy standard is achieved or the improvement is no longer significant.

[0201] In each iteration, the matches between feature points are recomputed and the registration parameters are updated to ensure the best correspondence between feature points.

[0202] 4. Convergence confirmation:

[0203] Check the changes in the error metric regularly to confirm whether the convergence condition has been met, that is, the error improvement has stabilized or reached the minimum error threshold allowed by the system.

[0204] By introducing an iterative optimization method, this embodiment can significantly improve the registration accuracy of infrared images and three-dimensional point cloud data. This high-precision registration is the key to achieving accurate matching of feature points with the same name, and more importantly, it allows the system to maintain a high degree of accuracy and robustness when facing complex scenes and changing environmental conditions.

[0205] In addition, the iterative optimization method ensures that the best registration effect can be achieved through step-by-step adjustments even when the initial conditions are not ideal. The application of this method greatly improves the practicality and efficiency of multi-sensor data fusion technology, and is particularly suitable for technical fields such as high-precision measurement, robot navigation, automated monitoring, and advanced computer vision systems.

[0206] The apparatus for solving feature points with the same name for heterogeneous data fusion described below and the method for solving feature points with the same name for heterogeneous data fusion described above can be referred to each other.

[0207] The present invention also provides an infrared thermal imager, a structured light camera and a processing unit, wherein the device comprises:

[0208] Data acquisition module, used for collecting infrared images and 3D point cloud data;

[0209] Data preprocessing module, used to remove noise and discrete points from the collected data;

[0210] A feature point extraction module is used to extract feature points from the preprocessed data;

[0211] Feature point classification module, used to classify and optimize the extracted feature points;

[0212] The feature point solving module is used to combine the feature points of infrared images and three-dimensional point clouds to solve the feature points with the same name.

[0213] The device of this embodiment can be used to execute the above method embodiment, and its principles and technical effects are similar, which will not be repeated here.

[0214] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for solving feature points with the same name for heterogeneous data fusion, characterized in that: The following steps are involved: Use infrared thermal imagers and structured light cameras to collect data on scenes containing "double light cross" markers; Preprocess the collected infrared images and three-dimensional point cloud data; Harris algorithm is used to extract feature points from the preprocessed infrared image; Use bilateral threshold filtering to optimize the feature points in the infrared image, and then use the K-Means algorithm to classify the optimized feature points; The RANSAC algorithm is used to segment the surface of the landmark from the preprocessed 3D point cloud, and the Alpha Shapes algorithm is used to extract the edge, and the intersection of the 3D point cloud is obtained by fitting the straight line; The intersection points obtained in the infrared image and the three-dimensional point cloud are combined to solve the feature points with the same name.

2. The method for solving feature points with the same name for heterogeneous data fusion according to claim 1 is characterized in that: The pre-processing step comprises: Use Gaussian filtering to denoise infrared images, and its mathematical expression is: Among them, G(i,j) is a one-dimensional Gaussian kernel function, defined as: Where σ is the standard deviation of the Gaussian distribution; Use statistical filtering methods to remove discrete points in 3D point cloud data, including: Calculate point P i (x i ,y i ,z i ) within the specified radius of the neighborhood i and standard deviation σ i ; Remove the items that satisfy z i >μ i +k·σ i or i <μ i -k·σ i where k is a user-defined parameter.

3. The method for solving feature points with the same name for heterogeneous data fusion according to claim 1 is characterized in that: The step of extracting feature points from the preprocessed infrared image using the Harris algorithm comprises: Harris algorithm is used to calculate the gradient I of each pixel in the infrared image. x and I y ; Create the structure matrix M: Among them, w(i,j) is the Gaussian window function; Calculate the response function R of the corner point: R=det(M)-k trace 2 (M) Where det(M) is the determinant of the matrix M, trace(M) is the trace of the matrix M, and k is a constant; If R is greater than the set threshold, it is determined to be a corner point.

4. The method for solving feature points with the same name for heterogeneous data fusion according to claim 1 is characterized in that: The steps of optimizing the feature points in the infrared image using bilateral threshold filtering and then classifying the optimized feature points using the K-Means algorithm include: Use bilateral threshold filtering to optimize the extracted feature points, retain edge information, and filter out noise; Use the K-Means algorithm to classify the optimized feature points and calculate the Euclidean distance between any two points: Compute the center point of each cluster: Iterative optimization objective function J: Where K is the number of clusters, C k is the number of data points in the kth cluster, c k is the center point of the kth cluster; Use the least squares method to fit a straight line and find the intersection points of the landmarks.

5. The method for solving feature points with the same name for heterogeneous data fusion according to claim 1 is characterized in that: The steps of segmenting the surface of the marker from the preprocessed three-dimensional point cloud using the RANSAC algorithm, extracting the edge using the Alpha Shapes algorithm, and obtaining the intersection of the three-dimensional point cloud by fitting the straight line include: Use the RANSAC algorithm to segment the surface of the landmark, randomly select three points P1 (x1, y1, z1), P2 (x2, y2, z2), and P3 (x3, y3, z3) from the point cloud, and calculate the coefficients A, B, C, and D of the plane equation: A=(y2-y1)(z3-z1)-(z2-z1)(y3-y1) B=(x2-x1)(z3-z1)-(z2-z1)(x3-x1) C=(x2-x1)(y3-y1)-(y2-y1)(x3-x1) D=-(Ax1+By1+Cz1) Calculate the distance l from each point in the point cloud to the plane, compare it with the tolerance error e, and count the number of internal points n; Iteratively select three new points and repeat the above steps until the best plane model is found; Use the AlphaShapes algorithm to extract the edge of the landmark. Assume that the radius of the rolling circle is a, process the point P(x,y), search for the point set Q whose distance from P is less than 2a, and calculate the center O1(x o1 ,y o1 ) and O2(x o2 ,y o2 ): in, S 2 =(x-x1) 2 +(y-y1) 2 .

6. The method for solving feature points with the same name for heterogeneous data fusion according to claim 5 is characterized in that: The AlphaShapes algorithm further includes determining whether point P is a contour point. If the distances from all points in the point set Q to O1 and O2 are greater than a, then P is a contour point; otherwise, traverse the remaining points and recalculate the center of the circle.

7. The method for solving feature points with the same name for heterogeneous data fusion according to claim 1 is characterized in that: The step of solving the feature points with the same name by combining the infrared image and the intersection points obtained in the three-dimensional point cloud comprises: Perform geometric transformation and registration on feature points of infrared images and 3D point clouds; The feature points with the same name are paired according to the matching of the transformed feature points.

8. The method for solving feature points with the same name for heterogeneous data fusion according to claim 7 is characterized in that: The step of solving the same-name feature points by combining the infrared image and the intersection points obtained in the three-dimensional point cloud also includes improving the registration accuracy by an iterative optimization method.

9. The method for solving feature points with the same name for heterogeneous data fusion according to claim 1, characterized in that: The marker comprises an aluminum base plate, on the surface of which is placed a "cross" shaped boss, the material of the boss being aluminum; Matte black plastic sheets are applied over the top of the bosses and around the aluminum base plate to enhance imaging contrast in the thermal imaging camera while ensuring shape recognizability in the structured light camera.

10. A device for solving feature points of the same name for heterogeneous data fusion, comprising an infrared thermal imager, a structured light camera and a processing unit, characterized in that: The processing unit is configured to execute the method according to any one of claims 1 to 9, and the device comprises: Data acquisition module, used for collecting infrared images and 3D point cloud data; Data preprocessing module, used to remove noise and discrete points from the collected data; A feature point extraction module is used to extract feature points from the preprocessed data; Feature point classification module, used to classify and optimize the extracted feature points; The feature point solving module is used to combine the feature points of infrared images and three-dimensional point clouds to solve the feature points with the same name.

Images