A method for solving homonymous feature points for heterogeneous data fusion

Abstract

This application relates to the field of image processing and data fusion, and discloses a method for solving for corresponding feature points in heterogeneous data fusion, applicable to heterogeneous data fusion, to improve the accuracy and stability of the system. The method steps are as follows: S1, acquire scene data containing "double-light cross" markers using an infrared thermal imager and a structured light camera; S2, preprocess the acquired infrared image and 3D point cloud; S3, extract feature points from the infrared image using the Harris algorithm; S4, optimize the feature points using bilateral threshold filtering, and then classify them using the K-Means algorithm; S5, process the 3D point cloud using the RANSAC algorithm and the Alpha Shapes algorithm, extract edges, and fit straight lines to obtain intersection points; S6, combine the intersection points from the infrared and point cloud data to solve for corresponding feature points. This invention utilizes the fusion of heterogeneous data to improve the accuracy of feature point detection and the stability of the system.

Description

Technical Field

[0001] This invention relates to the fields of image processing and data fusion technology, specifically to a method for solving for identical feature points in heterogeneous data fusion. Background Technology

[0002] For heterogeneous data fusion from infrared thermal imagers and structured light cameras, finding corresponding feature points is a crucial task, involving determining the camera's intrinsic and extrinsic parameters. Corresponding feature points are identical points in different images, typically found using feature matching algorithms. Commonly used feature point extraction algorithms include SIFT, SURF, and ORB. Matching algorithms can use nearest neighbor matching or more complex methods. Deep learning techniques are also widely used for feature extraction and matching. Furthermore, the problem of finding camera intrinsic and extrinsic parameters is often modeled as an optimization problem, where the objective is to minimize the reprojection error. Optimization methods include least squares and the Levenberg-Marquardt algorithm.

[0003] When using different sensors or cameras, due to differences in their imaging principles, image sensors, and lens performance, markers may produce different imaging effects on different devices. This can lead to difficulties in simultaneously capturing clear images in certain situations, thus affecting the accuracy of camera calibration. When designing markers, it is advisable to consider using features that are adaptable to various imaging conditions, such as markers with distinctive characteristics in color, contrast, and shape. Furthermore, noise in the image, as well as mismatches due to occlusion or changes in viewpoint, can lead to incorrect feature matching, thus affecting calibration accuracy. When the camera or scene moves, blurring may occur in the image, affecting the accuracy of feature matching. Camera lens distortion is also a common problem, requiring additional correction. As image resolution and data volume increase, the computational complexity of solving camera parameters also increases.

[0004] Therefore, this invention proposes a method for solving for homonymous feature points in heterogeneous data fusion, overcoming the above shortcomings. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for solving corresponding feature points for heterogeneous data fusion. It combines RANSAC, AlphaShapes, and K-Means algorithms, which stabilizes imaging quality while ensuring the accuracy of images and point clouds. Combined with "double light cross" markers, it can solve corresponding feature points in the extrinsic parameter calibration of infrared thermal imagers and structured light cameras.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for solving for homonymous feature points in heterogeneous data fusion, comprising the following steps:

[0007] Infrared thermal imagers and structured light cameras were used to collect data on scenes containing the "double light cross" marker.

[0008] Preprocess the acquired infrared images and 3D point cloud data;

[0009] Feature points were extracted from the preprocessed infrared image using the Harris algorithm.

[0010] The feature points in the infrared image are optimized using bilateral threshold filtering, and then the optimized feature points are classified using the K-Means algorithm.

[0011] The RANSAC algorithm was used to segment the marker surface from the preprocessed 3D point cloud, and the AlphaShapes algorithm was used to extract the edges. The intersection of the 3D point cloud was obtained by fitting the straight line.

[0012] We solve for the corresponding feature points by combining the intersection points obtained from infrared images and 3D point clouds.

[0013] Preferably, the preprocessing step includes:

[0014] The mathematical expression for using Gaussian filtering to denoise infrared images is:

[0015]

[0016] Where G(i,j) is a one-dimensional Gaussian kernel function, defined as:

[0017]

[0018] Where σ is the standard deviation of the Gaussian distribution;

[0019] Statistical filtering methods are used to remove discrete points from 3D point cloud data, including:

[0020] Calculate point P i (x i ,y i ,z i The mean μ of the neighborhood within the specified radius i and standard deviation σ i ;

[0021] Remove those that satisfy z i >μ i +k·σ i or z i <μ i -k·σ i The discrete points, where k is a user-defined parameter.

[0022] Preferably, the step of extracting feature points from the preprocessed infrared image using the Harris algorithm includes:

[0023] The Harris algorithm is used to calculate the gradient I of each pixel in the infrared image. x and I y ;

[0024] Establish the structure matrix M:

[0025]

[0026] Where w(i,j) is the Gaussian window function;

[0027] Calculate the response function R at the corner point:

[0028] R = det(M) - k·trace 2 (M)

[0029] Where det(M) is the determinant of matrix M, trace(M) is the trace of matrix M, and k is a constant;

[0030] If R is greater than the set threshold, it is determined to be a corner point.

[0031] Preferably, the step of optimizing feature points in the infrared image using bilateral threshold filtering includes:

[0032] The extracted feature points are optimized using bilateral threshold filtering to preserve edge information and filter out noise.

[0033] The K-Means algorithm is used to classify the optimized feature points, and the Euclidean distance between any two points is calculated:

[0034]

[0035] Calculate the center point of each cluster:

[0036]

[0037] Iterative optimization of objective function J:

[0038]

[0039] Where K is the number of clusters, C k c is the number of data points in the k-th cluster. k It is the center point of the k-th cluster;

[0040] The intersection points of the markers are solved by fitting a straight line using the least squares method.

[0041] Preferably, the steps of segmenting the marker surface from the preprocessed 3D point cloud using the RANSAC algorithm, extracting edges using the Alpha Shapes algorithm, and obtaining the intersection points of the 3D point cloud using fitted straight lines include:

[0042] The surface of the marker is segmented using the RANSAC algorithm. Three points, P1(x1,y1,z1), P2(x2,y2,z2), and P3(x3,y3,z3), are randomly selected from the point cloud. The coefficients A, B, C, and D of the plane equation are calculated.

[0043] A=(y2-y1)(z3-z1)-(z2-z1)(y3-y1)

[0044] B = (x2 - x1)(z3 - z1) - (z2 - z1)(x3 - x1)

[0045] C=(x2-x1)(y3-y1)-(y2-y1)(x3-x1)

[0046] D = -(Ax1 + By1 + Cz1)

[0047] Calculate the distance l from each point in the point cloud to the plane, compare it with the tolerance error e, and count the number of inliers n;

[0048] Iteratively select three new points and repeat the above steps until the optimal planar model is found.

[0049] The AlphaShapes algorithm is used to extract the edges of the marker. Let the radius of the rolling circle be *a*. For point P(x,y), the set of points Q whose distance from P is less than 2*a* is searched, and the center O1(x,y) of the circle is calculated. o1 ,y o1 ) and O2(x o2 ,y o2 ):

[0050]

[0051] in, S 2 =(x-x1) 2 +(y-y1) 2 .

[0052] Preferably, the AlphaShapes algorithm further includes determining whether point P is a contour point. If the distance from all points in the point set Q to O1 and O2 is greater than a, then P is a contour point; otherwise, the remaining points are traversed and the center of the circle is recalculated.

[0053] Preferably, the step of solving for corresponding feature points by combining the intersection points obtained from the infrared image and the 3D point cloud includes:

[0054] Perform geometric transformation and registration on feature points of infrared images and 3D point clouds;

[0055] Pairing of feature points with the same name is performed based on the transformed feature point matching.

[0056] Preferably, the step of solving for corresponding feature points by combining the intersection points obtained from the infrared image and the 3D point cloud further includes improving the registration accuracy through an iterative optimization method.

[0057] Preferably, the "double light cross" marker has the following structural features:

[0058] The marker includes an aluminum base plate with a cross-shaped boss on its surface, the boss being made of aluminum;

[0059] Black matte plastic sheets are attached above the boss and around the aluminum base plate to enhance the imaging contrast in the infrared thermal imager, while ensuring the shape's recognizability in the structured light camera.

[0060] The present invention also provides a device for solving corresponding feature points for heterogeneous data fusion, including an infrared thermal imager, a structured light camera and a processing unit, wherein the processing unit is configured to execute the method described, and the device includes: a data acquisition module for acquiring infrared images and three-dimensional point cloud data;

[0061] The data preprocessing module is used to denoise and remove discrete points from the collected data;

[0062] The feature point extraction module is used to extract feature points from the preprocessed data;

[0063] The feature point classification module is used to classify and optimize the extracted feature points;

[0064] The feature point solving module is used to solve for corresponding feature points by combining feature points from infrared images and 3D point clouds.

[0065] This invention provides a method for solving for identical feature points in heterogeneous data fusion. It has the following beneficial effects:

[0066] 1. The method of this invention utilizes the fusion of heterogeneous data to improve the accuracy of feature point detection and the stability of the system. Through precise preprocessing and feature point extraction techniques, it is possible to accurately match corresponding feature points in different sensor data, which is crucial for the calibration and application of multi-sensor systems.

[0067] 2. Based on the radiation coefficients of different materials and the imaging characteristics of sensors, this invention designs a "double light cross" marker, which enables the marker to be clearly imaged in different types of sensors. In addition, the designed "double light cross" marker can also be used for camera calibration in other bands.

[0068] 3. This invention, taking into account the characteristics of the "double-light cross" marker, designs a corresponding feature point extraction algorithm, including a RANSAC-based infrared image corresponding feature point extraction method and an AlphaShapes-based 3D point cloud corresponding feature point extraction method. The proposed corresponding feature point extraction method shows improved accuracy compared to traditional methods. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of the data acquisition system of the present invention;

[0070] Figure 2 This is a schematic diagram of the overall algorithm flow of the present invention;

[0071] Figure 3 This is a schematic diagram of the "cross-shaped double light" marker structure of the present invention;

[0072] Figure 4 This is a schematic diagram of the infrared image feature point solving algorithm of the present invention;

[0073] Figure 5 This is a schematic diagram of the algorithm for solving 3D point cloud feature points according to the present invention. Detailed Implementation

[0074] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] Please see the appendix Figure 1 -Appendix Figure 5 This invention provides a method for solving for identical feature points in heterogeneous data fusion, applicable to the extrinsic parameter calibration of infrared thermal imagers and 3D structured light cameras. The data acquisition system is as follows: Figure 1 As shown. The infrared thermal imager and structured light camera collect data from the "double-light cross" marker, then transmit the data to the industrial control computer for data processing. The processing flow is as follows: Figure 2 As shown.

[0076] Specifically, the method includes the following steps:

[0077] S1, Data Acquisition

[0078] In this step, scene data containing the "double light cross" markers is simultaneously acquired using an infrared thermal imager and a structured light camera. The double light cross markers are made of materials with different radiation properties to ensure clear imaging in both the infrared and visible light bands. The infrared thermal imager captures images based on the object's thermal radiation, making it suitable for nighttime or low-light environments; the structured light camera, on the other hand, obtains three-dimensional information about the object's surface by projecting specific light patterns and capturing their reflections.

[0079] The design of the "double-light cross" marker ensures that its color or texture provides sufficient contrast for both infrared thermal imagers and structured light cameras. In infrared thermal imagers, contrast depends more on the object's temperature difference than its shape. In structured light cameras, shape contrast is more important. The reflectivity of both infrared thermal imagers and structured light cameras is considered when selecting materials. The materials are ensured not to reflect or absorb excessively under infrared or structured light conditions, which could affect the detection of feature points. Introducing more feature points at different orientations and positions of the "double-light cross" marker increases calibration stability.

[0080] S2, Preprocessing

[0081] The acquired data needs to be preprocessed appropriately. Gaussian filtering is used on infrared images to remove random noise and enhance image quality; statistical filtering methods are used on 3D point cloud data, such as calculating the mean and standard deviation of points in the neighborhood to remove discrete points, which helps to extract feature points and landmark edges more accurately in the subsequent process.

[0082] Infrared image preprocessing of the "double-light cross" marker can optimize image quality, reduce noise, and highlight target features, making it better suited for subsequent tasks involving the identification of corresponding feature points. The marker images acquired by thermal imagers may contain various types of noise, including thermal noise and electronic noise. Filters are selected for preprocessing. 3D point cloud preprocessing of the "double-light cross" marker removes noise from the point cloud, improving the accuracy of subsequent corresponding feature point identification. When processing 3D point cloud data, noise typically manifests as outliers, sampling noise, and depth inconsistencies. The goal of preprocessing is to eliminate or mitigate this noise, making subsequent 3D point cloud feature point identification more accurate and reliable. Methods based on distance thresholds or statistical distributions are used to identify and remove outliers, improving processing accuracy.

[0083] S3, Feature Point Extraction

[0084] The Harris algorithm is used to extract feature points from preprocessed infrared images. The Harris algorithm is a corner detection algorithm based on the local autocorrelation function of image gradients, which can effectively identify corners in an image. These corners are often used as feature points due to their uniqueness within the image.

[0085] S4. Feature Point Optimization and Classification

[0086] Bilateral threshold filtering is employed to further optimize feature points in the infrared image. This method not only reduces noise but also maintains edge sharpness. The optimized feature points are then classified using the K-Means algorithm, grouping them for more accurate processing and analysis.

[0087] S5, 3D point cloud processing

[0088] The RANSAC algorithm is used to segment the surface of the marker from the 3D point cloud. This algorithm effectively identifies and removes outliers, retaining the most representative planar data. Next, the Alpha Shapes algorithm is used to extract the edges of the marker. This algorithm forms closed boundaries based on the 3D point cloud data, thereby identifying the outer contour. Through these processing steps, the edges and shapes of the marker can be accurately located from the point cloud.

[0089] S6. Solving for homonymous feature points

[0090] Finally, the intersection points obtained from the infrared image and the 3D point cloud are combined to solve for corresponding feature points. This step aligns the feature points from the two data sources through geometric transformation and registration techniques, thereby finding the corresponding points of the same physical location in different sensor data.

[0091] The method of this invention utilizes the fusion of heterogeneous data to improve the accuracy of feature point detection and the stability of the system. Through precise preprocessing and feature point extraction techniques, it is possible to accurately match corresponding feature points in different sensor data, which is crucial for the calibration and application of multi-sensor systems. Furthermore, this method is applicable to a wide range of application scenarios, such as autonomous driving, robot navigation, and augmented reality, providing an efficient and reliable technical solution.

[0092] Please see the appendix Figure 3 In a preferred embodiment of the present invention, when designing the "cross-shaped dual-light" marker, the emissivity characteristics of different materials in infrared thermal imagers and structured light cameras are fully utilized. From the material emissivity table, it can be seen that the emissivity of aluminum sheets is relatively low, only 0.05-0.07, while the emissivity of plastic sheets is very high, at 0.94, showing a significant difference. When the markers are placed adjacent to each other, the infrared thermal imager can capture clear images, but the structured light camera cannot acquire a three-dimensional point cloud containing the same feature points. To solve this problem, a 20mm thick "cross" protrusion is placed in the center of the aluminum plate, and a matte black plastic sheet is attached to the top. To improve the imaging effect, matte black plastic sheets are also attached to the perimeter of the aluminum plate.

[0093] This allows for simultaneous clear imaging in two imaging systems. This involves selecting materials that exhibit good radiometric contrast in both the infrared and visible light bands. The markers are designed to produce clear, distinguishable images under different sensors. This approach not only improves the overall imaging performance but also helps meet the needs of multi-sensor systems, thereby acquiring more comprehensive target information.

[0094] In a preferred embodiment of the present invention, the preprocessing step employs two main filtering techniques to optimize the infrared image and 3D point cloud data, in order to facilitate subsequent feature point extraction and analysis.

[0095] Gaussian filtering for denoising infrared images

[0096] Infrared image preprocessing includes denoising the image using Gaussian filtering to improve image quality and reduce processing errors. Gaussian filtering is a linear smoothing filter suitable for eliminating Gaussian noise while preserving image edges. This filtering process uses a one-dimensional Gaussian kernel function to process each pixel in the image; the specific mathematical expression is as follows:

[0097]

[0098] Where G(i,j) is a one-dimensional Gaussian kernel function, defined as:

[0099]

[0100] Where σ is the standard deviation of the Gaussian distribution, and its value is selected according to the actual denoising requirements to achieve the best image smoothing effect.

[0101] Statistical filtering of 3D point clouds to remove discrete points

[0102] For 3D point clouds, it is necessary to remove discrete points to improve the accuracy of the solution. Discrete points may be outliers caused by sensor noise, incomplete scanning, or other factors. Statistical filtering is used to remove discrete points by calculating the statistical information of the neighborhood points of each point to determine whether the point is discrete. This method typically uses the mean and standard deviation to detect outliers and removes points that exceed a threshold. This method includes the following steps:

[0103] First, statistical filtering is used to remove discrete points from the point cloud. Given a point P in the point cloud... i (x i ,y i ,z i Statistical filtering first calculates the statistical information of a point within a specified radius's neighborhood, typically using the mean and standard deviation. The set of points within the neighborhood is denoted as N. iFirst, calculate the mean μ. i :

[0104]

[0105] Then calculate the standard deviation σ. i :

[0106]

[0107] Among them, P j,z Point P j The z-coordinate, and |N i | indicates the number of points in the neighborhood.

[0108] After calculating the mean and standard deviation, the threshold T is defined:

[0109] T = μ i ±k·σ i

[0110] Here, k is a user-defined parameter used to adjust the sensitivity of the threshold.

[0111] If z i >T or z i <- T, then P is considered i These are discrete points. Through this process, statistical filtering can effectively identify and remove points with abnormal height variations within a given radius.

[0112] Through these optimized preprocessing steps, embodiments of the present invention can effectively improve the quality of data processing, providing a more accurate data foundation for subsequent feature point matching and the solving of corresponding feature points. Gaussian filtering ensures the clarity and accuracy of infrared images, while statistical filtering helps remove outliers from point cloud data that may interfere with analysis. These preprocessing measures are key steps in achieving efficient and reliable multi-sensor data fusion, and have a significant impact on improving the overall performance and stability of the system.

[0113] In a preferred embodiment of the present invention, the feature point extraction process utilizes the Harris algorithm, a widely used algorithm for corner detection, particularly suitable for feature point extraction in infrared images, as it can effectively identify boundaries and corners in images caused by temperature differences. The following are the detailed steps and principles of the Harris algorithm applied in this invention:

[0114] 1. Calculate the gradient

[0115] First, the gradient I of each pixel in the preprocessed infrared image is calculated. x and I y The formula is as follows:

[0116]

[0117] Where I is the intensity of the image, I x and I y These are the gradients of the image in the x and y directions, respectively.

[0118] This step is fundamental to identifying potential corner locations in an image, as corner regions typically exhibit significant variations along the gradient direction.

[0119] The Harris algorithm is used to calculate the gradient I of each pixel in the infrared image. x and I y ;

[0120] 2. Establish the structure matrix

[0121] Using the calculated gradient values, construct the structure matrix M:

[0122]

[0123] Here, w(i,j) is a Gaussian window function used to assign different weights to pixels at different locations, typically with the center point receiving the highest weight. This weighting process helps to enhance the influence of local features, thereby making corner detection more accurate.

[0124] 3. Calculate the response function at the corner point.

[0125] Calculate the corner response function R of the structure matrix M:

[0126] R = det(M) - k·trace 2 (M)

[0127] Where det(M) is the determinant of matrix M, trace(M) is the trace of matrix M, and k is a constant that affects the sensitivity of the response function. The higher the value of the response function R, the more likely the corresponding pixel is to be a corner point.

[0128] 4. Corner point determination

[0129] If the value of R is greater than a preset threshold, the corresponding pixel is identified as a corner point. The setting of this threshold depends on the characteristics of the image and the application requirements; a higher threshold means that only the most prominent corner points will be detected.

[0130] The Harris algorithm is used to extract corner points from infrared images, making feature point detection not only accurate but also robust, especially in scenes with high temperature contrast. Furthermore, the precise calculation of the structure matrix and response function enables the algorithm to effectively distinguish between real corner points and noise, thereby improving the accuracy of subsequent data fusion and feature point matching. This method is suitable for various applications requiring high-precision feature point detection, such as environmental perception in autonomous vehicles, event detection in security monitoring systems, and other machine vision systems.

[0131] Please see the appendix Figure 4 In a preferred embodiment of the present invention, the steps of optimizing feature points in an infrared image using bilateral threshold filtering and then classifying the optimized feature points using the K-Means algorithm include:

[0132] 1. Optimization of bilateral threshold filtering

[0133] The extracted feature points are then optimized using bilateral thresholding. Bilateral thresholding is an edge-preserving filtering technique that combines spatial proximity and pixel difference to effectively remove noise while retaining edge information. This step is crucial because it not only removes noise from the image but also preserves important edge information of the feature points, which is essential for subsequent feature point classification and matching.

[0134] 2. K-Means Algorithm Classification

[0135] The K-Means algorithm is used to classify the optimized feature points. K-Means is a widely used clustering algorithm that iteratively divides data points into K clusters to minimize the distance between each point and its cluster centroid. The specific process is as follows:

[0136] Calculate the Euclidean distance: For any two feature points p and q, calculate the Euclidean distance between them.

[0137]

[0138] This distance measures the similarity between feature points in the feature space.

[0139] Calculate the cluster center point:

[0140]

[0141] Among them, C k It is the set of data points in the k-th cluster, c k It is the mean of these points, representing the center of the cluster.

[0142] Optimize the objective function:

[0143]

[0144] The goal is to minimize the sum of squares of the distances from each point to its cluster center, thereby determining the optimal center point for each cluster.

[0145] 3. Solving for the least squares method by fitting the line and finding the intersection point.

[0146] After classifying the feature points, the least squares method is used to fit a straight line to each classified cluster to find the intersection points of the markers. This step uses mathematical methods to determine the best-fit line, thereby accurately calculating the intersection points of different lines. These intersection points are the corresponding feature points, which are crucial for further image analysis and data fusion.

[0147] This method not only improves the detection accuracy of feature points but also optimizes the data processing workflow. Accurate classification and precise intersection point determination of feature points are crucial for improving the overall performance and reliability of image processing systems, especially in technical fields requiring high-precision positioning and recognition, such as autonomous driving, robot navigation, and security monitoring. Furthermore, this method increases the automation level of feature point processing, reduces the need for manual intervention, and increases the efficiency of the processing.

[0148] Please see the appendix Figure 5 In a preferred embodiment of the present invention, the RANSAC algorithm and the Alpha Shapes algorithm are used to process the preprocessed 3D point cloud data to accurately segment the surface of the marker and extract its edges. This processing flow effectively utilizes the advantages of these two algorithms to improve the accuracy and efficiency of 3D data processing. The specific steps of this embodiment are described in detail below:

[0149] RANSAC algorithm for segmenting marker surfaces

[0150] 1. Random point selection and plane equation calculation:

[0151] Three points, P1(x1,y1,z1), P2(x2,y2,z2), and P3(x3,y3,z3), are randomly selected from the 3D point cloud.

[0152] Calculate the coefficients A, B, C, D of the plane defined by these three points using the following formula:

[0153] A=(y2-y1)(z3-z1)-(z2-z1)(y3-y1)

[0154] B = (x2 - x1)(z3 - z1) - (z2 - z1)(x3 - x1)

[0155] C=(x2-x1)(y3-y1)-(y2-y1)(x3-x1)

[0156] D = -(Ax1 + By1 + Cz1)

[0157] 2. Interior point calculation and model validation:

[0158] For each point in the point cloud, calculate its vertical distance l to the plane mentioned above;

[0159] The distance is compared with a preset tolerance error e to determine which points belong to the interior.

[0160] Repeat this process to find the plane with the largest set of interior points through iterative optimization, which is the optimal model.

[0161] Alpha Shapes algorithm extracts edges

[0162] 1. Edge extraction preparation:

[0163] The given plane is processed by projecting the point cloud data onto the plane. A point P(x,y) is selected, and all points that are less than 2a away from this point in the 2D plane are found to form a point set Q.

[0164] 2. Center calculation and edge determination:

[0165] For each point x1, y1 in the point set Q, calculate the possible center O1(x1, y1). o1 ,y o1 ) and O2(x o2 ,y o2 Use the following formula:

[0166]

[0167] in, S 2 =(x-x1) 2 +(y-y1) 2 .

[0168] 3. Edge Determination:

[0169] The AlphaShapes algorithm is used to determine the edges of the point cloud data based on the calculated center and radius of the circle.

[0170] This embodiment provides an efficient 3D point cloud processing method that combines the RANSAC and AlphaShapes algorithms, enabling accurate segmentation and extraction of landmark edges from complex backgrounds. This method not only enhances data processing accuracy but also improves processing speed, making it suitable for various technical fields requiring high-precision 3D point cloud processing, such as machine vision, automated production, robot navigation, and virtual reality.

[0171] In a preferred embodiment of the present invention, the application of the Alpha Shapes algorithm is not limited to extracting the edges of markers, but further includes a step of determining whether point P is a contour point, thereby enhancing the accuracy of edge detection. A detailed description of this step is as follows:

[0172] Contour point determination in the Alpha Shapes algorithm

[0173] 1. Calculation of the distance between the centers:

[0174] For the identified point set Q, after calculating the circle centers O1 and O2, the distance from each point in the point set to these two circle centers is further measured.

[0175] 2. Contour point recognition:

[0176] If all points in the point set Q are at a distance greater than the radius a of the circle from O1 and O2, then the initial point P(x,y) is considered a contour point. This means that point P lies outside the closed contour formed by the point set Q, marking a clear boundary point.

[0177] 3. Iterative process:

[0178] If one or more points in the point set Q are within a distance of radius a, this indicates that P may not be a contour point. In this case, it is necessary to traverse the remaining points in Q and recalculate the center O1 and O2 for each new point as a potential contour point.

[0179] Repeat the above distance detection and center recalculation process until a contour point is found that satisfies the condition that the distance between all points is greater than a, or confirm that P is not a contour point.

[0180] This enhanced Alpha Shapes algorithm step makes the edge extraction process more accurate, ensuring that only contour points are correctly identified, thereby reducing errors and uncertainties in subsequent processing. This accurate contour detection is particularly important for applications requiring highly precise 3D modeling and analysis, such as engineering modeling, geographic information systems (GIS), biomedical imaging, and other fields that require accurate edge recognition.

[0181] In a preferred embodiment of the present invention, the detailed steps for solving for corresponding feature points by combining the intersection points obtained from the infrared image and the 3D point cloud are as follows:

[0182] Geometric transformations and registration

[0183] 1. Geometric transformations:

[0184] Geometric transformations are performed on infrared images and 3D point cloud data to ensure that the two datasets are compared and analyzed in the same coordinate system. This step typically involves rotation, translation, and scaling operations to align the two data sources to have the same scale and orientation.

[0185] 2. Data registration:

[0186] After geometric transformation, data registration is performed, a crucial process that involves maximizing the correspondence between feature points in the infrared image and feature points in the 3D point cloud. Registration algorithms may employ feature-based methods, such as the Iterative Closest Point (ICP) algorithm, or other advanced image registration techniques to ensure spatial consistency between the two datasets.

[0187] Feature point matching and pairing of feature points with the same name

[0188] 1. Feature point matching:

[0189] Using the registered data, feature point matching is performed. This step uses algorithms to identify and match feature points in the infrared image with corresponding feature points in the 3D point cloud. The matching process typically depends on the geometric properties and relative positions of the feature points, as well as any texture or color information that may be included.

[0190] 2. Pairing of feature points with the same name:

[0191] After successfully matching feature points, we further identify those feature points with the same name that appear in both the infrared image and the 3D point cloud. These feature points are crucial for subsequent analysis, such as object recognition, scene reconstruction, or localization in navigation systems.

[0192] Through accurate geometric transformation, precise data registration, and effective feature point matching, this invention achieves high consistency and fusion of data from different sensors, thereby improving the accuracy and reliability of the system in practical applications. Furthermore, accurate pairing of corresponding feature points provides strong support for multi-sensor fusion, enabling the system to provide more accurate analysis results in complex environments. This is particularly suitable for advanced applications requiring precise environmental perception, such as autonomous vehicles, robotics, and augmented reality.

[0193] In a preferred embodiment of the present invention, the process of solving for corresponding feature points by combining the intersection points obtained from infrared images and 3D point clouds not only includes geometric transformation and registration, but also involves using iterative optimization methods to further improve registration accuracy. Detailed steps are as follows:

[0194] Iterative optimization methods improve registration accuracy

[0195] 1. Initial registration:

[0196] First, based on the geometric transformation and data registration steps described earlier, preliminary registration is performed to align the infrared image and 3D point cloud data to a unified coordinate system.

[0197] 2. Error Measurement and Adjustment:

[0198] The effectiveness of the initial registration is evaluated using some error metric (such as mean square error, surface distance difference, or other similarity measure). Based on this evaluation, parameters or areas that need adjustment are identified.

[0199] 3. Iterative optimization:

[0200] Iterative optimization techniques, such as the Iterative Closest Point (ICP) algorithm, are applied to progressively optimize transformation parameters (including rotation, translation, and scaling) to minimize the aforementioned error metrics. This process may involve multiple iterations, with each iteration adjusting the parameters based on the previous result, until a preset accuracy standard is reached or the improvement becomes insignificant.

[0201] In each iteration, the matching between feature points is recalculated and the registration parameters are updated to ensure the best correspondence between feature points.

[0202] 4. Convergence Confirmation:

[0203] Regularly check changes in error metrics to confirm whether convergence conditions have been met, i.e., error improvement has stabilized or reached the minimum error threshold allowed by the system.

[0204] By introducing an iterative optimization method, this embodiment can significantly improve the registration accuracy of infrared images and 3D point cloud data. This high-precision registration is key to achieving accurate matching of corresponding feature points. More importantly, it allows the system to maintain high accuracy and robustness when facing complex scenes and changing environmental conditions.

[0205] Furthermore, the iterative optimization method ensures that optimal registration can be achieved through gradual adjustments, even under less than ideal initial conditions. This approach significantly improves the practicality and efficiency of multi-sensor data fusion technology, making it particularly suitable for fields such as high-precision measurement, robot navigation, automated monitoring, and advanced computer vision systems.

[0206] The device for solving corresponding feature points for heterogeneous data fusion described below can be referred to in correspondence with the method for solving corresponding feature points for heterogeneous data fusion described above.

[0207] The present invention also provides an apparatus comprising an infrared thermal imager, a structured light camera, and a processing unit, the apparatus comprising:

[0208] The data acquisition module is used for acquiring infrared images and 3D point cloud data;

[0209] The data preprocessing module is used to denoise and remove discrete points from the collected data;

[0210] The feature point extraction module is used to extract feature points from the preprocessed data;

[0211] The feature point classification module is used to classify and optimize the extracted feature points;

[0212] The feature point solving module is used to solve for corresponding feature points by combining feature points from infrared images and 3D point clouds.

[0213] The device in this embodiment can be used to execute the above method embodiments, and its principle and technical effects are similar, so they will not be described again here.

[0214] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for solving for homonymous feature points in heterogeneous data fusion, characterized in that, Includes the following steps: Infrared thermal imagers and structured light cameras were used to collect data on scenes containing "double light cross" markers; Preprocess the acquired infrared images and 3D point cloud data; Feature points were extracted from the preprocessed infrared image using the Harris algorithm. The feature points in the infrared image are optimized using bilateral threshold filtering, and then the optimized feature points are classified using the K-Means algorithm. The RANSAC algorithm was used to segment the marker surface from the preprocessed 3D point cloud, and the Alpha Shapes algorithm was used to extract the edges. The intersection points of the 3D point cloud were obtained by fitting straight lines. We solve for the corresponding feature points by combining the intersection points obtained from infrared images and 3D point clouds.

2. The method for solving for homonymous feature points for heterogeneous data fusion according to claim 1, characterized in that, The preprocessing steps include: The mathematical expression for using Gaussian filtering to denoise infrared images is: Where G(i,j) is a one-dimensional Gaussian kernel function, defined as: Where σ is the standard deviation of the Gaussian distribution; Statistical filtering methods are used to remove discrete points from 3D point cloud data, including: Calculate point P i (x i ,y i ,z i The mean μ of the neighborhood within the specified radius i and standard deviation σ i ; Remove those that satisfy z i >μ i +k·σ i or z i <μ i -k·σ i The discrete points, where k is a user-defined parameter.

3. The method for solving for homonymous feature points for heterogeneous data fusion according to claim 1, characterized in that, The steps for extracting feature points from the preprocessed infrared image using the Harris algorithm include: The Harris algorithm is used to calculate the gradient I of each pixel in the infrared image. x and I y ; Establish the structure matrix M: Where w(i,j) is the Gaussian window function; Calculate the response function R at the corner point: R=det(M)-k trace 2 (M) Where det(M) is the determinant of matrix M, trace(M) is the trace of matrix M, and k is a constant; If R is greater than the set threshold, it is determined to be a corner point.

4. The method for solving for homonymous feature points for heterogeneous data fusion according to claim 1, characterized in that, The steps of optimizing feature points in an infrared image using bilateral threshold filtering and then classifying the optimized feature points using the K-Means algorithm include: The extracted feature points are optimized using bilateral threshold filtering to preserve edge information and filter out noise. The K-Means algorithm is used to classify the optimized feature points, and the Euclidean distance between any two points is calculated: Calculate the center point of each cluster: Iterative optimization of objective function J: Where K is the number of clusters, C k c is the number of data points in the k-th cluster. k It is the center point of the k-th cluster; The intersection points of the markers are solved by fitting a straight line using the least squares method.

5. The method for solving for homonymous feature points for heterogeneous data fusion according to claim 1, characterized in that, The steps of segmenting the marker surface from the preprocessed 3D point cloud using the RANSAC algorithm, extracting edges using the Alpha Shapes algorithm, and obtaining the intersection points of the 3D point cloud using fitted straight lines include: The surface of the marker is segmented using the RANSAC algorithm. Three points, P1(x1,y1,z1), P2(x2,y2,z2), and P3(x3,y3,z3), are randomly selected from the point cloud. The coefficients A, B, C, and D of the plane equation are calculated. A=(y2-y1)(z3-z1)-(z2-z1)(y3-y1) B = (x2 - x1)(z3 - z1) - (z2 - z1)(x3 - x1) C=(x2-x1)(y3-y1)-(y2-y1)(x3-x1) D = -(Ax1 + By1 + Cz1) Calculate the distance l from each point in the point cloud to the plane, compare it with the tolerance error e, and count the number of inliers n; Iteratively select three new points and repeat the above steps until the optimal planar model is found. The AlphaShapes algorithm is used to extract the edges of the marker. Let the radius of the rolling circle be *a*. For point P(x,y), the set of points Q whose distance from P is less than 2*a* is searched, and the center O1(x,y) of the circle is calculated. o1 ,y o1 ) and O2(x o2 ,y o2 ): in, S 2 =(x-x1) 2 +(y-y1) 2 .

6. The method for solving for homonymous feature points for heterogeneous data fusion according to claim 5, characterized in that, The AlphaShapes algorithm further includes determining whether point P is a contour point. If the distance from all points in the point set Q to O1 and O2 is greater than a, then P is a contour point; otherwise, the remaining points are traversed and the center of the circle is recalculated.

7. The method for solving for homonymous feature points for heterogeneous data fusion according to claim 1, characterized in that, The steps for solving for corresponding feature points by combining the intersection points obtained from the infrared image and the 3D point cloud include: Perform geometric transformation and registration on feature points of infrared images and 3D point clouds; Pairing of feature points with the same name is performed based on the transformed feature point matching.

8. The method for solving for homonymous feature points for heterogeneous data fusion according to claim 7, characterized in that, The step of solving for corresponding feature points by combining the intersection points obtained from infrared images and 3D point clouds also includes improving registration accuracy through iterative optimization methods.

9. The method for solving for homonymous feature points for heterogeneous data fusion according to claim 1, characterized in that, The marker includes an aluminum base plate with a cross-shaped boss on its surface, the boss being made of aluminum; Black matte plastic sheets are attached above the boss and around the aluminum base plate to enhance the imaging contrast in the infrared thermal imager, while ensuring the shape's recognizability in the structured light camera.

10. A device for solving for identical feature points in heterogeneous data fusion, comprising an infrared thermal imager, a structured light camera, and a processing unit, characterized in that, The processing unit is configured to perform the method according to any one of claims 1-9, and the apparatus includes: The data acquisition module is used for acquiring infrared images and 3D point cloud data; The data preprocessing module is used to denoise and remove discrete points from the collected data; The feature point extraction module is used to extract feature points from the preprocessed data; The feature point classification module is used to classify and optimize the extracted feature points; The feature point solving module is used to solve for corresponding feature points by combining feature points from infrared images and 3D point clouds.

Images