Pose solving method for thermal infrared imager and structured light camera

Abstract

The invention provides a pose solving method for a thermal infrared imager and a structured light camera. The pose solving method comprises the following steps: designing a dual-light circular ring marker; the thermal infrared imager and the structured light camera are controlled to collect information of the dual-light circular ring marker, and an infrared image and three-dimensional point cloud data are obtained; the infrared image and the three-dimensional point cloud data are preprocessed, and noise is removed; calculating circle center coordinates of each circular ring according to the infrared image, and generating a first group of circle center data; calculating circle center coordinates of each circular ring according to the three-dimensional point cloud, and generating a second group of circle center data; and performing external parameter solution based on the first group of circle center data and the second group of circle center data to obtain a rotation matrix and a translation vector between the thermal infrared imager and the structured light camera. According to the method, the accuracy and precision of solving the pose between the thermal infrared imager and the structured light camera are improved.

Description

Technical Field

[0001] This invention belongs to the field of heterogeneous data fusion technology, specifically relating to a pose solution method for infrared thermal imagers and structured light cameras. Background Technology

[0002] In heterogeneous data fusion between infrared thermal imagers and structured light cameras, pose determination between the two is a crucial task, involving the detection and matching of corresponding feature points between 2D images and 3D point cloud data. Corresponding feature points between an image and a point cloud refer to feature points on the same corresponding object; these points are image pixels in the 2D image and points with spatial coordinates (x, y, z) in the 3D point cloud. In image processing, traditional feature point extraction algorithms include SIFT, SURF, and ORB, while matching algorithms can employ nearest neighbor matching or bidirectional nearest neighbor matching. Meanwhile, deep learning for feature extraction and matching has also received widespread attention. In point cloud processing, traditional feature point extraction algorithms include ISS, SIFT-3D, and Harris 3D, while feature descriptor generation algorithms include FPFH, PFH, and SHOT. In the pose determination process for infrared thermal imagers and structured light cameras, it is necessary to find corresponding points in the 3D point cloud on the 2D plane image; currently, a high-precision and robust method is still lacking for this process. Furthermore, the pose determination problem is usually modeled as an optimization problem with the goal of minimizing the reprojection error. Common optimization methods include the least squares method and the Levenberg-Marquardt algorithm.

[0003] When using different sensors or cameras, the imaging effect of markers may vary on different devices due to differences in their imaging principles, photosensitive elements, and lens performance. This difference may make it difficult to achieve clear imaging simultaneously in certain situations, thus affecting the accuracy of camera calibration. Designing specific markers and corresponding feature point extraction algorithms is crucial when solving the matching problem between corresponding feature points in 2D images and 3D point clouds. This process directly affects the accuracy and precision of pose determination between infrared thermal imagers and structured light cameras.

[0004] Therefore, a pose-solving method for infrared thermal imagers and structured light cameras is needed to overcome the above shortcomings. Summary of the Invention

[0005] The purpose of this invention is to provide a pose solving method for infrared thermal imagers and structured light cameras, thereby improving the accuracy and precision of pose solving between infrared thermal imagers and structured light cameras.

[0006] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0007] A pose determination method for infrared thermal imagers and structured light cameras includes the following steps:

[0008] S1. Design a dual-light circular marker, the dual-light circular marker including a back plate and at least four circular protrusions set on the back plate, the centers of the four circular protrusions cannot be located on the same straight line, and the radiation coefficient of the material laid on the circular protrusions is not less than 0.45 different from that of the back plate material.

[0009] S2. Control the infrared thermal imager and structured light camera to collect information of the dual-light ring marker and obtain infrared images and three-dimensional point cloud data;

[0010] S3. Preprocess the infrared image and 3D point cloud data to remove noise;

[0011] S4. Calculate the center coordinates of each ring based on the infrared image to generate the first set of center data;

[0012] S5. Based on the coordinates of the center of each ring in the 3D point cloud computing, generate the second set of center data;

[0013] S6. Based on the first set of center data and the second set of center data, solve the external parameters to obtain the rotation matrix and translation vector between the infrared thermal imager and the structured light camera.

[0014] Furthermore, the backplate of the dual-light circular marker is an aluminum plate, and a matte black plastic sheet is set on the circular protrusion. A matte black plastic sheet is also set around the perimeter of the aluminum plate. This configuration allows for clear imaging of the circular shape and the background, and the matte black plastic sheet along the edge of the aluminum plate distinguishes it from the background.

[0015] Furthermore, there are four annular protrusions, the centers of which form the four vertices of a square, and the annular protrusions maintain a set distance from each other; the height of the annular protrusions ranges from 5 to 30 mm. This configuration ensures clear imaging from the infrared thermal imager and structured light camera, facilitates subsequent calculations, and improves calculation accuracy and efficiency.

[0016] Furthermore, the step of calculating the center coordinates of each ring based on the infrared image to generate the first set of center data includes the following steps:

[0017] S100. The Hough gradient method is used to calculate the inner and outer center of each ring in the infrared image, generating a set of zero-order center coordinates.

[0018] S200. Randomly select M1 random groups from the zero-level circle center coordinate set. Each random group includes the four circle center coordinates corresponding to the four annexes. Calculate the circle center loss value for each random group, whereby the circle center loss value includes the distance constraint loss value L.edges and geometric constraint loss value L angles The calculation method is as follows:

[0019]

[0020]

[0021] L angles =(θ-θ) expected ) 2

[0022]

[0023] Among them, (u p ,v p Let p = 1, 2, 3, 4, and d be the coordinates of the center of the circle. p d is the distance between the centers of two adjacent circles in the horizontal or vertical direction. expected θ is the average distance between adjacent center points. expected The angle is 90 degrees, and θ is the angle between the two diagonals of the square with the center coordinates as its vertices;

[0024] S300. Determine the group with the smallest center loss value among M1 random groups as the reference group. Using the center coordinates of each circle in the reference group as the center, randomly generate center sampling points within a pixel range. Randomly select M2 sampling groups, each sampling group including four center sampling points corresponding to the four circles. Calculate the center loss value of each sampling group. Output the sampling group with the smallest center loss value as the first group of center data.

[0025] Furthermore, the step of using the Hough gradient method to calculate the inner and outer center of each ring in the infrared image and generate a set of zero-order center coordinates specifically includes the following steps:

[0026] Perform Canny edge detection on the original image to obtain a binarized edge detection image;

[0027] Apply the Sobel operator to the original image to calculate the gradient value and direction of each pixel;

[0028] Create a center space and use an accumulator to accumulate the count of each occurrence of the center;

[0029] Traverse each non-zero pixel in the edge detection image, draw a line along the gradient direction of each non-zero pixel, and increment the count of the accumulator of all the circles the line passes through.

[0030] Sort the counts of the accumulators of all the centers in the center space;

[0031] Using the outer and inner radii of the circular boss of the dual-ring marker as constraints, the centers of the circles that do not meet the radius constraints are removed. Then, according to the accumulator count, the first four sets of inner circle centers that meet the inner circle radius constraints and the first four sets of outer circle centers that meet the outer circle radius constraints are determined, forming a zero-level circle center coordinate set.

[0032] Furthermore, the step of generating a second set of center data based on the center coordinates of each annulus using the three-dimensional point cloud computing specifically includes the following steps:

[0033] Construct a three-dimensional Cartesian coordinate system O-XYZ containing the three-dimensional point cloud;

[0034] Select at least three non-collinear points in the 3D point cloud, determine the initial plane where a double-light ring marker is located, and construct the equation of the initial plane.

[0035] Calculate the normal vector of the initial plane and convert it to a unit normal vector;

[0036] Calculate the first rotation matrix so that the unit normal vector is aligned with the Z-axis, and obtain the X′OY′ plane after the initial plane is rotated;

[0037] Apply the first rotation matrix to all points in the 3D point cloud to transform them to the new coordinate system;

[0038] The 3D point cloud transformed to the new coordinate system is projected onto the X′OY′ plane to obtain a 2D point set;

[0039] The center coordinates of each annulus are calculated based on the two-dimensional point set. The center coordinates of the three-dimensional annulus are obtained based on the normal vector of the initial plane and the first rotation matrix, generating the second set of center data.

[0040] Furthermore, the EPNP algorithm is used to solve the extrinsic parameters between the infrared thermal imager and the structured light camera.

[0041] Furthermore, the infrared image is denoised using Gaussian filtering;

[0042] The 3D point cloud data is denoised using statistical filtering, and the statistical filtering method is as follows:

[0043] For each point in the 3D point cloud, define a neighborhood, which is the set of all points within a defined radius centered at that point. Calculate the mean μ′ and standard deviation σ of the neighborhood.

[0044]

[0045]

[0046] Where n is the number of points in the neighborhood, p w This represents the coordinates of the w-th point in the neighborhood;

[0047] If |p w -μ′|>T, where T is the threshold value, then p is removed. w point.

[0048] Furthermore, the threshold T ranges from 1.5σ to 3σ.

[0049] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned pose solving method.

[0050] The beneficial effects of this invention compared to the prior art are as follows:

[0051] Based on the emissivity of different materials and the imaging characteristics of sensors, this invention designs a dual-light circular marker. The marker includes a back plate and a circular boss; the emissivity of the two materials differs significantly, and the boss is positioned at a certain height, enabling the marker to achieve clear imaging in various sensors. Furthermore, the designed dual-light circular marker can also be used for camera calibration in other wavelength bands.

[0052] This invention combines the characteristics of dual-light circular markers and designs a corresponding feature point solving algorithm, including an infrared image feature point extraction method based on Hough transform and a three-dimensional point cloud feature point extraction method based on planar mapping, which improves the accuracy of pose solving between cameras.

[0053] This invention proposes a center coordinate optimization method based on the Hough gradient method for infrared images of circular structures. After optimization and correction based on the geometric features of the circular ring, random sampling optimization is performed again to improve the center detection accuracy from the pixel level to the sub-pixel level. Attached Figure Description

[0054] The accompanying drawings, which form part of this specification, are provided to further illustrate embodiments of the invention and, together with the textual description, explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0055] Figure 1 A schematic diagram of an infrared thermal imager and structured light camera data acquisition system provided for a specific embodiment of the present invention;

[0056] Figure 2 A flowchart illustrating a pose determination method for infrared thermal imagers and structured light cameras, provided as a specific embodiment of the present invention.

[0057] Figure 3A schematic diagram of the dual-ring marker structure is provided for a specific embodiment of the present invention;

[0058] Figure 4 A flowchart of the infrared image feature point solving algorithm provided for a specific embodiment of the present invention;

[0059] Figure 5 The flowchart of the algorithm for solving 3D point cloud feature points is provided for a specific embodiment of the present invention.

[0060] The above figures include the following reference numerals:

[0061] 1. Dual-light circular marker; 2. Structured light camera; 3. Infrared thermal imager; 4. Industrial control computer; 11. Aluminum plate; 12. Circular boss. Detailed Implementation

[0062] Specific embodiments of the present invention will now be described in detail. In the following description, specific details are set forth for purposes of explanation and not limitation, in order to aid in a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced in other embodiments departing from these specific details.

[0063] It should be noted that, in order to avoid obscuring the invention with unnecessary details, only the device structure and / or processing steps closely related to the solution of the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0064] As one aspect of this invention, the proposed markers and algorithms are used for pose determination using infrared thermal imagers and structured light cameras. The system mainly includes an infrared thermal imager, a structured light camera, dual-light circular markers, and image and point cloud data preprocessing modules. The main principle of this pose determination method for infrared thermal imagers and structured light cameras is as follows:

[0065] 1. When designing dual-light circular markers, ensuring sufficient contrast for both infrared thermal imagers and structured light cameras is crucial. In infrared thermal imagers, contrast primarily depends on the temperature difference of the object, while in structured light cameras, geometric contrast is paramount. Material selection must carefully consider the reflectivity of both infrared thermal imagers and structured light cameras, ensuring the material neither reflects nor absorbs excessively under infrared or structured light conditions to avoid affecting feature point detection. The circular design exhibits strong robustness and numerous advantages due to its unique geometric characteristics. Preferably, the dual-light circular marker includes a backplate and at least four circular protrusions on the backplate. The centers of the four protrusions cannot be aligned, and the emissivity difference between the material on the protrusions and the backplate material is not less than 0.45.

[0066] 2. Infrared image preprocessing can optimize image quality, reduce noise, and highlight target features, thus making it better suited for subsequent tasks involving the calculation of corresponding feature points. Images acquired by thermal imagers may contain various types of noise, such as thermal noise and electronic noise. Infrared image preprocessing algorithms can effectively mitigate the impact of this noise.

[0067] 3. 3D point cloud preprocessing can remove noise from the point cloud, thereby improving the accuracy of subsequent marker center identification. When processing 3D point cloud data, noise typically manifests as outliers, sampling noise, and depth inconsistencies. The goal of preprocessing is to eliminate or reduce this noise, making subsequent 3D point cloud marker center identification more accurate and reliable. Using a distance threshold-based point cloud denoising method is a very effective strategy. It calculates the distance between points and compares it with a preset threshold to identify and remove outlier, noisy points.

[0068] 4. Infrared image center detection employs the Hough gradient method, an optimized circle detection algorithm that combines the results of Canny edge detection and Sobel gradient calculation. A two-dimensional Cartesian coordinate system UOV is constructed in the infrared image, with the origin O at the top left corner of the image, the U-axis pointing horizontally to the right, and the V-axis pointing horizontally downwards.

[0069] 4.1 Center detection

[0070] (1) Canny edge detection: First, Canny edge detection is performed on the original image to obtain a binarized edge detection image, which includes the edge pixels of the inner and outer circles of each ring.

[0071] (2) Sobel operator to calculate gradient: Apply the Sobel operator to the original image to calculate the gradient value G(u,v) and direction θ(u,v) of each pixel (u,v). These gradient values ​​are used to determine the tangent direction of each edge pixel.

[0072]

[0073]

[0074] Where G1(u,v) represents the horizontal gradient of pixel (u,v) and G2(u,v) represents the vertical gradient of pixel (u,v). The following formula is the calculation formula for G1(u,v) and G2(u,v), where Gx is the horizontal convolution kernel of the Sobel operator, Gy is the vertical convolution kernel of the Sobel operator, Gx(i,j) is the horizontal convolution kernel of the Sobel operator, and Gy(i,j) is the vertical convolution kernel of the Sobel operator.

[0075]

[0076]

[0077] Where i and j are integers, ranging from -1 to 1, μ and v represent the horizontal and vertical pixel values ​​of pixel (μ, v), I(μ, v) represents the pixel value of pixel (u, v), Gx(i, j) is the horizontal convolution kernel of the Sobel operator corresponding to i and j, and Gy(i, j) is the vertical convolution kernel of the Sobel operator corresponding to i and j.

[0078] (3) Initialize the center space: Create a center space V(α,β) and initialize it to zero. This space is used to accumulate the center of each occurrence (α,β). q ,β q The count of ).

[0079] (4) Accumulator Voting: Traverse each non-zero pixel in the Canny edge detection image (i.e., the edge pixels of each inner and outer circle of the ring), draw a line starting from the non-zero pixel along the gradient direction of that non-zero pixel (i.e., the perpendicular direction of the tangent), and mark all the centers (α) of the circles the line passes through. q ,β q The accumulator V(α) q ,β q The count in the () is incremented by one.

[0080] (5) Center estimation: Finally, based on the accumulator V(α) q ,β q By counting and sorting in α, we can obtain all possible center points. The count V(α) q ,β q The larger the position, the more likely it is to be the center of the circle.

[0081] (6) After obtaining the preliminary center coordinates, the noise feature points are filtered according to the geometric characteristics of the center on the marker to obtain the accurate center coordinates. Specifically, the obtained center coordinates are filtered according to the outer and inner radii of each annular boss to remove the center coordinates that do not meet the radius constraints. The center coordinates are then sorted according to the accumulator count to obtain the first four sets of inner circle center coordinates that meet the inner circle radius constraints and the first four sets of outer circle center coordinates that meet the outer circle radius constraints, forming a zero-level center coordinate set.

[0082] 4.2 Optimization of Center Coordinates

[0083] Due to image noise and edge sharpness, the inner and outer centers of the ring are not completely aligned. Based on the obtained center coordinates, the center position of the ring can be further optimized and corrected using the following methods to improve the accuracy of ring detection:

[0084] (1) Based on distance constraints and geometric constraints, randomly select a group from the detected inner and outer ring centers, calculate the center loss value of the four rings, and the distance constraint loss value L. edges and geometric constraint loss value L angles The calculation formula is as follows:

[0085]

[0086]

[0087] L angles =(θ-θ) expected ) 2

[0088]

[0089] Among them, (u p ,v p ) represents the center coordinates (in this embodiment, 1 to 4 represent the centers of the four rings respectively, with the upper left ring marked as 1, the upper right ring as 2, the lower left ring as 3, and the lower right ring as 4), d p d represents the distance between the centers of two adjacent circles in the horizontal or vertical direction (also the length of the four sides of a square with the center coordinates as its vertices). expected θ is the average distance. expected The angle is 90 degrees, and θ is the angle between the two diagonals of the square with the center coordinates as its vertices.

[0090] In this embodiment, M1 random groups are obtained through multiple random selections. Each random group includes the coordinates of the four center points of the four corresponding rings, and the center point loss value of each random group is calculated. Preferably, M1 is not less than 4; in this embodiment, M1 is 8.

[0091] (2) Select the four center coordinates with the smallest center loss value in the random group, take each center coordinate as the center, randomly generate corresponding sampling points within a pixel range, calculate the center loss value of the corresponding sampling points, and select the sampling point with the smallest center loss value as the final detected center coordinate.

[0092] loss = L edges +L angles

[0093] Where loss is the center loss value, L edges L represents the distance constraint loss value. angles This represents the geometric constraint loss value.

[0094] In this embodiment, the center sampling points are randomly generated within a pixel range based on the initially determined center coordinates, and M2 sampling groups are randomly selected. Each sampling group includes four center sampling points corresponding to the four annular protrusions. The center loss value of each sampling group is calculated, and the sampling group with the smallest center loss value is used as the final infrared image center data output.

[0095] 5. Center detection of the 3D point cloud: The planar point cloud of the dual-light ring marker is mapped to a new coordinate system with the same z-coordinate through coordinate transformation. In the new coordinate system, the point cloud is mapped to the X'OY' plane, and the center coordinates are detected by an algorithm. A 3D Cartesian coordinate system O-XYZ is constructed in the 3D point cloud data, with the upper left front of the 3D point cloud as the origin, the horizontal rightward direction as the X-axis, the vertical downward direction as the Y-axis, and the right-hand screw rule to determine the Z-axis. The transformed new coordinate system is denoted as O-X'Y'Z'.

[0096] The process of 3D point cloud center detection includes the following steps:

[0097] (1) Determine the plane equation: First, use at least three non-collinear points in the 3D point cloud to determine the equation of the plane containing the double-light ring marker. Assume the equation is ax + by + cz + d = 0, where (a, b, c) are the plane normal vectors and d is a constant.

[0098] (2) Calculate the normal vector: Determine the normal vector of the plane. And calculate the unit normal vector.

[0099] (3) Calculate the first rotation matrix: Calculate the first rotation matrix such that the unit normal vector Align with the Z-axis. The first rotation matrix will rotate the original plane to the new X′OY′ plane.

[0100] (4) Apply rotation matrix: Apply the first rotation matrix to all points in the 3D point cloud to transform them to a new coordinate system. In the new coordinate system, all points have the same z-coordinate.

[0101] (5) Projecting the point cloud onto the X′OY′ plane: In the new coordinate system, ignoring the z-coordinate, the point cloud is projected onto the X′OY′ plane to obtain a two-dimensional point set. After projection, the z-coordinate of all points is 0.

[0102] (6) Circle center detection algorithm: The Hough gradient algorithm is used to solve for the center coordinates of each ring in the two-dimensional point cloud image at this time, and the plane normal vector calculated in the above steps is used as the basis for the detection algorithm. By combining the first rotation matrix, we obtain the coordinates of the center of the three-dimensional annulus.

[0103] 6. Solve for extrinsic parameters based on the camera pinhole model and the EPNP algorithm. Find corresponding feature points in the infrared thermal imager and the structured light camera, i.e., project the 3D point in space onto the infrared thermal imager. Assume there is a 3D point with coordinates (X, Y, Z) in the camera coordinate system. The projection equation in the camera coordinate system is:

[0104]

[0105] Where is the K-infrared thermal imager intrinsic parameter matrix, R is the rotation matrix, t is the translation vector, and (u, v) are the pixel coordinates on the infrared image plane.

[0106] Rearranging the infrared image parameters and the 3D point cloud parameters yields a new equation:

[0107]

[0108] Among them, f u f v These represent the focal lengths along the U and V axes of the infrared thermal imager, in pixels, c u c v These represent the U and V axis offsets of the infrared thermal imager's optical axis in the image coordinate system, in pixels, r. ef Let t be the element in the e-th row and f-th column of the rotation matrix. x , t y , t z These are the translation vector elements along the X, Y, and Z axes, respectively. Further expansion yields:

[0109]

[0110] Simplified, we get:

[0111]

[0112]

[0113] A projection equation can be established for a single point in space, and multiple projection equations can be established for multiple corresponding feature points. Finally, the EPNP algorithm is used to solve the extrinsic parameters.

[0114] In this embodiment, the rotation matrix and translation vector between the infrared thermal imager and the structured light camera can be obtained by substituting the infrared imager center data and the three-dimensional point cloud center data of the four circular coordinates obtained in the aforementioned steps into the above formula.

[0115] Based on the imaging principles of infrared thermal imagers and structured light cameras, this invention designs a dual-light circular marker, which stabilizes the imaging quality while ensuring the accuracy of the image and point cloud. The dual-light circular marker can be used to solve for the corresponding feature points in the pose calibration of infrared thermal imagers and structured light cameras, and the pose parameters between the two devices can be solved using the EPNP algorithm.

[0116] As another aspect of the present invention, the technical solution of the present invention will be described in detail with reference to a specific embodiment.

[0117] In this embodiment, a pose determination method for infrared thermal imagers and structured light cameras is presented, using a dual-light ring marker such as... Figure 1 , 3 As shown, the structured light camera 2 and the infrared thermal imager 3 collect data from the dual-light circular marker 1, and then transmit the data to the industrial control computer 4 for data processing. The processing flow is as follows: Figure 2 As shown.

[0118] The first step was to design a dual-light circular marker. By fully utilizing the radiation characteristics of different materials in infrared thermal imagers and structured light cameras, the emissivity tables showed that aluminum had a relatively low emissivity (0.05-0.07), while plastic had a significantly higher emissivity (0.94). Placing these materials adjacent to each other allowed the infrared thermal imager to capture a clear image. Simultaneously, to enable the structured light camera to acquire a 3D point cloud containing corresponding feature points, a circular boss was placed in the center of the aluminum plate, and a matte black plastic sheet was attached to it. To further improve imaging, matte black plastic sheets were also attached around the perimeter of the aluminum plate. The specific structure is as follows: Figure 3 As shown, the dual-light circular marker includes an aluminum plate 11 and four circular protrusions 12 disposed on the aluminum plate. A matte black plastic sheet is placed on each of the circular protrusions, and matte black plastic sheets are also placed around the edges of the aluminum plate. The centers of the four circular protrusions form the four vertices of a square, and the circular protrusions maintain a predetermined distance from each other. The height of the circular protrusions ranges from 5 to 30 mm. This configuration allows for clear imaging of the circular shape and background, distinguishing it from the background compared to using matte black plastic sheets on the edges of the aluminum plate. It also facilitates subsequent calculations, improving calculation accuracy and efficiency.

[0119] The second step, image preprocessing, is crucial in solving for feature points in infrared images and 3D point clouds. Its main purpose is to improve the quality of the image and point cloud, reduce noise, and highlight key information, thereby enabling more effective feature point extraction. In image processing, to improve detection accuracy, preprocessing includes grayscale conversion, smoothing and denoising the image using a Gaussian kernel, edge detection using a Laplacian kernel, and image enhancement, which helps reduce noise and highlight circular structures.

[0120] The third step involves preprocessing the point cloud data. To improve the robustness and accuracy of the algorithm, discrete points or outliers introduced by sensor noise, incomplete scanning, or other factors must be removed. In this embodiment, statistical filtering is used, which can effectively identify and remove these outliers.

[0121] For each point p in a 3D point cloud, define a neighborhood N, which is the set of all points within a specified radius r centered at p. Let the number of points in N be n, and the points in each neighborhood be represented as p1, p2, ..., p... n .

[0122] Calculate the mean μ′ and standard deviation σ of the neighborhood N:

[0123]

[0124]

[0125] Based on the above statistics, a threshold T is set to determine which points are discrete points. For any point p in the neighborhood N... w If |p w -μ′|>T, assuming p w These are discrete points, which are removed in subsequent processing. In dense point clouds, the threshold T is between 1.5σ and 3σ.

[0126] The third step involves identifying the center of the marker in the infrared image. This is done using the Hough transform circle detection algorithm, followed by intrinsic parameter calibration. The detection process is as follows: Figure 4 As shown.

[0127] The fourth step involves identifying the center of the marker circle from the point cloud data. This is achieved by mapping the planar point cloud of the dual-light ring marker to a new coordinate system with the same z-coordinate. In this new coordinate system, the point cloud is mapped to the X'OY' plane. The center coordinates are then obtained using the Hough gradient transform algorithm on the two-dimensional plane. Finally, the three-dimensional point cloud coordinates of the ring center are obtained by solving for the z-coordinate and the first rotation matrix, among other parameters. The detection process is as follows: Figure 5 As shown.

[0128] The fifth step uses the EPNP algorithm to solve for the extrinsic parameters. After obtaining multiple pairs of corresponding feature points, a linear equation system is established by projecting the equations of multiple 3D points:

[0129] Ap = b

[0130] Where A is a matrix consisting of camera intrinsic parameters K, transformation matrix R, and point cloud coordinates X, p is a vector of rotation and translation parameters to be solved, and b is the known two-dimensional image coordinates.

[0131] Then perform singular value decomposition on A:

[0132] A·=U∑V T

[0133] Where U is an orthogonal matrix, Σ is a diagonal matrix, and the elements on the diagonal are called singular values, V T It is another orthogonal matrix.

[0134] Then, the least squares solution is obtained, and the linear system is rewritten as follows:

[0135] ∑V T p = U T b

[0136] Let m = V T p, s = U T b, therefore we can obtain:

[0137] ∑m=s

[0138] Since Σ is a diagonal matrix, it can be solved by taking the reciprocal of its non-zero diagonal elements and then multiplying it by U. T Multiplying b by v yields p'. The final least-squares solution p can be obtained by multiplying p' by V. This least-squares solution provides the camera's rotation and translation parameters, thus completing the camera pose calculation.

[0139] The features described and / or illustrated above with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, and / or in combination with or in lieu of features in other embodiments.

[0140] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, whole, step, or component, but does not exclude the presence or addition of one or more other features, wholes, steps, components, or combinations thereof.

[0141] Many features and advantages of these embodiments are apparent from this detailed description, and therefore the appended claims are intended to cover all such features and advantages of these embodiments that fall within their true spirit and scope. Furthermore, since many modifications and alterations will readily occur to those skilled in the art, the embodiments of the invention are not intended to be limited to the precise structures and operations illustrated and described, but rather to encompass all suitable modifications and equivalents falling within their scope.

[0142] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0143] The parts of this invention not described in detail are techniques known to those skilled in the art.

Claims

1. A pose solving method for infrared thermal imager and structured light camera, characterized in that, The method comprises the following steps: S1, designing a double-light circular ring marker, the double-light circular ring marker comprising a back plate and at least four circular ring bosses arranged on the back plate, the centers of the four circular ring bosses cannot be located on the same straight line, and the radiation coefficient of the material applied on the circular ring bosses and the material of the back plate differs by no less than 0.45; S2, controlling an infrared thermal imager and a structured light camera to collect information of the double-light circular ring marker, and obtaining an infrared image and three-dimensional point cloud data; S3, respectively pre-processing the infrared image and the three-dimensional point cloud data; S4, calculating the center coordinates of each circular ring according to the infrared image, and generating a first group of center data; S5, calculating the center coordinates of each circular ring according to the three-dimensional point cloud, and generating a second group of center data; S6, performing extrinsic parameter solving based on the first group of center data and the second group of center data, and obtaining a rotation matrix and a translation vector between the infrared thermal imager and the structured light camera.

2. The pose solving method of claim 1, wherein, The back plate of the double-light circular ring marker is an aluminum plate, black matte plastic sheets are arranged on the circular ring bosses, and black matte plastic sheets are arranged on the edges of the aluminum plate.

3. The pose solving method of claim 2, wherein, The circular ring bosses are four in number, the centers of the four circular ring bosses constitute four vertices of a square, and the circular ring bosses are kept at a set distance apart; The height of the circular ring bosses ranges from 5 mm to 30 mm.

4. The pose solving method of claim 3, wherein, The method of calculating the center coordinates of each circular ring according to the infrared image and generating the first group of center data comprises the following steps: S100, calculating the inner circle center and the outer circle center of each circular ring in the infrared image by using a Hough gradient method, and generating a zero-level center coordinate set; S200、In the zero-level circle center coordinate set, M1 random groups are randomly selected, each random group includes four circle center coordinates corresponding to four annuli, the circle center loss value of each random group is calculated, the circle center loss value includes distance constraint loss value L edges and geometric constraint loss value L angles , the calculation method is as follows: L angles = (θ - θ expected ) 2 wherein (u p ,v p ) are the coordinates of the circle centers, p = 1, 2, 3, 4, d p is the distance between two adjacent circle centers in the horizontal direction or the vertical direction, d expected is the average value of the distance between adjacent circle centers, θ exoected is 90 degrees, and θ is the angle between two diagonals of the square with the circle center coordinates as the vertex. S300, determining a group with the minimum center loss value in M1 random groups as a reference group, generating center sampling points in a pixel range with each center coordinate in the reference group as the center, randomly selecting M2 sampling groups, each sampling group comprising four center sampling points corresponding to the four circular rings, calculating the center loss value of each sampling group, and outputting a sampling group with the minimum center loss value as the first group of center data.

5. The pose solving method of claim 4, wherein, The method of calculating the inner circle center and the outer circle center of each circular ring in the infrared image by using the Hough gradient method and generating the zero-level center coordinate set comprises the following steps: performing Canny edge detection on the original image to obtain a binary edge detection image; applying a Sobel operator to the original image to calculate the gradient value and direction of each pixel point; creating a center space, and accumulating the count of each appearing center by using an accumulator; traversing each non-zero pixel point in the edge detection image, drawing a line along the gradient direction of each non-zero pixel point, and adding one to the count of all the centers of the accumulator passed by the line; sorting the count of all the centers of the accumulator in the center space; using the outer radius and the inner radius of the circular ring boss of the double-light circular ring marker as a constraint, removing the centers not satisfying the radius constraint, and determining the first four groups of inner circle centers satisfying the inner radius constraint and the first four groups of outer circle centers satisfying the outer radius constraint according to the count of the accumulator, to form the zero-level center coordinate set.

6. The pose solving method of claim 1, wherein, The method of calculating the center coordinates of each circular ring according to the three-dimensional point cloud and generating the second group of center data comprises the following steps: Construct a three-dimensional rectangular coordinate system O-XYZ in which the three-dimensional point cloud is located; Select at least three non-collinear points in the three-dimensional point cloud to determine an initial plane in which a double-light ring marker is located, and construct an initial plane equation; Calculate a normal vector of the initial plane and convert it into a unit normal vector; Calculate a first rotation matrix to align the unit normal vector to the Z axis and obtain an X'OY' plane after rotation of the initial plane; Apply the first rotation matrix to all points of the three-dimensional point cloud to convert them to a new coordinate system; Project the three-dimensional point cloud converted to the new coordinate system onto the X'OY' plane to obtain a two-dimensional point set; Calculate the center coordinates of the ring according to the two-dimensional point set, obtain the three-dimensional ring center coordinates according to the normal vector of the initial plane and the first rotation matrix, and generate a second set of center data.

7. The pose solving method of claim 1, wherein, The infrared image is denoised by Gaussian filtering.

8. The pose solving method of claim 1, wherein, The three-dimensional point cloud data is denoised by statistical filtering, and the statistical filtering method is as follows: For each point in the three-dimensional point cloud, define a neighborhood, which is a set of all points within a set radius centered on the point, calculate the mean μ' and standard deviation σ of the neighborhood: where n is the number of points in the neighborhood, p w denotes the coordinates of the wthpoint in the neighborhood. If |p w -μ' | > T, T is a set threshold, then remove p w point.

9. The pose solving method of claim 8, wherein, The threshold T is in the range of 1.5σ to 3σ.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method in any one of claims 1 to 9.

Images