An adaptive real-time fast labeling method and system for tracking scanners

Abstract

This application relates to the technical field of 3D scanning, and in particular to an adaptive real-time fast calibration method and system for a tracking scanner. The method includes: acquiring the known 3D coordinates of each marker point on the scanning head in the scanning head coordinate system; collecting the set of marker point coordinates corresponding to the scanning head and combining it with the known 3D coordinates to perform initial extrinsic parameter estimation; during the scanning operation, acquiring the tracker's observations of the 3D coordinates of visible marker points in the current frame; constructing an error objective function about the extrinsic parameter transformation matrix based on the known 3D coordinates and the observed 3D coordinates; obtaining the updated extrinsic parameter matrix for the current frame based on the error objective function; and using the updated extrinsic parameter matrix to transform the scanning point cloud of the current frame from the scanning head coordinate system to the global coordinate system. This application achieves real-time and adaptive tracking calibration during the scanning operation.

Description

Technical Field

[0001] This application relates to the technical field of 3D scanning, and in particular to an adaptive real-time fast labeling method and system for a tracking scanner. Background Technology

[0002] A tracking-type 3D scanning system consists of two parts: a handheld scanner and a fixed optical tracker. The scanner, typically handheld, uses structured light or laser lines on its head to acquire high-precision 3D data of the object under test in a localized area. The tracker continuously observes several marker points mounted on the scanner to determine the spatial pose of the scanner in real time, thereby stitching together the scan data from each frame into a global coordinate system to achieve large-scale, high-precision 3D measurement.

[0003] In the aforementioned system, the extrinsic transformation matrix (including the rotation matrix R and the translation vector t) between the scanning head coordinate system and the tracker coordinate system is a core parameter for system accuracy. Its solution process is known in the industry as "rapid calibration" or "fast calibration." The accuracy of this extrinsic parameter directly determines the point cloud stitching accuracy, thus affecting the reliability of the final measurement results.

[0004] In existing technologies, the calibration schemes between the scanning head and the tracker mainly fall into the following two categories:

[0005] The first type is the offline pre-calibration method: as shown in Embodiment 2 of the utility model publication CN224517750U, entitled "A Semi-automatic Calibration Device for a 3D Scanning Device," before the formal scanning, the operator holds or uses a support to move a dedicated calibration rod within the tracker's field of view according to prescribed actions. The system uses the precisely known geometric features on the calibration rod and the tracker's measurements to perform optimal fitting, solving the extrinsic parameter transformation matrix in one step. The drawback of this method is:

[0006] (1) The calibration process is independent of the scanning operation, which causes the workflow to be interrupted and reduces the efficiency of operation;

[0007] (2) During the scanning process, the drift of external parameters caused by factors such as changes in ambient temperature, mechanical vibration, and equipment collision cannot be sensed and compensated in real time, resulting in the deterioration of accuracy after a long period of scanning.

[0008] (3) When the equipment location is moved or the tracker is re-installed, the complete calibration process must be repeated;

[0009] (4) The calibration rod, as a special accessory, increases the carrying burden and poses a risk of accidental damage that would prevent the operation from being carried out.

[0010] The second category is offline calibration methods based on dedicated reference components. For example, as shown in Embodiment 1 of the utility model publication CN224517750U, entitled "A Semi-Automatic Calibration Device Suitable for 3D Scanning Devices," a 3D calibration plate or a specially made 3D reference body is used as the calibration target. The registration error between the point cloud obtained by scanning the calibration target and the known CAD model is used as the optimization objective to solve for the extrinsic parameters. This method is more accurate than the calibration rod method, but it also requires dedicated accessories and cannot achieve online updates; its essence remains offline static calibration.

[0011] Neither of the two existing solutions mentioned above can solve the problem of online dynamic updating of external parameters during the scanning operation, and both rely on additional dedicated calibration tools, which limits the practicality of the system in complex industrial sites.

[0012] Furthermore, although some existing products have introduced an automatic gimbal calibration mechanism, this mechanism is essentially a physical alignment of the mechanical angle of the gimbal axis, rather than an online optimization of the external parameter transformation matrix between the scanning head coordinate system and the tracker coordinate system, which is fundamentally different from the technical problem to be solved by this invention.

[0013] Therefore, there is an urgent need in this field for a technical solution that can perform tracking calibration in real time and adaptively during the scanning process without relying on dedicated calibration tools. Summary of the Invention

[0014] In order to perform tracking calibration in real time and adaptively during the scanning process, this application provides an adaptive real-time fast calibration method and system for tracking scanners.

[0015] Firstly, this application provides an adaptive real-time fast labeling method for tracking scanners, employing the following technical solution:

[0016] An adaptive real-time fast tagging method for tracking scanners includes the following steps:

[0017] Perform system initialization and obtain the known three-dimensional coordinates of each marker point on the scanning head in the scanning head coordinate system;

[0018] The tracker is used to collect the set of coordinates of the marker points corresponding to the scanning head, and the initial extrinsic parameters are estimated by combining the known three-dimensional coordinates.

[0019] During the scanning operation, the tracker acquires the 3D coordinate observations of the visible marker points in the current frame;

[0020] Based on the known three-dimensional coordinates and the observed three-dimensional coordinates, an error objective function about the external parameter transformation matrix is ​​constructed through rigid body invariance constraints;

[0021] Based on the error objective function, the extrinsic parameter transformation matrix is ​​updated online to obtain the updated extrinsic parameter matrix for the current frame;

[0022] Using the updated extrinsic parameter matrix, the scanned point cloud of the current frame is transformed from the scanning head coordinate system to the global coordinate system. The online calibration results are then integrated into the point cloud stitching process in real time using the updated extrinsic parameter matrix of the current frame.

[0023] In some embodiments, the initial extrinsic parameter estimation is performed by using a tracker to collect the set of coordinates of the marker points corresponding to the scanning head and combining them with the known three-dimensional coordinates, including the following steps:

[0024] The scanning head is controlled to stay in at least three different postures within the field of view of the tracker, and the spatial coordinate observation values ​​of the marker point by the tracker under each posture are obtained as the set of marker point coordinates;

[0025] Based on the set of marker point coordinates under each attitude and the known three-dimensional coordinates, the initial extrinsic parameter transformation matrix between the scanning head coordinate system and the tracker coordinate system is calculated using a point set registration algorithm.

[0026] In some embodiments, based on the known three-dimensional coordinates and the observed three-dimensional coordinates, an error objective function about the extrinsic parameter transformation matrix is ​​constructed using rigid body invariance constraints, including the following steps:

[0027] For each visible marker point, calculate the reprojection residual between the known 3D coordinates after transformation by the extrinsic parameter transformation matrix and the observed 3D coordinates;

[0028] The adaptive weight of the visible marker is determined based on the observation confidence, geometric contribution, and reprojection residual of the visible marker.

[0029] The reprojection residuals of each visible marker point are nonlinearly mapped using a robust kernel function, and then weighted and summed using the adaptive weights to obtain the extrinsic error objective function.

[0030] In some embodiments, based on the error objective function, the extrinsic transformation matrix is ​​iteratively updated online to obtain the updated extrinsic matrix for the current frame, including the following steps:

[0031] The error objective function is minimized using a sliding window iterative optimization algorithm to obtain the optimized extrinsic parameter matrix for the current frame;

[0032] The optimized extrinsic matrix is ​​fused and smoothed with the historical extrinsic matrix using a Kalman filter to obtain the updated extrinsic matrix.

[0033] In some embodiments, the window size of the sliding window iterative optimization algorithm is dynamically adjusted according to the calibration accuracy, specifically including:

[0034] When the rate of change of the scanning head's posture is detected to be greater than a preset motion threshold, the window size is reduced.

[0035] When the attitude change rate is less than or equal to the preset motion threshold, and the reprojection residuals of several historical frames are in a stable state, the window size is increased.

[0036] After calculating the initial extrinsic transformation matrix, the method further includes validating the initial extrinsic transformation matrix. The validity check includes at least one of the following:

[0037] First type of verification:

[0038] Calculate whether the three-dimensional registration residual between the known three-dimensional coordinates after transformation by the initial extrinsic parameter transformation matrix and the corresponding three-dimensional coordinate observation value meets the preset first condition. If it meets the condition, the initial calibration is determined to be valid and the scanning operation process is entered. If it does not meet the condition, a prompt to re-execute the initial calibration is generated.

[0039] The second type of verification:

[0040] Calculate the root mean square value of the reprojection residual corresponding to the initial extrinsic parameter transformation matrix and the spatial distribution condition number of the marker points used;

[0041] When the root mean square value of the reprojection residual satisfies the second condition and the number of spatial distribution conditions satisfies the third condition, the initial calibration is determined to be valid and the scanning operation process is entered. If the conditions are not met, a prompt to re-execute the initial calibration is generated.

[0042] In some embodiments, the scan point cloud of the current frame is transformed from the scan head coordinate system to the global coordinate system using the updated extrinsic parameter matrix, including the following steps:

[0043] Obtain the real-time pose matrix of the scanning head in the current frame as measured by the tracker;

[0044] The real-time pose matrix, the updated extrinsic parameter matrix, and the scanned point cloud are multiplied sequentially to obtain the global point cloud data in the global coordinate system.

[0045] In some embodiments, the following steps are also included:

[0046] The update magnitude of the updated extrinsic parameter matrix, the reprojection residual of the current frame, and the change in device operating temperature are monitored in real time.

[0047] When the reprojection residual continuously exceeds the preset residual threshold, or when the change in the operating temperature of the device exceeds the preset temperature difference threshold, the soft recalibration process is triggered.

[0048] When the update magnitude exceeds the preset drift threshold, a hard recalibration prompt is triggered.

[0049] In some of these embodiments, it also includes:

[0050] In the soft recalibration process, the initial extrinsic parameter transformation matrix is ​​re-solved using the multi-frame marker point observation data accumulated in the current scanning scene to update the extrinsic parameter transformation matrix without interrupting the current scanning operation;

[0051] In the hard recalibration process, a pause scan prompt signal is generated to instruct the operator to re-execute the initial extrinsic parameter estimation process for multi-pose dwell.

[0052] Secondly, this application provides an adaptive real-time fast labeling system for tracking scanners, employing the following technical solution:

[0053] An adaptive real-time fast labeling system for tracking scanners, comprising:

[0054] The tracking scanning device consists of a scanning head and a tracker. The scanning head shell is provided with a number of non-coplanar reflective markers. The arrangement of the markers ensures that, under any scanning posture, the number of visible markers in the tracker's field of view is not less than four. The spacing between the markers is not less than a preset value, and no three markers are collinear.

[0055] An online calibration processing unit is used to initialize the system and obtain the known three-dimensional coordinates of each marker point on the scanning head in the scanning head coordinate system. It also obtains the set of marker point coordinates corresponding to the scanning head collected by the tracker and performs initial extrinsic parameter estimation by combining the known three-dimensional coordinates. During the scanning operation, it obtains the tracker's observation values ​​of the three-dimensional coordinates of visible marker points in the current frame. Based on the known three-dimensional coordinates and the observed three-dimensional coordinates, it constructs an error objective function about the extrinsic parameter transformation matrix through rigid body invariance constraints. Based on the error objective function, it iteratively updates the extrinsic parameter transformation matrix online through dynamic adaptive weights to obtain the updated extrinsic parameter matrix for the current frame.

[0056] The point cloud fusion unit is used to transform the scanned point cloud of the current frame from the scanning head coordinate system to the global coordinate system using the updated extrinsic parameter matrix, and to fuse the online calibration results into the point cloud stitching process in real time using the updated extrinsic parameter matrix of the current frame.

[0057] The technical solutions provided by the embodiments of this application have the following technical effects:

[0058] No special calibration tools required: This completely eliminates the reliance on special accessories such as calibration rods and calibration plates, reduces system configuration costs, and avoids the risk of work interruption due to damage or loss of accessories.

[0059] Calibration and scanning are performed simultaneously: the external parameter calibration process is completely embedded in the scanning operation, and the operator does not need to switch operating modes or interrupt the measurement, which significantly improves work efficiency, especially for rapid on-site testing scenarios.

[0060] Real-time compensation for external parameter drift: The system continuously monitors and updates external parameters throughout the scanning process, and can automatically compensate for external parameter drift caused by factors such as temperature changes and mechanical vibration, ensuring the accuracy and stability of long-term, large-area scanning.

[0061] Adaptive quality control: By evaluating the visibility, geometric distribution, and residuals of marker points in real time, the system can dynamically adjust the calibration weights and update frequency, maintaining calibration reliability under various complex operating conditions.

[0062] Redundancy enhances robustness: The design of multiple marker points on the scanning head ensures sufficient effective observation constraints even under partial occlusion, and robust estimation implemented through the Huber kernel function further suppresses the interference of anomalous observations. Attached Figure Description

[0063] Figure 1 This is a schematic diagram illustrating the steps of an adaptive real-time fast labeling method for a tracking scanner provided in this embodiment.

[0064] Figure 2 This is a schematic diagram of the overall process of the adaptive real-time fast labeling method for the tracking scanner provided in the embodiments of this application.

[0065] Figure 3 This is a schematic diagram of the adaptive weight calculation process in the adaptive real-time fast labeling method for the tracking scanner provided in the embodiments of this application.

[0066] Figure 4 This is a schematic diagram of the overall structure of the adaptive real-time fast labeling system for the tracking scanner provided in this application embodiment.

[0067] Figure 5 This is a schematic diagram of the internal module of the online calibration processing unit in the adaptive real-time fast labeling system of the tracking scanner provided in this application embodiment.

[0068] Figure 6 This is a schematic diagram of the scanning head provided in this embodiment. Detailed Implementation

[0069] To better understand the purpose, technical solutions, and advantages of this application, it has been described and illustrated below with reference to the accompanying drawings and embodiments. However, those skilled in the art should understand that this application can be implemented without these details. In some cases, to avoid obscuring various aspects of this application due to unnecessary description, well-known methods, processes, systems, components, and / or circuits already described at a higher level will not be elaborated upon. It will be apparent to those skilled in the art that various modifications can be made to the embodiments disclosed in this application, and the general principles defined in this application can be applied to other embodiments and application scenarios without departing from the principles and scope of this application. Therefore, this application is not limited to the illustrated embodiments, but conforms to the broadest scope consistent with the scope of protection claimed in this application.

[0070] It should be noted that the descriptions of these embodiments are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0071] In the description of this application, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0072] In the description of this application, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples.

[0073] First, the tracking scanning device in this application will be described. The tracking scanning device consists of a scanning head and a tracker. Specifically,

[0074] Scanner: A 3D measurement sensor employing structured light or laser line scanning principles. The top, left, right, and rear surfaces of the scanner housing are equipped with 99 reflective spherical markers, each 6mm in diameter. The precise 3D coordinates of each marker have been measured and calibrated at the factory using a high-precision handheld scanner and stored in the scanner's built-in memory. The marker layout design ensures that at least eight markers are simultaneously visible from any angle within a 150° field of view directly in front of the tracker.

[0075] Tracker: Employs a binocular optical tracking unit with two high-resolution CMOS cameras (resolution no less than 2000×1500 pixels) fixed at a 120° angle. It is equipped with a near-infrared (NIR) active illumination source to improve the reliability of marker detection. The tracker's frame rate is no less than 55Hz, synchronized with the scanning head's frame rate, and its working distance ranges from 1.5m to 5m.

[0076] It also includes a computing unit: an embedded computing platform equipped with an industrial-grade GPU, responsible for running the real-time optimization algorithm of the online calibration processing unit, with a processing latency of no more than 16ms (per frame).

[0077] like Figure 1 and Figure 2 As shown in the figure, this application discloses an adaptive real-time fast tagging method for a tracking scanner, including the following steps:

[0078] S100: Perform system initialization and obtain the known three-dimensional coordinates of each marker point on the scanning head in the scanning head coordinate system.

[0079] First, the system is initialized. The operator fixes the tracker at a distance of 1.5m to 5m from the workpiece being measured, ensuring the workpiece is within the overlapping area of ​​the tracker's binocular field of view. After the system is powered on, it automatically reads the coordinate data of the marker points from the built-in memory of the scanning head. ={ , ,..., (Scanning head coordinate system), and initialize the intrinsic parameter matrix of the tracker's binocular camera (already calibrated at the factory).

[0080] S200 uses a tracker to collect the set of coordinates of the marker points corresponding to the scanning head and combines them with known three-dimensional coordinates to estimate the initial external parameters.

[0081] After the system initialization is completed, the tracker collects the actual three-dimensional coordinates of the visible marker points corresponding to various states of the scanning head, and combines them with the known three-dimensional coordinates stored in the corresponding marker points to perform initial extrinsic parameter estimation.

[0082] In other embodiments, the following steps are included:

[0083] S210, control the scanning head to stay in at least three different postures within the tracker's field of view, and obtain the spatial coordinate observation values ​​of the marker points by the tracker in each posture as the set of marker point coordinates.

[0084] S220 calculates the initial extrinsic transformation matrix between the scanning head coordinate system and the tracker coordinate system based on the set of marker point coordinates under each attitude and the known three-dimensional coordinates through a point set registration algorithm.

[0085] The operator holds the scanning head stationary within the tracker's field of view for at least three different poses (it is recommended that the rotation angle between adjacent poses be no less than 15°), with each pose lasting 0.5 seconds (approximately 30 frames of data), and the system automatically acquires the data. For each pose i, the initial transformation matrix between the scanning head coordinate system and the tracker coordinate system is calculated using either the Umeyama algorithm or a point set registration algorithm. Specifically, the set of three-dimensional coordinates of visible marker points obtained by binocular measurement using a tracker. and the corresponding known coordinate subset of the scanning head The initial transformation matrix is ​​solved by minimizing the following objective function. : .

[0086] Where R is the rotation matrix (R∈SO(3)) and t is the translation vector. The known coordinates of the marker point in the scan head coordinate system. Let be the three-dimensional coordinate measurement value of the j-th marker point by the tracker in the i-th pose.

[0087] For initial extrinsic parameter estimation, in addition to Umeyama / SVD, RANSAC (to improve robustness to outliers) or the hand-eye calibration method AX=XB (suitable for scenarios with controllable motion paths) can be used to achieve the same goal of solving initial extrinsic parameters.

[0088] Furthermore, after calculating the initial extrinsic parameter transformation matrix, a validity check based on confidence level assessment is also included, specifically:

[0089] First type of verification:

[0090] S221, calculate whether the three-dimensional registration residual between the known three-dimensional coordinates after transformation by the initial extrinsic parameter transformation matrix and the corresponding three-dimensional coordinate observation value meets the preset first condition. If it meets the condition, the initial calibration is determined to be valid and the scanning operation process is entered. If it does not meet the condition, a prompt to re-execute the initial calibration is generated.

[0091] The solution uses the Singular Value Decomposition (SVD) method. After the initial calibration is completed, the system calculates the 3D registration residual RMSE. If the reprojection residual RMSE is less than 0.3mm, the first condition is met, the initial calibration is deemed valid, and the system enters the real-time scanning stage; otherwise, the operator is prompted to re-execute the initial calibration.

[0092] The 3D registration residual is a type of 3D point set registration residual. It directly calculates the spatial distance deviation between the marker point in the scanning head coordinate system after extrinsic parameter transformation and the 3D coordinates obtained by the binocular measurement of the tracker. The unit is millimeters, which directly reflects the alignment accuracy of the extrinsic parameters in 3D space.

[0093] The second type of verification:

[0094] S222, calculate the root mean square value of the reprojection residual corresponding to the initial extrinsic parameter transformation matrix and the spatial distribution condition number of the marker points used.

[0095] S223. When the root mean square value of the reprojection residual satisfies the second condition and the number of spatial distribution conditions satisfies the third condition, the initial calibration is determined to be valid and the scanning operation process is entered. If the conditions are not met, a prompt to re-execute the initial calibration is generated.

[0096] When the root mean square value of the reprojection residual of the initial calibration is less than 0.3 pixels, and the number of spatial distribution conditions of the marker points in the scanning head coordinate system is less than the preset value, the second and third conditions are considered to be matched, the initial calibration is determined to be valid, and the system enters the real-time scanning stage; otherwise, the operator is prompted to adjust the posture and re-execute the initial calibration.

[0097] The reprojection residual RMSE is calculated by transforming the marker points from the scanner coordinate system to the camera coordinate system, projecting the 3D points onto the pixel plane using a known intrinsic parameter matrix to obtain the theoretical projected coordinates, and then calculating the reprojection residual of a single marker point as the Euclidean distance between the projected coordinates and the measured coordinates, in pixels. The root square of the error for n valid marker points yields the reprojection residual RMSE.

[0098] The two methods mentioned above have different sources of error. Pixel-level error includes the combined effects of intrinsic parameters, distortion, and detection error, while three-dimensional error mainly reflects the rigid body transformation error of extrinsic parameters. They also have different functions. Pixel-level judgment can detect underlying problems in camera calibration and marker detection in advance, while three-dimensional judgment directly ensures the final measurement accuracy.

[0099] The spatial distribution condition number is a numerical indicator that measures the impact of the spatial distribution of marker points on the stability of parameter solving. The calculation logic is as follows:

[0100] 1. When solving for extrinsic parameters, the coordinates of all marked points will be used to construct the coefficient matrix (such as the design matrix or covariance matrix of the Umeyama algorithm).

[0101] 2. Perform singular value decomposition (SVD) on the matrix. The condition number = maximum singular value / minimum singular value.

[0102] The smaller the condition number, the more uniform the spatial distribution of the marker points, the better the non-coplanar / non-collinearity, the stronger the numerical stability of the external parameter solution, and the stronger the resistance to noise interference.

[0103] The larger the condition value, the closer the marked points are to being coplanar, collinear, or focused. This makes it easier for ill-conditioned problems to occur during the solution process, and even small observation noise can lead to significant deviations in the external parameter results.

[0104] The effect of adding spatial distribution condition numbers is to avoid "false pass" initial calibration caused by improper operator posture (marking points are almost on the same plane), and to ensure the reliability of extrinsic parameter solution from a geometric distribution perspective.

[0105] During the scanning process, the S300 acquires the three-dimensional coordinate observations of the visible marker points in the current frame from the tracker.

[0106] Once the formal scanning phase begins, the scanning head and tracker operate at a synchronized frame rate. Within each frame acquisition cycle: the scanning head outputs the local 3D point cloud of the object being measured in the current frame; the tracker synchronously outputs the binocular image coordinates of the currently visible marker points, and the 3D coordinate observations of each marker point are recovered through binocular triangulation. (Frame k).

[0107] Furthermore, it is also necessary to count the number of visible marker points. and the detection confidence of each marker point in the stereo image. .

[0108] The confidence level represents the reliability of a marker point being correctly detected, and it is usually obtained by weighted fusion of several image features, for example:

[0109] Gray-scale contrast: The difference between the gray-scale value of the reflective marker and the gray-scale value of the surrounding background. The larger the difference, the clearer the target and the higher the confidence level.

[0110] Shape matching degree: Calculate the circularity of the detection spot (the ratio of the contour area to the square of the perimeter, and the deviation from the standard circle). The closer it is to the standard circle, the higher the confidence level.

[0111] Size consistency: Based on the currently estimated depth, the imaging diameter of the marker should be within the theoretical range; the smaller the deviation, the higher the confidence level.

[0112] Binocular matching degree: Whether the matching points of the left and right cameras satisfy the epipolar geometry constraints. The smaller the epipolar distance, the more reliable the matching and the higher the confidence.

[0113] Finally, the indicators of each dimension are normalized and weighted to obtain the confidence level in the 0-1 interval. The higher the value, the better the observation quality.

[0114] S400, based on known three-dimensional coordinates and three-dimensional coordinate observations, constructs an error objective function about the external parameter transformation matrix through rigid body invariance constraints.

[0115] like Figure 3 As shown, specifically, it includes the following steps:

[0116] S410, for each visible marker point, calculate the reprojection residual between the known 3D coordinates after transformation by the extrinsic parameter transformation matrix and the observed 3D coordinates.

[0117] S420, based on the observation confidence, geometric contribution, and reprojection residual of the visible marker, determine the adaptive weight of the visible marker.

[0118] S430 uses a robust kernel function to perform nonlinear mapping on the reprojection residuals of each visible marker point, and combines adaptive weights to perform weighted summation to obtain the extrinsic error objective function.

[0119] For the k-th frame, construct the transformation matrix with respect to the extrinsic parameters. ={ , The objective function for the error of} is: .

[0120] in, Let k be the set of visible marker points. ρ is the adaptive weight for the j-th marker point, and ρ(·) is the Huber kernel function used to suppress the influence of anomalous observations.

[0121] The reprojection residual, input to the robust kernel function, represents the known fixed coordinates in the coordinate system of the j-th marker point. The transformation matrix of the extrinsic parameters to be optimized is used. ={ , Projecting this onto the tracker coordinate system yields the theoretical three-dimensional coordinates: The theoretical coordinates are compared with the measured three-dimensional coordinates obtained by binocular triangulation of the tracker. The difference is calculated to obtain the residual vector; the Euclidean distance (L2 norm) is taken from the residual vector to obtain a measurement value.

[0122] The robust kernel function (Huber kernel) operates on this scalar residual: when the residual is small, it maintains the squared penalty to ensure accuracy; when the residual is large, it switches to linear penalty to reduce the interference of outlier observations (occlusion, mismatched points) on the optimization of extrinsic parameters.

[0123] The Huber kernel can be replaced by the Tukey kernel (which has a stronger penalty for large residuals) or RANSAC out-of-point removal, with the same purpose.

[0124] For each visible marker point j, its corresponding adaptive weight is calculated as follows:

[0125] =α· +β· -γ· .

[0126] in, The confidence level for binocular detection is (0~1). Geometric contribution (normalized to 0~1) is the measure of the diagonal elements of the Fisher information matrix. The reprojection residuals for the current frame are α=0.4, β=0.4, and γ=0.2, which are the default balance coefficients (configurable by the user).

[0127] Geometric contribution measures the contribution value of a single marker point to the accuracy of extrinsic parameter estimation. Its Fisher information matrix describes the amount of information the observed data provides to the unknown parameters (the 6 degrees of freedom of the extrinsic parameters). The inverse of the matrix corresponds to the covariance matrix of the parameter estimation. The greater the amount of information, the higher the estimation accuracy.

[0128] Specifically, let the extrinsic parameter vector be ξ (6-dimensional, corresponding to rotation + translation), and the Jacobian matrix of the error term at the j-th marker point with respect to the parameters be... ;

[0129] Fisher information contribution of a single marker point The overall Fisher information matrix is ​​obtained by summing all the points. ;

[0130] Geometric contribution of the j-th point , represents the contribution percentage of this point to the Fisher information matrix trace (the sum of the diagonal elements), reflecting the improvement in the overall accuracy of extrinsic parameter estimation after adding this point;

[0131] Finally, the contribution of all points is normalized to the 0-1 range.

[0132] Meanwhile, when the number of visible markers is less than 3, the update of this frame is paused while the valid extrinsic parameters of the previous frame are maintained.

[0133] By using the adaptive weights described above, the corresponding scenarios are optimized under different feature changes:

[0134] Points with high confidence are given higher weights, automatically reducing the impact of blurry, partially occluded, and low-quality observations, and adapting to industrial sites with changes in lighting and partial occlusion;

[0135] Points with higher geometric contribution have higher weights, and high-value constraints are retained first, ensuring solution stability even when the number of visible points is small.

[0136] Points with large residuals have their weights reduced. Combined with the Huber kernel function, this further weakens the interference of mismatches and outliers on external parameter updates, thus improving robustness.

[0137] Meanwhile, the aforementioned weights change dynamically with the observation status of each frame, automatically maintaining calibration accuracy under different working conditions and adapting to complex field environments.

[0138] S500, based on the error objective function, performs online iterative updates on the extrinsic transformation matrix to obtain the updated extrinsic matrix for the current frame.

[0139] Specifically,

[0140] S510 uses a sliding window iterative optimization algorithm to minimize the error objective function and obtain the optimized extrinsic parameter matrix for the current frame.

[0141] S520 uses a Kalman filter to fuse and smooth the optimized extrinsic parameter matrix with the historical extrinsic parameter matrix to obtain the updated extrinsic parameter matrix.

[0142] Maintain the observation buffer of the marker points in the most recent W frames, and minimize the cumulative objective function using the Gauss-Newton method to obtain the updated optimized extrinsic parameter matrix. The updated results are fused with historical extrinsic parameter estimates using a Kalman filter, outputting a smooth sequence of updated extrinsic parameters. .

[0143] Besides the weighted least squares sliding window, extended Kalman filter (EKF) (low computational cost, suitable for embedded systems), unscented Kalman filter (UKF) (handles strong nonlinearity), or factor graph optimization (GTSAM) (highest accuracy) can be used to achieve online iterative updates of extrinsic parameters.

[0144] Meanwhile, the extrinsic smoothing Kalman filter can be replaced by the exponentially weighted moving average (EWMA) (for simplicity) or the Savitzky-Golay filter (to preserve transient trends).

[0145] The sliding window in this application undergoes iterative optimization, specifically:

[0146] S511: When the rate of change of the scanning head's posture is detected to be greater than the preset motion threshold, the window size is reduced.

[0147] S512, when the rate of change of attitude is less than or equal to the preset motion threshold and the reprojection residuals of several historical frames are in a stable state, the window size is increased.

[0148] When the scanning head rotation angular velocity exceeds 30° / s within 5 consecutive frames, the window W is shortened to 5 frames; when the angular velocity is less than 5° / s and the residual RMSE of the last 20 frames is less than 0.2mm, the window W is expanded to 5-50 frames.

[0149] The smaller the window, the higher the proportion of new observation data, which can quickly track sudden changes in external parameters (such as equipment collisions or sudden temperature changes that cause external parameter jumps) and will not cause update delays due to a large amount of old historical data. When moving at high speed, the imaging of marker points is prone to blurring and the observation noise increases. A small window avoids the accumulation of a large amount of low-quality historical data and reduces the accuracy loss caused by motion.

[0150] The larger the window, the more effective observation constraints are accumulated. By averaging multiple frames of data to suppress random observation noise, the accuracy and stability of extrinsic parameter estimation are significantly improved. When moving at low speeds, the observation quality is high, and a large window can make full use of the geometric constraints of multiple frames, making the extrinsic parameter estimation more reliable.

[0151] The dynamic window mechanism allows the system to automatically adapt to the optimal calibration strategy under different operating habits, such as the operator moving the scanning head quickly or performing slow, fine scanning. It balances the real-time performance of dynamic scenarios with the high precision of static scenarios, thereby improving calibration stability under all operating conditions.

[0152] Furthermore, it also includes an adaptive recalibration triggering step, specifically:

[0153] The S700 monitors in real time the update magnitude of the extrinsic parameter matrix, the reprojection residual of the current frame, and the change in the device's operating temperature.

[0154] S710: When the reprojection residual continuously exceeds the preset residual threshold, or the change in the operating temperature of the equipment exceeds the preset temperature difference threshold, the soft recalibration process is triggered.

[0155] When the update magnitude exceeds the preset drift threshold, the S720 triggers a hard recalibration prompt.

[0156] The system continuously monitors the following three types of triggering conditions:

[0157] Residual trigger: If the reprojection residual RMSE of the current frame exceeds 0.5mm for 5 consecutive frames, the system determines that the external parameters have drifted significantly and triggers soft recalibration.

[0158] Update magnitude triggered: relatively If the rotational component changes by more than 1° or the translational component changes by more than 1 mm, the system will display an orange warning on the user interface and recommend performing a hard recalibration.

[0159] Temperature trigger: The system has a built-in temperature sensor that continuously monitors the operating temperature of the tracker and the scanning head. When the temperature change exceeds the preset temperature difference threshold ΔT℃ (5℃ in this embodiment), the system automatically triggers the soft recalibration process.

[0160] In the soft recalibration process, the initial extrinsic parameter transformation matrix is ​​re-solved using the multi-frame marker point observation data accumulated in the current scanning scene to update the extrinsic parameter transformation matrix without interrupting the current scanning operation.

[0161] During the hard recalibration process, a pause scan prompt signal is generated to instruct the operator to re-execute the initial extrinsic parameter estimation process for multi-pose dwell.

[0162] The S600 uses an updated extrinsic parameter matrix to transform the scanned point cloud of the current frame from the scan head coordinate system to the global coordinate system, and uses the updated extrinsic parameter matrix of the current frame to integrate the online calibration results into the point cloud stitching process in real time.

[0163] The final step involves point cloud coordinate fusion, specifically:

[0164] S610: Obtain the real-time pose matrix of the scanning head in the current frame as measured by the tracker.

[0165] S620 multiplies the real-time pose matrix, the updated extrinsic parameter matrix, and the scanned point cloud sequentially to obtain global point cloud data in the global coordinate system.

[0166] Scan the point cloud of the k-th frame When transforming from the scan head coordinate system to the global coordinate system, the smoothed extrinsic parameters output by the Kalman filter are used. Real-time pose of the scanning head measured by the tracker Combined coordinate transformation:

[0167] The transformed point cloud is written to the global point cloud buffer in real time to complete the stitching of multiple frames.

[0168] like Figure 4 As shown, this application also discloses an adaptive real-time fast labeling system for tracking scanners, comprising:

[0169] The tracking scanning device consists of a scanning head and a tracker, wherein, for example... Figure 6 As shown, the scanning head housing is provided with several non-coplanar reflective markers, and the arrangement of these markers ensures that, under any scanning posture, the number of visible markers in the tracker's field of view is no less than eight, the spacing between each marker is no less than a preset value, and no three points are collinear.

[0170] Meanwhile, in other embodiments, in addition to reflective spherical markers, coded markers (such as ArUco codes, which have unique IDs and are not matched), LED active light-emitting markers (resistant to ambient light interference), or IMU-assisted schemes (compensating for occlusion frames) can also be used for the design of markers. All three can achieve reliable observation of the scanning head's attitude.

[0171] like Figure 5 As shown, the online calibration processing unit specifically includes an extrinsic parameter initial estimation submodule, a real-time constraint construction submodule, an adaptive weight optimization submodule, and a recalibration triggering submodule, which is responsible for continuously updating the extrinsic parameter transformation matrix between the scanning head and the tracker throughout the scanning operation.

[0172] Among them, the extrinsic parameter initial estimation submodule is used to perform system initialization and obtain the known three-dimensional coordinates of each marker point on the scanning head in the scanning head coordinate system, obtain the set of marker point coordinates corresponding to the scanning head collected by the tracker, and combine them with the known three-dimensional coordinates to perform initial extrinsic parameter estimation;

[0173] During the scanning operation, the tracker acquires the 3D coordinate observation values ​​of the visible marker points in the current frame. The real-time constraint construction submodule is used to construct the error objective function about the external parameter transformation matrix based on the known 3D coordinates and the 3D coordinate observation values ​​through rigid body invariance constraints.

[0174] The adaptive weight optimization submodule is used to dynamically update the extrinsic transformation matrix online based on the error objective function, thereby obtaining the updated extrinsic matrix for the current frame.

[0175] The recalibration trigger submodule is used to monitor in real time the update magnitude of the updated extrinsic parameter matrix, the reprojection residual of the current frame, and the change in the device's operating temperature. When the reprojection residual continuously exceeds the preset residual threshold, or the change in the device's operating temperature exceeds the preset temperature difference threshold, the soft recalibration process is triggered. When the update magnitude exceeds the preset drift threshold, a hard recalibration prompt is triggered.

[0176] The point cloud fusion unit is used to transform the scanned point cloud of the current frame from the scanning head coordinate system to the global coordinate system by updating the extrinsic parameter matrix, and to fuse the online calibration results into the point cloud stitching process in real time using the updated extrinsic parameter matrix of the current frame.

[0177] It also includes a system management unit, responsible for calibration quality monitoring, residual early warning display, and recalibration process scheduling. The system management unit also includes a calibration quality visualization interface, which displays the following indicators in real time: the number of visible marker points in the current frame and a heat map of their spatial distribution, the external parameter reprojection residual curve, the external parameter update amplitude trend graph, and the temperature-residual correlation curve, and uses color coding (green / yellow / red) to indicate the current calibration status to the operator.

[0178] The implementation principle is as follows:

[0179] The marker points originally used for tracker positioning and mounted on the scanner head housing are reused as constraint sources for online calibration of scanner head-tracker extrinsic parameters. During each frame of data acquisition in the scanning operation, the system synchronously utilizes the rigid constraints between the tracker's observed 3D coordinates of the marker points and the known fixed coordinates of the marker points in the scanner head coordinate system to continuously optimize and update the extrinsic parameter transformation matrix. This allows the calibration process to be fully integrated into the normal scanning workflow without any additional operations or accessories.

[0180] It should be understood that although the steps in the flowcharts in the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise expressly stated herein, there is no strict order in which these steps are performed, and they may be performed in other orders.

[0181] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. An adaptive real-time fast flagging method for tracking scanners, characterized by, Includes the following steps: Perform system initialization and obtain the known three-dimensional coordinates of each marker point on the scanning head in the scanning head coordinate system; The tracker is used to collect the set of coordinates of the marker points corresponding to the scanning head, and the initial extrinsic parameters are estimated by combining the known three-dimensional coordinates. During the scanning operation, the tracker acquires the 3D coordinate observations of the visible marker points in the current frame; Based on the known three-dimensional coordinates and the observed three-dimensional coordinates, an error objective function about the external parameter transformation matrix is ​​constructed through rigid body invariance constraints; Based on the error objective function, the extrinsic parameter transformation matrix is ​​updated online to obtain the updated extrinsic parameter matrix for the current frame; Using the updated extrinsic parameter matrix, the scanned point cloud of the current frame is transformed from the scanning head coordinate system to the global coordinate system. The online calibration results are then integrated into the point cloud stitching process in real time using the updated extrinsic parameter matrix of the current frame.

2. The method of claim 1, wherein, The initial extrinsic parameter estimation is performed by collecting the set of coordinates of the marker points corresponding to the scanning head using a tracker and combining them with the known three-dimensional coordinates, including the following steps: The scanning head is controlled to stay in at least three different postures within the field of view of the tracker, and the spatial coordinate observation values ​​of the marker point by the tracker under each posture are obtained as the set of marker point coordinates; Based on the set of marker point coordinates under each attitude and the known three-dimensional coordinates, the initial extrinsic parameter transformation matrix between the scanning head coordinate system and the tracker coordinate system is calculated using a point set registration algorithm.

3. The method of claim 1, wherein, Based on the known three-dimensional coordinates and the observed three-dimensional coordinates, an error objective function about the external parameter transformation matrix is ​​constructed through rigid body invariance constraints, including the following steps: For each visible marker point, calculate the reprojection residual between the known 3D coordinates after transformation by the extrinsic parameter transformation matrix and the observed 3D coordinates; The adaptive weight of the visible marker is determined based on the observation confidence, geometric contribution, and reprojection residual of the visible marker. The reprojection residuals of each visible marker point are nonlinearly mapped using a robust kernel function, and then weighted and summed using the adaptive weights to obtain the error objective function.

4. The method of claim 3, wherein, Based on the error objective function, the extrinsic parameter transformation matrix is ​​updated online to obtain the updated extrinsic parameter matrix for the current frame, including the following steps: The error objective function is minimized using a sliding window iterative optimization algorithm to obtain the optimized extrinsic parameter matrix for the current frame; The optimized extrinsic matrix is ​​fused and smoothed with the historical extrinsic matrix using a Kalman filter to obtain the updated extrinsic matrix.

5. The method of claim 4, wherein, The window size of the sliding window iterative optimization algorithm is dynamically adjusted according to the calibration accuracy, specifically including: When the rate of change of the scanning head's posture is detected to be greater than a preset motion threshold, the window size is reduced. When the attitude change rate is less than or equal to the preset motion threshold, and the reprojection residuals of several historical frames are in a stable state, the window size is increased.

6. The adaptive real-time fast labeling method for tracking scanners according to claim 1, characterized in that, After calculating the initial extrinsic transformation matrix, the method further includes validating the initial extrinsic transformation matrix. The validity check includes at least one of the following: First type of verification: Calculate whether the three-dimensional registration residual between the known three-dimensional coordinates after transformation by the initial extrinsic parameter transformation matrix and the corresponding three-dimensional coordinate observation value meets the preset first condition. If it meets the condition, the initial calibration is determined to be valid and the scanning operation process is entered. If it does not meet the condition, a prompt to re-execute the initial calibration is generated. The second type of verification: Calculate the root mean square value of the reprojection residual corresponding to the initial extrinsic parameter transformation matrix and the spatial distribution condition number of the marker points used; When the root mean square value of the reprojection residual satisfies the second condition and the number of spatial distribution conditions satisfies the third condition, the initial calibration is determined to be valid and the scanning operation process is entered. If the conditions are not met, a prompt to re-execute the initial calibration is generated.

7. The adaptive real-time fast labeling method for tracking scanners according to claim 2, characterized in that, Using the updated extrinsic parameter matrix, the scan point cloud of the current frame is transformed from the scan head coordinate system to the global coordinate system, including the following steps: Obtain the real-time pose matrix of the scanning head in the current frame as measured by the tracker; The real-time pose matrix, the updated extrinsic parameter matrix, and the scanned point cloud are multiplied sequentially to obtain the global point cloud data in the global coordinate system.

8. The adaptive real-time fast labeling method for tracking scanners according to claim 3, characterized in that, It also includes the following steps: The update magnitude of the updated extrinsic parameter matrix, the reprojection residual of the current frame, and the change in device operating temperature are monitored in real time. When the reprojection residual continuously exceeds the preset residual threshold, or when the change in the operating temperature of the device exceeds the preset temperature difference threshold, the soft recalibration process is triggered. When the update magnitude exceeds the preset drift threshold, the hard recalibration process is triggered.

9. The adaptive real-time fast labeling method for tracking scanners according to claim 8, characterized in that, Also includes: In the soft recalibration process, the initial extrinsic parameter transformation matrix is ​​re-solved using the multi-frame marker point observation data accumulated in the current scanning scene to update the extrinsic parameter transformation matrix without interrupting the current scanning operation; In the hard recalibration process, a pause scan prompt signal is generated to instruct the operator to re-execute the initial extrinsic parameter estimation process for multi-pose dwell.

10. An adaptive real-time rapid labeling system for a tracking scanner, characterized in that, include: The tracking scanning device consists of a scanning head and a tracker. The scanning head shell is provided with a number of non-coplanar reflective markers. The arrangement of the markers ensures that, under any scanning posture, the number of visible markers in the tracker's field of view is not less than four. The spacing between the markers is not less than a preset value, and no three markers are collinear. An online calibration processing unit is used to initialize the system and obtain the known three-dimensional coordinates of each marker point on the scanning head in the scanning head coordinate system. It also obtains the set of marker point coordinates corresponding to the scanning head collected by the tracker and performs initial extrinsic parameter estimation based on the known three-dimensional coordinates. During the scanning operation, it obtains the tracker's observation values ​​of the three-dimensional coordinates of visible marker points in the current frame. Based on the known three-dimensional coordinates and the observed three-dimensional coordinates, it constructs an error objective function about the extrinsic parameter transformation matrix through rigid body invariance constraints. Based on the error objective function, it iteratively updates the extrinsic parameter transformation matrix online to obtain the updated extrinsic parameter matrix for the current frame. The point cloud fusion unit is used to transform the scanned point cloud of the current frame from the scanning head coordinate system to the global coordinate system using the updated extrinsic parameter matrix, and to fuse the online calibration results into the point cloud stitching process in real time using the updated extrinsic parameter matrix of the current frame.

Images