Tracking three-dimensional scanning calibration method, device and equipment

Abstract

This invention provides a tracking-based 3D scanning calibration method, apparatus, and device, relating to the field of calibration technology. Based on observation data of the calibration structure, it calculates the pose data of the calibration structure in the tracker coordinate system, which is used to project the visualized object corresponding to the calibration structure onto the visualized area corresponding to the target sampling unit in the depth scene image. Using the visualized object as a mapping reference, dynamic guide markers corresponding to the pose data are projected onto the depth scene image, guiding the user to adjust the pose of the calibration structure according to the dynamic guide markers, thus obtaining effective observation data and corresponding effective pose data. Calibration is performed based on the effective observation data and corresponding effective pose data of the calibration structure in multiple target sampling units to obtain the calibration result. This invention can reduce the operator's conversion cost from schematic information to actual actions and the reliance on calibration experience, while significantly improving the quality of spatial and pose coverage.

Description

Technical Field

[0001] This invention relates to the field of calibration technology, and in particular to a tracking-type three-dimensional scanning calibration method, apparatus, and equipment. Background Technology

[0002] In tracking-based 3D scanning or spatial measurement systems, external trackers are typically equipped with two depth cameras and other observation units to acquire depth scene information, spatial position, and attitude information of the calibrator, scanner, target, or other tracked objects. To ensure the accuracy of subsequent scanning, measurement, or pose guidance results, the external tracker must be calibrated before use. This calibration covers the observation geometry between the two depth cameras, the reference relationship between the tracking coordinate systems, the stability of the target recognition state, and the consistency of the pose acquisition process.

[0003] In current engineering practice, mainstream calibration methods include: static calibration using fixed calibration components; collecting samples and fitting calibration parameters using calibration rods, calibration balls, or the tracked target at multiple positions and in multiple postures; relying on human experience to judge the sufficiency of sampling spatial coverage; and uniformly analyzing the reprojection residuals after all data acquisition is completed to determine whether resampling is necessary. With increasing scanning accuracy requirements and the growing complexity of on-site operating environments, the above methods are showing increasing limitations in terms of sampling coverage integrity, operational repeatability, real-time feedback error correction capabilities, and overall calibration efficiency. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a tracking-type three-dimensional scanning calibration method, apparatus and equipment, which can reduce the operator's conversion cost from schematic information to actual action and the degree of dependence on calibration experience, while significantly improving the quality of spatial coverage and attitude coverage.

[0005] In a first aspect, the present invention provides a tracking-based three-dimensional scanning calibration method. The method is based on a calibration system, which includes a calibration structure, a three-dimensional tracker, and a guide display. The three-dimensional tracker includes at least two depth sensors, which are used to collect observation data from the calibration structure. The guide display initially displays a depth scene image corresponding to the work scene, which is divided into multiple sampling units, including: Based on the observation data of the calibration structure, the pose data of the calibration structure in the tracker coordinate system is calculated, so as to project the visualization object corresponding to the calibration structure onto the visualization area corresponding to the target sampling unit in the depth scene image. Using the visualized object as a mapping reference, the dynamic guide markers corresponding to the pose data are projected onto the depth scene image, guiding the user to adjust the pose of the calibration structure according to the dynamic guide markers, thereby obtaining effective observation data and corresponding effective pose data; The calibration is performed based on the effective observation data and corresponding effective pose data of the calibration structure in multiple target sampling units, and the calibration results are obtained.

[0006] In one implementation, before calculating the pose data of the calibration structure in the tracker coordinate system based on the observation data of the calibration structure, the method further includes: Obtain the coverage priority corresponding to each sampling unit in the job scenario; Based on the coverage priority of the sampling units that have not been sampled in the operation scenario, determine the next target sampling unit to be sampled; The mapping relationship between the pre-built tracker coordinate system and the display coordinate system corresponding to the guide display is invoked to project the three-dimensional spatial range of the target sampling unit in the tracker coordinate system to the display coordinate system, and generate and render the visualization area of ​​the target sampling unit in the depth scene image in the specified first visual state.

[0007] In one implementation, before projecting dynamic guide markers corresponding to the pose data onto a depth scene image using a visualized object as a mapping reference, and guiding the user to adjust the pose of the calibration structure according to the dynamic guide markers to obtain effective observation data and corresponding effective pose data, the method further includes: Maintain the visualization area corresponding to the target sampling unit in the depth scene image in the specified first visual state to represent that the target sampling unit has not completed sampling; A guide path is generated based on the pose data and rendered in the visualization area. The guide path is used to guide the user to move the calibration structure to the target sampling unit. The second visual state of the guide path represents the distance between the calibration structure and the target sampling unit and is dynamically updated as the distance between the calibration structure and the target sampling unit changes.

[0008] In one implementation, using a visualized object as a mapping reference, the dynamic guide markers corresponding to the pose data are projected onto a depth scene image, including: Determine whether the pose data satisfies the pose constraint conditions corresponding to the target sampling unit; If not, based on the pose data and pose constraints, determine the recommended movement direction and attitude adjustment amount corresponding to the calibration structure; Using a specified point in the visualized object as the rendering reference, and based on the recommended movement direction and attitude deviation, render the dynamic guidance flag corresponding to the calibration structure in the visualized area. The third visual state of the dynamic guidance sign includes direction and visual salience. Direction represents the recommended direction of movement, and visual salience represents the amount of posture adjustment. Both the visualized object and the third visual state are dynamically updated as the pose data changes.

[0009] In one implementation, the dynamic guidance flag corresponding to the depth scene image rendering calibration structure includes: Maintain the visualization area corresponding to the target sampling unit in the depth scene image in the specified first visual state; Determine the change between the pose error of the current frame's pose data relative to the pose constraints and the pose error of the previous frame's pose data relative to the pose constraints. If the change in the value represents the pose of the calibration structure and approaches the pose constraint, then the visual saliency is reduced. If the change represents a deviation of the pose of the calibration structure from the pose constraints, then the visual saliency is enhanced.

[0010] In one implementation, the method further includes: If the pose data satisfies the pose constraints corresponding to the target sampling unit, render the visualization area of ​​the target sampling unit in the depth scene image in the specified fourth visual state to represent that the target sampling unit has completed sampling. The observation data of the current frame is used as the effective observation data, and the pose data calculated based on the observation data of the current frame is used as the effective pose data. The method also includes updating the sampling coverage index and refreshing the sampling coverage index in real time on the display screen of the guide display. The sampling coverage index includes the number of completed sampling units, the total number of sampling units, and the coverage percentage.

[0011] In one implementation, the method further includes: If the calibration structure fails to enter the target sampling unit within multiple consecutive frames, a timeout timer is started. If the calibration structure still fails to enter the target sampling unit after a preset time threshold has elapsed since the start timeout timer, it is determined to be a guidance failure. In response to a boot failure determination, perform at least one of the following operations: Clear the dynamic guidance flags and re-render the dynamic guidance flags corresponding to the pose data based on the same target sampling unit; determine the new target sampling unit based on the coverage priority of the sampling units that have not completed sampling in the operation scenario.

[0012] In one implementation, the method further includes: Based on the number of visible coded feature points in the observation data and the preset feature point number threshold, abnormal observation states are identified. In response to the identification of anomalies observed, anomaly status prompts are overlaid on the depth scene image, and updates to the third visual status of dynamic guidance signs are paused. When the number of visible coded feature points in the observation data of multiple consecutive frames meets the feature point number threshold, the third visual state of the dynamic guidance sign is updated or initialized.

[0013] In one embodiment, before calibrating based on the effective observation data and corresponding effective pose data of the calibration structure within multiple target sampling units to obtain the calibration result, the method further includes: Determine the effective observation data of the calibration structure within the target sampling unit and the corresponding effective pose data sampling quality evaluation index; If the sampling quality evaluation index meets the resampling conditions, render the visualization area corresponding to the target sampling unit in the depth scene image in the specified fifth visual state.

[0014] Secondly, the present invention also provides a tracking-type three-dimensional scanning calibration device. The device is based on a calibration system, which includes a calibration structure, a three-dimensional tracker, and a guide display. The three-dimensional tracker includes at least two depth sensors, which are used to collect observation data from the calibration structure. The initial display screen of the guide display is a depth scene image corresponding to the work scene. The work scene is divided into multiple sampling units, including: The calibration structure projection module is used to calculate the pose data of the calibration structure in the tracker coordinate system based on the observation data of the calibration structure, and to project the visualization object corresponding to the calibration structure onto the visualization area corresponding to the target sampling unit in the depth scene image. The guidance sign projection module is used to project dynamic guidance signs corresponding to the pose data onto the depth scene image, using the visualized object as a mapping reference. This guides the user to adjust the pose of the calibration structure according to the dynamic guidance signs, thereby obtaining effective observation data and corresponding effective pose data. The calibration module is used to calibrate the calibration structure based on the effective observation data and corresponding effective pose data in multiple target sampling units, and obtain the calibration results.

[0015] Thirdly, the present invention also provides an electronic device including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement any of the methods provided in the first aspect.

[0016] Fourthly, the present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement any of the methods provided in the first aspect.

[0017] This invention provides a tracking-based 3D scanning calibration method, apparatus, and device, based on a calibration system. The calibration system includes a calibration structure, a 3D tracker, and a guide display. The 3D tracker includes at least two depth sensors, which are used to collect observation data of the calibration structure. The guide display initially displays a depth scene image corresponding to the work scene, which is divided into multiple sampling units. First, based on the observation data of the calibration structure, the pose data of the calibration structure in the tracker coordinate system is calculated to project the visualization object corresponding to the calibration structure onto the visualization area corresponding to the target sampling unit in the depth scene image. Then, using the visualization object as a mapping reference, dynamic guide marks corresponding to the pose data are projected onto the depth scene image to guide the user to adjust the pose of the calibration structure according to the dynamic guide marks, thereby obtaining effective observation data and corresponding effective pose data. Finally, calibration is performed based on the effective observation data and corresponding effective pose data of the calibration structure in multiple target sampling units to obtain the calibration result. This method accurately projects the visualized objects of the calibration structure and their dynamic guide markers onto the corresponding target sampling unit areas in the depth scene image based on real-time calculated pose data. This allows operators to directly perform movement and posture adjustment actions based on image content aligned with the actual work scene space, without the need for secondary understanding and mapping between abstract schematic diagrams and physical space. The guidance information is updated in real time with changes in pose, and combined with the status feedback of the sampling units, operators can intuitively identify covered and uncovered areas, thereby reducing the conversion cost from schematic information to actual actions and the dependence on calibration experience, while significantly improving the quality of spatial and posture coverage.

[0018] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description and the drawings.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a tracking-type three-dimensional scanning calibration method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a calibration rod provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a three-dimensional tracker provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a guide display provided in an embodiment of the present invention; Figure 5 A schematic diagram of range division provided for an embodiment of the present invention; Figure 6 This is a schematic diagram of a guide interface for a tracking-type three-dimensional scanning calibration method provided in an embodiment of the present invention; Figure 7 A schematic diagram of a dynamic guide sign provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of a guide interface provided in an embodiment of the present invention when the pose constraint conditions are not met; Figure 9 This is a schematic diagram of a guide interface that satisfies pose constraints according to an embodiment of the present invention. Figure 10 This is a schematic diagram of the structure of a tracking-type three-dimensional scanning calibration device provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Currently, existing technologies typically use a calibration rod as a spatial reference target, which is either handheld by the operator or fixed in place by a fixture. This calibration rod is sampled in multiple positions and poses within the field of view of an external tracker consisting of two depth cameras. Its working principle is as follows: the tracker detects coded feature points on the calibration rod in real time, and continuously estimates the pose of the calibration rod by calculating the three-dimensional spatial coordinates of these feature points. After the entire sampling process is completed, the system centrally calculates calibration parameters and evaluates the calibration quality based on the recorded observation data.

[0024] However, this solution has the following technical problems in practical applications: Problem 1: The formulation of sampling paths is highly dependent on human experience. Operators lack quantitative basis and find it difficult to judge in real time which spatial regions or attitude intervals have not been fully sampled. This can easily lead to non-uniform distribution of samples in the six degrees of freedom space, resulting in local oversampling or undersampling, which in turn affects the global consistency and accuracy of the calibration solution.

[0025] Question 2: Existing solutions typically provide feedback on sampling status to operators through two-dimensional diagrams, text prompts, or post-sampling density reports. This non-intuitive guidance requires operators to mentally map abstract information to their actual three-dimensional physical space. This cognitive load not only reduces operational efficiency but may also introduce biases in spatial mapping, leading to duplicate or missed sampling.

[0026] Question 3: The assessment of calibration data quality usually occurs after all sampling operations are completed. If errors are found to exceed limits during this offline assessment phase, the operator often needs to discard all data from this round and start a completely new resampling process, which is costly in terms of time and manpower.

[0027] Question 4: In complex environments such as industrial production lines and outdoors, the sampling process is susceptible to dynamic interference such as obstruction, operator hand tremors, and partial loss of the tracker's field of view. Existing solutions lack real-time monitoring of these abnormal events and targeted dynamic resampling prompts or mechanisms, failing to guide operators to perform data repair at the optimal time and location. Once inferior data is mixed in, it will affect the stability of the final calibration results.

[0028] Question 5: The two depth cameras constituting the external tracker may differ in their hardware sampling frequency, internal data processing latency, and timestamp mechanism for generating observation data. If the dual data streams are not precisely time-aligned before calculating the 3D coordinates of feature points, feature point triangulation errors will be introduced, directly compromising the timing consistency and stability of the calibration rod pose calculation.

[0029] Question 6: The final output quality of the entire calibration process is highly sensitive to the operator's proficiency, experience, and the consistency of actions performed in each operation. This heavy reliance on human factors makes it difficult to reliably guarantee the repeatability and consistency of results between different operators, and even between multiple operations performed by the same operator, thus limiting its application in precision measurement scenarios.

[0030] Therefore, this invention provides a tracking-type 3D scanning calibration method, apparatus, and device. The main technical problem it solves is: how to directly overlay and display guidance information such as the target sampling area, recommended movement direction, real-time attitude deviation, and dynamic sampling status onto the corresponding real scene position in the depth image during the multi-directional calibration of the calibration rod, relying on the precise geometric mapping relationship between the depth scene image and the work space. When the position, attitude, and tracking stability of the calibration rod simultaneously meet preset conditions, it automatically triggers effective sampling records and subsequent calibration solutions, thereby reducing the operator's burden of understanding and converting abstract prompts, and improving the integrity of spatial coverage, the balance of attitude distribution, and the consistency of cross-calibration results.

[0031] To facilitate understanding of this embodiment, a detailed description of the tracking-type 3D scanning calibration method disclosed in this invention is provided first. This method is based on a calibration system, which includes a calibration structure, a 3D tracker, and a guidance display. The 3D tracker includes at least two depth sensors, which are devices capable of sensing depth information, such as depth cameras, and are used to collect observation data from the calibration structure. The initial display screen of the guidance display is a real / depth scene image corresponding to the work scene. That is, the display screen of the guidance display always uses the real / depth scene image as the underlying background image. During the guidance process, the visualized object of the calibration rod structure and its corresponding dynamic guidance markers are superimposed on the real scene position in the depth scene image, so that the guidance information is spatially aligned with the real scene position acquired by the depth camera. The work scene is divided into multiple sampling units, such as dividing the work scene according to preset region division rules (including spatial grid division, viewpoint distribution division, or trajectory planning division, etc.) to obtain multiple sampling units, see [link to relevant documentation]. Figure 1 The diagram shows a flowchart of a tracking-type three-dimensional scanning calibration method, which mainly includes the following steps S102 to S106: Step S102: Based on the observation data of the calibration structure, calculate the pose data of the calibration structure in the tracker coordinate system, so as to project the visualization object corresponding to the calibration structure onto the visualization area corresponding to the target sampling unit in the depth scene image.

[0032] The calibration structure is a rigid object (such as a calibration rod) with known coded feature points and 3D geometric relationships, used to provide structural constraints for spatial pose calculation. A sampling unit refers to the smallest effective sampling area in the working scene, while a target sampling unit refers to the sampling unit currently sampling data. The tracker coordinate system is a fixed reference coordinate system uniformly established by the depth sensor through pre-calibrated extrinsic parameters, serving as the benchmark for the entire system's pose calculation. Observation data refers to the image and depth information, including the calibration structure, synchronously acquired by the depth sensor in the 3D tracker. Visualized objects are graphical elements representing the current state of the calibration structure (such as models, axes, and markers), and must be consistent with its actual spatial pose. The visualization area corresponding to the target sampling unit is the display area obtained by mapping the 3D spatial range of that unit to the depth scene image via the depth sensor.

[0033] In one implementation, the calibration structure is identified based on observation data to determine its location region and related candidate feature information in the current observation. Then, feature calculation is performed based on the candidate feature information to obtain the pose data of the calibration structure in the tracker coordinate system. Combined with the pre-calibrated mapping model between the display coordinate system of the guide display and the tracker coordinate system, the visualization object corresponding to the calibration structure is mapped to the visualization area corresponding to the target sampling unit where it is currently located based on the above pose data.

[0034] Step S104: Using the visualized object as the mapping reference, project the dynamic guide markers corresponding to the pose data onto the depth scene image to guide the user to adjust the pose of the calibration structure according to the dynamic guide markers, thereby obtaining effective observation data and corresponding effective pose data.

[0035] The third visual state of the dynamic guidance sign includes pointing and visual salience (such as color, length, and flashing frequency). Pointing represents the recommended direction of movement, and visual salience represents the amount of posture adjustment. Both the visualized object and the third visual state are dynamically updated as the pose data changes. Effective observation data refers to the observation data within the target sampling unit that has a preset sampling coverage strategy (including coverage requirements, pose constraints, tracking stability thresholds, spatial distribution density, and re-sampling priority). Effective pose data is the six-degree-of-freedom pose of the calibration structure in the tracker coordinate system, calculated based on the effective observation data.

[0036] In one implementation, the real-time rendering position and pose of the visualized object in the depth scene image are used as a mapping reference. Based on the deviation between its current pose and the expected pose within the target sampling unit, a guidance flag is dynamically generated and updated. This flag is superimposed onto the corresponding spatial position in the depth scene image through the same projection link (i.e., the mapping model between the display coordinate system and the tracker coordinate system). The user is guided to move or rotate the calibration structure in the direction indicated by the flag, so that the visualized object gradually approaches the target area and pose. At the same time, the direction and visual salience of the guidance flag are dynamically enhanced or reduced based on the real-time changes in the deviation. When the pose constraint conditions are met, the current frame is locked, and its corresponding observation data is marked as valid observation data. The verified pose data is also marked as valid pose data.

[0037] Step S106: Calibrate the calibration structure based on the effective observation data and corresponding effective pose data in multiple target sampling units to obtain the calibration result.

[0038] In one implementation, the calibration results are obtained by solving calibration parameters for valid observation data and corresponding valid pose data that meet the pose constraints and have been filtered for validity, in combination with the known three-dimensional geometric relationships of the encoded feature points on the calibration structure. The calibration results include: the relative pose relationship between the coordinate systems of the depth sensors and the tracker coordinate system; the matching relationship between the observed three-dimensional coordinates of the encoded feature points in the tracker coordinate system and the known three-dimensional geometric relationships; and the spatial-pixel mapping relationship between the tracker coordinate system and the display coordinate system.

[0039] The tracking-type 3D scanning calibration method provided in this invention accurately projects the visualized object of the calibration structure and its dynamic guide markers onto the corresponding target sampling unit area in the depth scene image based on the real-time calculated pose data. This allows the operator to directly perform movement and posture adjustment actions based on the image content aligned with the actual work scene space, without the need for secondary understanding and mapping between the abstract schematic diagram and the physical space. The guidance information is updated in real time with changes in pose, and combined with the status feedback of the sampling unit, the operator can intuitively identify the covered and uncovered areas, thereby reducing the conversion cost from schematic information to actual actions and the dependence on calibration experience, while significantly improving the quality of spatial and posture coverage.

[0040] This invention, based on a 3D tracker consisting of two synchronously calibrated depth sensors, acquires a depth scene image and real-time six-DOF pose (i.e., pose data) of the calibration structure. The sampling unit boundaries, dynamic guidance markers, recommended movement directions, and pose adjustments are geometrically registered and directly superimposed onto the corresponding pixel positions of the depth scene image, according to their spatial poses in the real scene. The operator, using these visual cues that are strictly aligned with the real scene, completes calibration rod sampling under multiple spatial positions and pose angle combinations. Simultaneously, the system integrates features for 3D coordinate calculation, dynamic screening of sample validity, and calibration parameter solving.

[0041] To facilitate understanding, the calibration system will first be explained. The calibration system includes at least a calibration structure, a 3D tracker, and a guide display.

[0042] (a) Calibration structure (such as calibration rod): See Figure 2 The diagram illustrates the structure of a calibration rod, a rigid structural carrier used for spatial pose calculation and system calibration. The calibration rod has multiple coded feature points with known geometric relationships on its body. Each coded feature point is a marker point carrying coded information, and the distances, relative directions, topological connections, or preset three-dimensional coordinate relationships between the coded feature points are pre-known data. The calibration rod can adopt an asymmetrical layout to provide clear directional information and reduce attitude ambiguity. It should be noted that the calibration rod and coded point structure can adopt industry-standard structures. The improvement of this invention does not lie in the calibration rod hardware itself, but in using the real-time pose results of the calibration rod for deep scene overlay guidance, conditional trigger sampling, and closed-loop control for supplementary sampling.

[0043] (ii) 3D tracker: See Figure 3 The diagram shows a schematic of a 3D tracker. The 3D tracker is used to acquire the six-degree-of-freedom pose information of a calibration rod or related target in the tracker coordinate system. It consists of a first depth camera and a second depth camera, serving as the physical core for establishing the spatial reference and calculating the pose of the entire system. The computer software receives image data, depth map data, or point cloud data output from the tracker module, and sequentially performs coded feature point recognition, 3D coordinate calculation of feature points, and pose calculation of the calibration rod, generating display information for augmented reality guidance.

[0044] The first and second depth cameras are used to acquire depth scene images or point cloud data of the work space where the calibration rod is located. The two cameras can acquire the target surface depth information based on the principles of structured light, binocular stereo vision or laser triangulation, and map it to the same tracker coordinate system through pre-calibrated intrinsic parameters, distortion parameters and relative extrinsic parameters.

[0045] The 3D tracker constructs and maintains a unified spatial reference coordinate system, continuously observes multiple coded feature points attached to the calibration rod, and calculates their six-degree-of-freedom pose in the coordinate system in real time. Based on this pose, multiple frames of data collected by the scanning device during its movement can be dynamically registered to the same spatial reference, thereby achieving real-time positioning and spatial alignment of the scanning data, reducing or avoiding the cumulative errors introduced by post-processing stitching, and providing stable spatial constraints for 3D reconstruction and measurement, thus improving the overall accuracy and consistency of the results.

[0046] (iii) Guiding display: See Figure 4 The diagram shows a schematic of a guide display. The guide display is used to generate a target sampling area, a guide movement path, positional relationship prompts and attitude prompts based on a preset sampling strategy and the current sampling state. Based on the real-time pose of the calibration rod and the depth camera imaging model, the above prompt information is accurately superimposed and displayed on the corresponding physical position of the depth scene image after spatial-pixel mapping.

[0047] Furthermore, the aforementioned calibration system may also include a sampling and recording module, a data synchronization module, a calibration rod feature calculation module, a calibration solution module, and a quality assessment and re-sampling prompt module. The sampling and recording module records the depth scene image data, coded feature point observation data (i.e., effective observation data), the calculated six-degree-of-freedom pose data of the calibration rod (i.e., effective pose data), and sampling state data corresponding to the sampling moment when the calibration rod enters the target sampling unit and simultaneously meets preset position, attitude, and tracking stability conditions. The data synchronization module performs hardware-level or software-level time alignment of the observation data acquired by the first and second depth cameras, ensuring that the pose data upon which the display guidance depends is strictly consistent in time with the corresponding depth scene image. The calibration rod feature calculation module identifies and locates the coded feature points on the calibration rod from the observation data acquired by the first and second depth cameras, and, combined with their known three-dimensional geometric relationships, calculates the six-degree-of-freedom pose of the calibration rod in the tracker coordinate system. The calibration solution module is used to solve for the calibration parameters between the camera-robotic arm / calibration rod system based on valid observation data that meets the sampling conditions, valid pose data, time synchronization results of the dual depth cameras, and observation quality assessment information. The quality assessment and resampling prompt module is used to assess the quality of the current sampled data based on indicators such as pose residuals, spatial coverage, and attitude distribution uniformity, and generates resampling prompts accordingly. This module is optional.

[0048] The embodiments of the present invention further define the coordinate system involved in the tracking-type three-dimensional scanning calibration method, including: (1) a first depth camera coordinate system (1) Cartesian coordinate system defined by the first depth camera in the 3D tracker; (2) Second depth camera coordinate system : The Cartesian coordinate system defined by the second depth camera in the 3D tracker; (3) Tracker coordinate system By using the pre-calibrated relative extrinsic parameters between the first and second depth cameras, and (4) Display coordinate system : A two-dimensional coordinate system determined by the pixel plane of the depth scene image and its imaging projection relationship, used to characterize the pixel position of the guidance prompt in the display screen; (5) Known three-dimensional geometric relationship of the calibration rod: the spatial constraint relationship between each coded feature point on the calibration rod that has been preset and calibrated, including the Euclidean distance between them, relative orientation vector, topological connection relationship and / or preset three-dimensional coordinates in the calibration rod body coordinate system.

[0049] The objective of the solution required by this invention includes three types of spatial relationships: (1) the first depth camera coordinate system Second depth camera coordinate system With tracker coordinate system (2) Tracker coordinate system The matching relationship between the observed three-dimensional coordinates of the coded feature points and the known three-dimensional geometric relationships; (3) Tracker coordinate system With display coordinate system The spatial-pixel mapping relationship between them. The known three-dimensional geometric relationship of the calibration rod serves as a priori spatial constraint, characterizing the fixed Euclidean distance, relative orientation, and topological structure between each encoded feature point on it. The system solves for the calibration rod in the tracker coordinate system by minimizing the geometric deviation between the observed coordinates and this prior constraint. The six-degree-of-freedom pose is shown below.

[0050] Based on the above calibration system and coordinate system, this invention provides a specific implementation of a tracking-type three-dimensional scanning calibration method, which includes the following steps: Step S1: System initialization.

[0051] The computer software, the two depth cameras in the 3D tracker machine, and the guide display are started; the device calibration parameters, preset sampling coverage strategies, and timestamp synchronization source information are loaded; based on this, an end-to-end data acquisition, real-time pose calculation, and augmented reality display guide link is constructed.

[0052] The equipment calibration parameters include the intrinsic parameters and distortion coefficients of the first depth camera and the second depth camera, as well as the relative extrinsic parameters between them; The preset sampling coverage strategy includes sampling region division rules, coverage requirements, attitude constraints, tracking stability thresholds, spatial distribution density, and resampling priority. Among these, the sampling region division rules define the method for generating the geometric boundaries of the target sampling region in the workspace, including spatial grid-based division, viewpoint distribution-based division, or trajectory planning-based division; coverage requirements characterize the coverage level of the sampling units; attitude constraints specify the six-degree-of-freedom attitude range that the calibration rod must satisfy at each sampling point; the tracking stability threshold is a dynamic quality index threshold used to determine whether the current tracking state meets the sampling prerequisites, including the upper limit of pose jitter standard deviation, the upper limit of feature point reprojection error change rate, and the lower limit of continuous effective frame rate; spatial distribution density describes the local density control parameters of the sampling units in the three-dimensional position space; and resampling priority characterizes the execution order of unqualified areas or attitude intervals when triggering resampling, determined by a weighted average of residual size, coverage gap degree, and task criticality. The timestamp synchronization source information specifies the hardware or software clock source used to unify the data time base of each module, and the type of its synchronization mechanism.

[0053] Step S2, scene mapping initialization.

[0054] Obtain the calibration parameters of the two depth cameras in the 3D tracker (including their intrinsic parameters, distortion coefficients, and relative extrinsic parameters), the resolution parameters of the display device, and the display coordinate system. Define reference information for spatial alignment (including positioning markers, tooling reference points, known 3D coordinate points or pixel-3D point correspondence obtained from image registration), and depth images or point cloud data acquired synchronously by two depth cameras; Based on the calibration parameters, the 3D points reconstructed from the depth scene image are respectively transferred from the first depth camera coordinate system. Second Depth Camera Coordinate System Transform to a unified tracker coordinate system Then, based on the camera projection model and display resolution parameters, The target sampling unit, sampling envelope, or guide path in the image is projected onto the display coordinate system. This achieves strict alignment between three-dimensional spatial geometric elements and the pixel positions of the displayed image; The final result is: (1) Displaying the coordinate system With tracker coordinate system (1) A complete mapping model between the camera intrinsics, the extrinsic parameters from the camera coordinate system to the tracker coordinate system, the display resolution, and the projection relationship from the three-dimensional position to the display coordinate system; (2) An augmented reality visualization scene image superimposed with target sampling units, sampling envelopes, or guide paths; (3) Image superimposed display parameters for subsequent modules to call, including the pixel correspondence between the depth scene image and the display image, the display resolution, and the projection relationship from the three-dimensional position to the display coordinate system.

[0055] Step S3: Calibration rod identification and pose calculation.

[0056] Acquire observation data such as image data, depth map data, or point cloud data synchronously acquired by the first and second depth cameras in the 3D tracker, including information about the calibration pole and its surrounding spatial scene; simultaneously acquire candidate feature information extracted from the observation data (including image coordinates, 3D spatial coordinates, edge contours, or pole point sets), prior structural information of the calibration pole (including feature point encoding rules, known 3D geometric relationships, axis constraints, and pole dimension parameters), and complete calibration parameters of the 3D tracker (including intrinsic parameters of each camera, distortion coefficients, and coordinate system of the first depth camera). Second Depth Camera Coordinate System (relative external parameters between them) Optionally, a straight line fit or cylindrical fit is performed on the point set of the rod to estimate the axial direction of the rod; optionally, the end regions of the rod are identified and the three-dimensional coordinates of the endpoints are extracted; based on the known coded ID, distance between points, relative orientation, and topological order of the calibration rod, a one-to-one correspondence between the observation points and the prior points is established, their spatial consistency is verified, and least squares optimization, RANSAC robust estimation, or weighted registration strategies are used to suppress the influence of outliers and reduce the weight of abnormal observations; finally, based on the matching results of the coded feature points and the three-dimensional coordinates, the coordinates of the calibration rod in the tracker coordinate system are calculated. The six-degree-of-freedom pose is shown below.

[0057] Among them, the six-degree-of-freedom pose includes position coordinates ( ) and attitude parameters (direction vector, rotation matrix, quaternion or Euler angles), along with encoded feature points. The quality assessment information, such as the three-dimensional coordinate set, the number of visible points, the registration residual, and the observation confidence level, is used to drive subsequent position / attitude deviation calculations, sampling condition judgments, and supplementary sampling prompts.

[0058] Step S4, guided sampling based on depth scene images.

[0059] Obtain the preset sampling coverage strategy and calibration rod in the tracker coordinate system The real-time pose is determined according to this strategy. The calibration sampling range is divided into several spatially discrete sampling units. Based on the camera projection model and display parameters, the visualization area, recommended movement direction vector, current attitude deviation, completed area marker, and incomplete area marker corresponding to each sampling unit are mapped and superimposed on the corresponding physical position of the depth scene image in real time. The operator uses this visual guidance to move the calibration rod. When the calibration rod enters the target sampling unit and its pose simultaneously meets the preset position tolerance, attitude angle threshold, and tracking stability conditions, the system triggers sampling recording, and finally obtains effective observation data including the depth scene image at the sampling time and the corresponding calculated six-degree-of-freedom pose of the calibration rod.

[0060] The inputs for this step include: (1) the display coordinate system output from step S2. With tracker coordinate system The mapping relationship between the three-dimensional positions; (2) The calibration rod output in step S3 in the tracker coordinate system (2) Pose data and optional set of three-dimensional coordinates of encoded feature points or observation quality information; (3) Preset sampling coverage strategy, including sampling area division rules, coverage requirements, pose constraints, tracking stability threshold, spatial distribution density and resampling priority; (4) Deep scene image or point cloud visualization results.

[0061] Based on this, the specific implementation process of this step is as follows: (a) Target sampling unit generation.

[0062] The system adjusts the tracker coordinate system according to a preset sampling coverage strategy. The calibration sampling range is divided into multiple sampling units, and the preset sampling coverage strategy specifically includes: Spatial mesh generation: In the three-dimensional location space, the sampling range is discretized into regular or adaptive spatial voxel units to achieve full location coverage of the work space; Viewpoint distribution division: See Figure 5 The diagram shows a range division. Taking the center of the sampling range or the desired pose of the calibration rod as the reference point, discrete sampling directions are set according to the azimuth and pitch angles in the spherical or hemispherical attitude space to construct a sampling unit with multiple attitude angle combinations. Trajectory planning and partitioning: The sampling process is organized into an ordered sequence of path nodes, with each node corresponding to the tracker coordinate system. It includes predefined target position and attitude constraints, and supports dynamic advancement to the next node based on the current completion status.

[0063] Based on this, the work scenario can be divided into multiple sampling units, each corresponding to a visualization area, representing a specific space-attitude joint subdomain that needs to be effectively collected during this calibration process. The system only needs to guide the operator to cover the visualization area corresponding to the target sampling unit in this set, without exhaustively sampling the entire work space. Each sampling unit includes: spatial location range; desired attitude range, including axial direction constraints or Euler angle feasible region; and coverage priority or weight, used to dynamically recommend the next target area or determine the re-sampling order. Specifically, the next target sampling unit to be sampled is determined according to the following steps: (1.1) Obtain the coverage priority corresponding to each sampling unit in the job scenario.

[0064] (1.2) Determine the next target sampling unit to be sampled based on the coverage priority of the sampling unit that has not been sampled in the operation scenario.

[0065] (1.3) Call the pre-built mapping relationship between the tracker coordinate system and the display coordinate system corresponding to the guide display, project the three-dimensional spatial range of the target sampling unit in the tracker coordinate system to the display coordinate system, generate and render the visualization area of ​​the target sampling unit in the depth scene image in the specified first visual state.

[0066] Call the tracker coordinate system established in step S2 The display coordinate system corresponding to the guide display The mapping relationship between them will place the target sampling unit in The three-dimensional spatial range and attitude constraints defined in the code are mapped to the display coordinate system via geometric projection. Based on the mapping result, a corresponding visualization region is generated in the depth scene image and rendered in a first visual state. The first visual state includes highlight filling, border thickening or flashing outline, which is used to intuitively indicate that the target sampling unit is the sampling unit to be sampled or is being sampled.

[0067] Furthermore, the system maintains a set of sampling states for each sampling unit, including: unsampled units, sampled units, and the aforementioned target sampling unit. When the calibration rod enters a sampling unit and its pose simultaneously meets preset position tolerance, attitude threshold, and tracking stability conditions, the unit is marked as "sampled," and the overall sampling coverage state is updated in real time. Preferably, unsampled units, sampled units, and the aforementioned target sampling unit are presented in the depth scene image through mutually distinguishable visual styles (such as color, transparency, border style, or dynamic effects) to achieve intuitive recognition of state semantics.

[0068] (ii) Dynamic guidance sign projection.

[0069] (2.1) Normal boot state.

[0070] In one implementation, the visualization area corresponding to the target sampling unit in the depth scene image is maintained in a specified first visual state to indicate that the target sampling unit has not completed sampling. A guide path is generated based on pose data, and the guide path is rendered in the visualization area. The guide path is used to guide the user to move the calibration structure to the target sampling unit. The second visual state of the guide path represents the distance between the calibration structure and the target sampling unit, and its visual attributes (including length, thickness, transparency, color saturation, or dynamic flicker frequency) are continuously and monotonically updated dynamically as the distance between them changes. For example, when the calibration pole is far away from the target area, the guide path is displayed as a long, thin, low-saturation gray dashed line; as the distance decreases, the path gradually shortens, thickens, and increases in saturation, and changes from gray to green; when it enters a threshold range, the path becomes a short, thick solid line with a slight pulsating effect.

[0071] For example, see Figure 6 The diagram shows a guide interface for a tracking-based 3D scanning calibration method. The display and behavior logic when not entering target sampling is as follows: In the depth scene image, the target sampling unit is displayed in a first visual state (such as a highlighted border or semi-transparent fill); a guide path is overlaid, which can be used to indicate the recommended movement direction, in the form of a vector arrow pointing to the center of the target sampling unit or a segmented guide path; the current position of the calibration rod in the tracker coordinate system (such as a 3D coordinate point or simplified geometry) is rendered in real time.

[0072] The system dynamically updates the position display of the calibration rod in the depth scene image and the arrow direction and length of the guide path based on the real-time pose data of the calibration rod calculated in step S3. When the operator moves along the recommended direction and the position deviation continues to decrease, the system represents the degree of approach through visual feedback (such as the arrow color gradually changing from cool to warm, or the superimposed "approaching" status prompt).

[0073] (2.2) Entering the target sampling unit but the attitude does not meet the requirements.

[0074] In one implementation, the specific process is as follows: (2.21) Maintain the visualization area corresponding to the target sampling unit in the depth scene image rendered in the specified first visual state to characterize that the target sampling unit has not completed sampling.

[0075] (2.22) Determine whether the pose data satisfies the pose constraint conditions corresponding to the target sampling unit. In one embodiment, calculate the Euclidean distance deviation between its position in the tracker coordinate system and the center point of the preset spatial range of the target sampling unit, and the angular distance or component deviation (i.e., the pose deviation) between its attitude parameters (rotation matrix, quaternion or Euler angle) in the tracker coordinate system and the center of the preset feasible region of the target sampling unit; if any deviation exceeds the corresponding preset threshold, it is determined that the pose constraint conditions of the target sampling unit are not satisfied.

[0076] (2.23) If not, based on the pose data and pose constraints, determine the recommended movement direction and attitude adjustment amount corresponding to the calibration structure.

[0077] In one implementation, the system calculates the Euclidean distance deviation and attitude deviation based on the spatial relationship between the current pose of the calibration rod, the center position of the target sampling unit, and the desired pose, thereby obtaining the recommended movement direction and attitude adjustment amount.

[0078] Furthermore, by comparing the error magnitudes of Euclidean distance deviation and attitude deviation with their corresponding thresholds, the deviations can be divided into three levels: satisfying the condition (error ≤ threshold), nearly satisfying the condition (threshold < error ≤ 1.2 × threshold), and not satisfying the condition (error > 1.2 × threshold). This level of status is used to drive visual feedback color mapping, sampling trigger determination, and re-sampling priority ranking.

[0079] (2.24) Using a specified point in the visualized object as the rendering reference, and based on the recommended movement direction and posture deviation, render dynamic guide signs corresponding to the calibration structure in the visualized area. The third visual state of the dynamic guide signs includes pointing and visual salience. Pointing represents the recommended movement direction and is presented intuitively as a vector arrow, path tangent, or coordinate system rotation axis direction. Visual salience represents the posture adjustment amount and is quantified through visual attributes such as arrow length, thickness, brightness, color saturation, or dynamic pulse frequency. Both the visualized object and the third visual state are dynamically updated as the pose data changes. For example... Figure 7 The diagram shows a dynamic guide sign used to display prompts for incorrect angle, incorrect position, and correct and properly positioned.

[0080] For example, when the attitude deviation is large, the arrow points to the target attitude direction and appears as a long, thick, bright red arrow that continuously pulsates; as the deviation decreases, the arrow gradually shortens, becomes thinner, and its saturation decreases, gradually changing from red to yellow and then to green; when the deviation approaches zero, the arrow shrinks into a short scale indicator ring, and the pulsation disappears.

[0081] For example, using a preset reference point in the visualized object (such as the center of the target sampling area, the origin of the calibration rod coordinate system, or the starting point of the guide path) as the rendering anchor point, and based on the calculated recommended movement direction vector and attitude deviation, the guide markers corresponding to the calibration structure are dynamically generated and rendered in the depth scene image. The guide markers include, but are not limited to: directional arrows (with the direction consistent with the recommended movement vector), attitude deviation angle dials (angle values ​​are updated in real time with the deviation), or direction correction indicator rings superimposed on the calibration rod model. Their spatial position, orientation, and morphological parameters are all displayed in the coordinate system. The pixel mapping relationship is updated in real time.

[0082] Furthermore, if the operator does not move as required, the following procedure can be used to guide the operator's movement: determine the change between the pose error of the current frame's pose data relative to the pose constraints and the pose error of the previous frame's pose data relative to the pose constraints; if the change indicates that the pose of the calibration structure is approaching the pose constraints, then the visual salience is reduced; if the change indicates that the pose of the calibration structure deviates from the pose constraints, then the visual salience is increased. See also Figure 8 The diagram shown illustrates a guide interface when pose constraints are not met. The specific implementation process is as follows: The target sampling area remains continuously highlighted; attitude deviation prompts are simultaneously presented through dynamic guidance signs, including directional arrows, real-time angle values, or attitude indicators; when the calibration rod is detected to be adjusting towards the target attitude and the deviation is continuously decreasing, the interface updates the attitude error display in real time, which is manifested as a decreasing value or a gradual change in color from cool colors (such as blue) to warm colors (such as green); if the adjustment direction is correct, the visual intensity of the guidance signs gradually weakens (such as shortening the arrow length, increasing transparency, and reducing the flashing frequency); if the adjustment direction is incorrect, i.e., the deviation increases or the direction of movement is opposite to the recommended attitude correction direction, the guidance signs immediately update their direction in the opposite direction and simultaneously strengthen the visual intensity (such as thickening the arrow, changing the color to a warning color, increasing the flashing frequency, or superimposing the text prompt "Deviation in Progress"); the system continuously recalculates and updates the recommended direction based on the latest pose to ensure that the guidance information always points to the target attitude.

[0083] In this embodiment of the invention, the system tracks the pose of the calibration rod in real time across consecutive image frames and dynamically presents its movement process in the following ways: continuously updating the current position marker or historical trajectory line of the calibration rod in the depth scene image; adjusting the recommended movement direction indicator (including arrow direction, length, and orientation) in real time based on the spatial relationship between the current pose and the target area; and synchronously updating the numerical display of position and posture deviations and associated visual cues (such as color, size, or dynamic effects) so that they change continuously and gradually with the movement process. This mechanism constitutes a spatial-visual closed-loop feedback, supporting the operator to continuously correct the pose of the calibration rod based on real-time, consistent, and interpretable augmented reality guidance.

[0084] Furthermore, if the operator fails to move as required, and the deviation continuously exceeds the threshold for a preset duration, a re-guidance prompt or automatic switching to the next sampling unit can be triggered. Specifically: when the calibration structure fails to enter the target sampling unit within multiple consecutive frames, a timeout timer is started; if the calibration structure still fails to enter the target sampling unit after a preset time threshold has elapsed since the timeout timer started, it is determined to be a guidance failure; in response to the guidance failure determination, at least one of the following operations is performed: clear the dynamic guidance flag and re-render the dynamic guidance flag corresponding to the pose data based on the same target sampling unit; determine a new target sampling unit based on the coverage priority corresponding to the sampling units that have not completed sampling in the work scene. The specific implementation process is as follows: When the calibration rod is in continuous If the position and orientation constraints of the target sampling unit are not met within any frame, the system starts a timeout timer; if a preset time threshold has elapsed since the timer started... If, after 8 seconds (for example), the calibration rod still has not entered the target sampling unit, the guidance is deemed to have failed. In response to this determination, the system performs at least one of the following operations: immediately clear the currently displayed arrows, angle values, and deviation scale rings, and recalculate the recommended movement direction and attitude correction based on the pose data of the latest frame to redraw the guidance marks; or: traverse all incomplete sampling units, select the one with the highest coverage priority value, generate a new target area highlight rendering and corresponding guidance path based on its corresponding pose condition constraints, and pop up a prompt at a specified location on the interface (such as the upper right corner): "Switch to the next target sampling unit: XXX".

[0085] (3) The pose constraints are satisfied.

[0086] If the pose data satisfies the pose constraints corresponding to the target sampling unit, the visual region corresponding to the target sampling unit in the depth scene image is rendered in a specified fourth visual state to indicate that the target sampling unit has completed sampling. The fourth visual state is used to explicitly indicate that the target sampling unit has completed effective sampling; its visual attributes (including fill color, border style, transparency, icon markings, or dynamic effects) are significantly distinguishable from the incomplete / in-process state and remain stably rendered until the sampling process ends or the state is reset. Figure 9 The diagram shows a guide interface that meets pose constraints. For example, the target area is rendered with a green semi-transparent fill and a light gray border, and a checkmark icon of a fixed size is superimposed in its center; or a low-frequency smooth breathing micro-flicker (frequency ≤ 0.5Hz) is used, which is different from the high-frequency warning flashing of the highlight prompt, to convey the semantics of "confirmation completed", and the current calibration rod status is displayed as "valid sampling".

[0087] The observation data of the current frame is used as valid observation data, and the pose data calculated based on the observation data of the current frame is used as valid pose data. In one embodiment, when the calibration rod enters the target sampling unit and its pose simultaneously meets the preset position tolerance, attitude threshold and tracking stability conditions, the system marks the depth scene image data and coded feature point observation data (including two-dimensional image coordinates and corresponding depth values) corresponding to the current image frame as valid observation data; and the six-degree-of-freedom pose of the calibration rod calculated based on the data of this frame is simultaneously marked as valid pose data and stored in the sampling record set.

[0088] Further, based on the aforementioned coverage priority, the next target sampling unit to be sampled is determined, the sampling coverage index is updated, and the sampling coverage index is refreshed in real time on the display screen of the guide display. The sampling coverage index includes the number of completed sampling units, the total number of sampling units, and the coverage percentage. Specifically, the system counts the number of completed sampling units and the total number of sampling units in real time, calculates the sampling coverage percentage based on this, and uses these three indicators (number of completed units, total number, and coverage percentage) as the sampling coverage index; through the display coordinate system established in step S2... The mapping relationship renders the indicator as structured text overlaid on a designated area of ​​the guide display screen, and dynamically refreshes it with each valid sample trigger.

[0089] (4) Obstruction or abnormal observation.

[0090] In one implementation, an abnormal observation state is identified based on the number of visible coded feature points in the observation data and a preset feature point number threshold. In response to the identification of the abnormal observation state, an abnormal state prompt is overlaid in the depth scene image, and the update of the third visual state of the dynamic guidance sign is paused. When the number of visible coded feature points in the observation data in multiple consecutive frames meets the feature point number threshold, the third visual state of the dynamic guidance sign is updated or initialized.

[0091] In practical applications, the calibration rod encoding feature points may become invisible or insufficient in number. Causes include occlusion, viewing angle exceeding the effective recognition range, surface reflection, abnormal image exposure, motion blur, feature points located outside the camera's field of view, or invalid corresponding depth values. A semantically clear prompt, such as "Target Invisible" or "Insufficient Feature Points," is overlaid on the depth scene image, and the refresh of the sampling status (including completed marking, coverage statistics, and guidance flag updates) is paused. If the feature points become stably visible in subsequent frames, the guidance and judgment process of the current sampling unit is automatically continued; or, a re-initialization operation is triggered, clearing the current guidance status of the unit and reconstructing the dynamic guidance flags and deviation calculation benchmark based on the latest valid observation data.

[0092] The final output includes the following results: valid sampling data that meets the sampling conditions, including depth scene images or point cloud data acquired at the sampling time (i.e., valid observation data), and the calibration rod six-degree-of-freedom pose calculated based on the data (i.e., valid pose data); sampling status information, covering the completion mark, incomplete mark and overall sampling coverage of each sampling unit; and a set of valid data that has been screened and verified and can be used for subsequent calibration model solving.

[0093] Step S5, optional dual depth camera data synchronization processing.

[0094] When there is a time discrepancy between the first depth camera and the second depth camera, the system performs sub-frame-level time alignment of the two observation data based on the timestamp, selects the synchronous data group that is closest in time, including depth image, point cloud or coded feature point observation results, for subsequent pose calculation and calibration parameter solution, so as to suppress the spatial registration error caused by asynchronous acquisition.

[0095] Step S6: Solve for calibration parameters.

[0096] The system solves for calibration parameters on the sampled data that meet the sampling conditions and have been filtered for validity. The data includes depth images or point clouds at the sampling time and the six-degree-of-freedom pose of the calibration rod calculated based on the data. The system combines the known three-dimensional geometric structure of the coded feature points on the calibration rod to jointly optimize and solve multiple sets of valid samples at different positions and attitudes to obtain the final calibration result.

[0097] During the solution process, the system dynamically assigns weights based on the quality indicators of each valid sample. The quality indicators include: the number of visible encoded feature points, the feature point reprojection residual, the percentage of valid pixels in the depth data, the consistency error between the two depth cameras, the occlusion detection status, and the completion status of the corresponding sampling unit.

[0098] Step S7, quality assessment and optional supplementary sampling.

[0099] The sampling quality evaluation indicators for the effective observation data and corresponding effective pose data of the calibration structure within the target sampling unit are determined, including residual distribution, spatial coverage, pose distribution uniformity, or observability indicators. The residual distribution is obtained by statistically analyzing the spatial error between the observation point and the predicted point of the calibration model; the spatial coverage is determined by the ratio of the number of completed sampling units to the total number of target sampling units; the pose distribution uniformity is determined by the dispersion or coverage entropy of the sampled poses within a preset pose space (such as a spherical or Euler angle interval); and the observability indicators are calculated comprehensively based on the number of visible coded feature points, the angle between the calibration rod and the camera, the effectiveness of depth data, the reprojection error of feature points, and the condition number of the Jacobian matrix or information matrix in pose calculation.

[0100] When the sampling quality assessment indicators meet the resampling conditions, the visual area corresponding to the target sampling unit in the depth scene image is rendered using the specified fifth visual state. The fifth visual state clearly identifies that the sampling unit requires resampling. Its visual attributes (such as border style, fill mode, and dynamic effects) are significantly distinguishable from the three states of "not sampled," "in progress," and "completed," and it possesses identifiable warning or prompt semantics. For example, the area is rendered with a yellow dashed border overlaid with a slowly flashing (1Hz) semi-transparent orange fill, and a rotating arrow icon is overlaid in its center, simultaneously displaying the text prompt "Resampling recommended."

[0101] Once the system determines that the calibration has passed, it outputs the final calibration parameters and simultaneously generates residual distribution statistics, sampling trajectory sequences, and calibration quality assessment information, which are then written into the device configuration file or calibration database.

[0102] In summary, the core inventive point of the tracking-type 3D scanning calibration method provided in this embodiment of the invention lies not in the simple display of depth images, nor in the independent execution of calibration parameter solving, but in: constructing a depth scene image and a tracker coordinate system based on the depth scene image acquired by an external tracker composed of two depth cameras and the real-time pose of the tracked target (calibration rod). The system establishes a strict geometric-semantic mapping relationship between the calibration sampling space and the preset sampling conditions. Based on this, guidance information such as the target sampling area, recommended movement direction, attitude deviation, sampling status, and coverage priority are precisely superimposed onto the corresponding real spatial position in the depth scene image, forming an augmented reality closed-loop guidance for the physical operation process. When the calibration rod simultaneously meets the preset position conditions, attitude constraints, and tracking stability conditions, it automatically triggers the recording of valid observation data. Finally, by combining the time synchronization compensation of the dual depth cameras, sample validity screening, and weighted calibration parameter solution, a high-confidence external tracker calibration result is output.

[0103] Compared to existing solutions that rely on abstract diagrams and human experience, this invention integrates scene guidance, action execution, condition-triggered sampling, quality assessment, and parameter solving into a closed-loop process. This significantly reduces the operator's spatial mapping cognitive load, improves the spatial consistency and posture distribution uniformity of sampling coverage, and effectively avoids full-round resampling due to missing local coverage. The essential differences of this solution are reflected in the following four aspects: All guiding elements (target area, directional arrows, attitude indicators, status markers) are traversed through a coordinate system. With display coordinate system The mapping relationship is strictly projected onto the corresponding real 3D position in the depth scene image, rather than presented as an independent window or abstract diagram detached from the scene; the operator can directly complete the pose adjustment based on the visual cues superimposed in the image, without the need for secondary understanding and coordinate conversion between the abstract interface and the real space; the guidance process itself drives the sampling trigger logic, and the corresponding observation data is only recorded when the pose meets the multidimensional constraints, ensuring that each valid sample has a clear physical semantics and acquisition context; in some embodiments, the system can dynamically evaluate the calibration quality based on residual distribution, spatial coverage, pose uniformity and observability indicators, and generate supplementary sampling prompts when necessary, realizing a closed-loop extension from single sampling to global calibration quality convergence.

[0104] Based on the foregoing embodiments, this invention provides a tracking-type 3D scanning calibration device. This device is implemented based on a calibration system, which includes a calibration structure, a 3D tracker, and a guide display. The 3D tracker includes at least two depth sensors, which are used to collect observation data from the calibration structure. The guide display initially displays a depth scene image corresponding to the work scene, which is divided into multiple sampling units. (See [link to relevant documentation]). Figure 10 The schematic diagram shown illustrates the structure of a tracking-type three-dimensional scanning calibration device, which includes: The calibration structure projection module 1002 is used to calculate the pose data of the calibration structure in the tracker coordinate system based on the observation data of the calibration structure, so as to project the visualization object corresponding to the calibration structure onto the visualization area corresponding to the target sampling unit in the depth scene image. The guidance sign projection module 1004 is used to project the dynamic guidance sign corresponding to the pose data onto the depth scene image using the visualized object as a mapping reference, so as to guide the user to adjust the pose of the calibration structure according to the dynamic guidance sign and obtain effective observation data and corresponding effective pose data. The calibration module 1006 is used to perform calibration based on the effective observation data and corresponding effective pose data of the calibration structure in multiple target sampling units, and obtain the calibration result.

[0105] The tracking-type 3D scanning calibration device provided in this invention accurately projects the visualized object of the calibration structure and its dynamic guide markers onto the corresponding target sampling unit area in the depth scene image based on the real-time calculated pose data. This allows the operator to directly perform movement and posture adjustment actions based on the image content aligned with the actual work scene space, without the need for secondary understanding and mapping between the abstract schematic diagram and the physical space. The guidance information is updated in real time with changes in pose, and combined with the status feedback of the sampling unit, the operator can intuitively identify the covered and uncovered areas, thereby reducing the conversion cost from schematic information to actual actions and the dependence on calibration experience, while significantly improving the quality of spatial and posture coverage.

[0106] In one embodiment, a region projection module is further included, for: Obtain the coverage priority corresponding to each sampling unit in the job scenario; Based on the coverage priority of the sampling units that have not been sampled in the operation scenario, determine the next target sampling unit to be sampled; The mapping relationship between the pre-built tracker coordinate system and the display coordinate system corresponding to the guide display is invoked to project the three-dimensional spatial range of the target sampling unit in the tracker coordinate system to the display coordinate system, and generate and render the visualization area of ​​the target sampling unit in the depth scene image in the specified first visual state.

[0107] In one embodiment, the guide sign projection module 1004 is further configured to: Maintain the visualization area corresponding to the target sampling unit in the depth scene image in the specified first visual state to represent that the target sampling unit has not completed sampling; A guide path is generated based on pose data, and the guide path is rendered in a visual region. The guide path is used to guide the user to move the calibration structure to the target sampling unit. The second visual state of the guide path represents the distance between the calibration structure and the target sampling unit, and it is dynamically updated as the distance between the calibration structure and the target sampling unit changes.

[0108] In one embodiment, the guide sign projection module 1004 is specifically used for: Determine whether the pose data satisfies the pose constraint conditions corresponding to the target sampling unit; If not, based on the pose data and pose constraints, determine the recommended movement direction and attitude adjustment amount corresponding to the calibration structure; Using a specified point in the visualized object as the rendering reference, and based on the recommended movement direction and attitude deviation, render the dynamic guidance flag corresponding to the calibration structure in the visualized area. The third visual state of the dynamic guidance sign includes direction and visual salience. Direction represents the recommended direction of movement, and visual salience represents the amount of posture adjustment. Both the visualized object and the third visual state are dynamically updated as the pose data changes.

[0109] In one embodiment, the guide sign projection module 1004 is specifically used for: Maintain the visualization area corresponding to the target sampling unit in the depth scene image in the specified first visual state; Determine the change between the pose error of the current frame's pose data relative to the pose constraints and the pose error of the previous frame's pose data relative to the pose constraints. If the change in the value represents the pose of the calibration structure and approaches the pose constraint, then the visual saliency is reduced. If the change represents a deviation of the pose of the calibration structure from the pose constraints, then the visual saliency is enhanced.

[0110] In one embodiment, the guide sign projection module 1004 is further configured to: If the pose data satisfies the pose constraints corresponding to the target sampling unit, render the visualization area of ​​the target sampling unit in the depth scene image in the specified fourth visual state to represent that the target sampling unit has completed sampling. The observation data of the current frame is used as the effective observation data, and the pose data calculated based on the observation data of the current frame is used as the effective pose data. The method also includes updating the sampling coverage index and refreshing the sampling coverage index in real time on the display screen of the guide display. The sampling coverage index includes the number of completed sampling units, the total number of sampling units, and the coverage percentage.

[0111] In one embodiment, the guide sign projection module 1004 is further configured to: If the calibration structure fails to enter the target sampling unit within multiple consecutive frames, a timeout timer is started. If the calibration structure still fails to enter the target sampling unit after a preset time threshold has elapsed since the start timeout timer, it is determined to be a guidance failure. In response to a boot failure determination, perform at least one of the following operations: Clear the dynamic guidance flags and re-render the dynamic guidance flags corresponding to the pose data based on the same target sampling unit; determine the new target sampling unit based on the coverage priority of the sampling units that have not completed sampling in the operation scenario.

[0112] In one embodiment, the guide sign projection module 1004 is further configured to: Based on the number of visible coded feature points in the observation data and the preset feature point number threshold, abnormal observation states are identified. In response to the identification of anomalies observed, anomaly status prompts are overlaid on the depth scene image, and updates to the third visual status of dynamic guidance signs are paused. When the number of visible coded feature points in the observation data of multiple consecutive frames meets the feature point number threshold, the third visual state of the dynamic guidance sign is updated or initialized.

[0113] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0114] This invention provides an electronic device, specifically, the electronic device includes a processor and a memory; the memory stores a computer program, which, when run by the processor, executes the method described in any of the above embodiments.

[0115] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 100 includes: a processor 110, a memory 111, a bus 112, and a communication interface 113. The processor 110, the communication interface 113, and the memory 111 are connected through the bus 112. The processor 110 is used to execute executable modules, such as computer programs, stored in the memory 111.

[0116] The memory 111 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 113 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0117] Bus 112 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0118] The memory 111 is used to store programs. After receiving an execution instruction, the processor 110 executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 110 or implemented by the processor 110.

[0119] Processor 110 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 110 or by instructions in software form. Processor 110 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 111, and processor 110 reads the information in memory 111 and, in conjunction with its hardware, completes the steps of the above method.

[0120] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0121] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A tracking-type three-dimensional scanning calibration method, characterized in that, The method is based on a calibration system, which includes a calibration structure, a 3D tracker, and a guide display. The 3D tracker includes at least two depth sensors, which are used to acquire observation data from the calibration structure. The guide display initially displays a depth scene image corresponding to the work scene, which is divided into multiple sampling units, including: Based on the observation data within the calibration structure, the pose data of the calibration structure in the tracker coordinate system is calculated, so as to project the visualization object corresponding to the calibration structure onto the visualization area corresponding to the target sampling unit in the depth scene image; Using the visualized object as a mapping reference, the dynamic guide markers corresponding to the pose data are projected onto the depth scene image to guide the user to adjust the pose of the calibration structure according to the dynamic guide markers, thereby obtaining effective observation data and corresponding effective pose data. The calibration structure is calibrated based on the effective observation data and corresponding effective pose data in multiple target sampling units to obtain the calibration result.

2. The tracking-type three-dimensional scanning calibration method according to claim 1, characterized in that, Before calculating the pose data of the calibration structure in the tracker coordinate system based on the observation data of the calibration structure, the method further includes: Obtain the coverage priority corresponding to each sampling unit included in the operation scenario; Based on the coverage priority corresponding to the sampling unit that has not completed sampling in the operation scenario, determine the next target sampling unit to be sampled; The mapping relationship between the pre-built tracker coordinate system and the display coordinate system corresponding to the guide display is invoked to project the three-dimensional spatial range of the target sampling unit in the tracker coordinate system to the display coordinate system, and generate and render the visualization area corresponding to the target sampling unit in the depth scene image in a specified first visual state.

3. The tracking-type three-dimensional scanning calibration method according to claim 1, characterized in that, Before projecting the dynamic guide markers corresponding to the pose data onto the depth scene image using the visualized object as a mapping reference, and guiding the user to adjust the pose of the calibration structure according to the dynamic guide markers to obtain effective observation data and corresponding effective pose data, the method further includes: The target sampling unit is rendered in a specified first visual state, representing the visual region in the depth scene image that the target sampling unit has not completed sampling. A guide path is generated based on the pose data and rendered in the visualization area. The guide path is used to guide the user to move the calibration structure to the target sampling unit. The second visual state of the guide path represents the distance between the calibration structure and the target sampling unit and is dynamically updated as the distance between the calibration structure and the target sampling unit changes.

4. The tracking-type three-dimensional scanning calibration method according to claim 1, characterized in that, Using the visualized object as a mapping reference, the dynamic guide markers corresponding to the pose data are projected onto the depth scene image, including: Determine whether the pose data satisfies the pose constraint conditions corresponding to the target sampling unit; If not, based on the pose data and the pose constraints, determine the recommended movement direction and attitude adjustment amount corresponding to the calibration structure; Using a specified point in the visualization object as a rendering reference, and based on the recommended movement direction and the attitude deviation, render the dynamic guidance mark corresponding to the calibration structure in the visualization area; The third visual state of the dynamic guide sign includes pointing and visual salience. The pointing represents the recommended movement direction, and the visual salience of the third visual state represents the posture adjustment amount. Both the visualized object and the third visual state are dynamically updated as the pose data changes.

5. The tracking-type three-dimensional scanning calibration method according to claim 4, characterized in that, Rendering the dynamic guide flag corresponding to the calibration structure in the depth scene image includes: Determine the change between the pose error of the pose data in the current frame relative to the pose constraints and the pose error of the pose data in the previous frame relative to the pose constraints. If the change indicates that the pose of the calibration structure is approaching the pose constraint, then the visual saliency is reduced. If the change represents a deviation of the pose of the calibration structure from the pose constraints, then the visual saliency is enhanced.

6. The tracking-type three-dimensional scanning calibration method according to claim 4, characterized in that, The method further includes: When the pose data satisfies the pose constraint conditions corresponding to the target sampling unit, the visualization area corresponding to the target sampling unit in the depth scene image is rendered in a specified fourth visual state to indicate that the target sampling unit has completed sampling. The observation data of the current frame is taken as the valid observation data, and the pose data calculated based on the observation data of the current frame is taken as the valid pose data. The method further includes: updating the sampling coverage index and refreshing the sampling coverage index in real time on the display screen of the guide display. The sampling coverage index includes the number of completed sampling units, the total number of sampling units, and the coverage percentage.

7. The tracking-type three-dimensional scanning calibration method according to claim 1, characterized in that, The method further includes: If the calibration structure does not enter the target sampling unit within multiple consecutive frames, a timeout timer is started. If the calibration structure still fails to enter the target sampling unit after a preset time threshold has elapsed since the start of the timeout timer, it is determined to be a guidance failure. In response to the determination of boot failure, perform at least one of the following operations: The dynamic guidance flag is cleared, and the dynamic guidance flag corresponding to the pose data is re-rendered based on the same target sampling unit; a new target sampling unit is determined based on the coverage priority of the sampling unit that has not completed sampling in the operation scenario.

8. The tracking-type three-dimensional scanning calibration method according to claim 1, characterized in that, The method further includes: Based on the number of visible coded feature points in the observation data and a preset feature point number threshold, abnormal observation states are identified. In response to the identification of the observed abnormal state, an abnormal state prompt is overlaid and displayed in the depth scene image, and the updating of the third visual state of the dynamic guidance sign is paused. When the number of visible coded feature points in the observation data within multiple consecutive frames meets the feature point number threshold, the third visual state of the dynamic guidance sign is updated or initialized.

9. The tracking-type three-dimensional scanning calibration method according to claim 1, characterized in that, Before obtaining the calibration result by calibrating the effective observation data and corresponding effective pose data in multiple target sampling units based on the calibration structure, the method further includes: Determine the effective observation data of the calibration structure within the target sampling unit and the corresponding sampling quality evaluation index of the effective pose data; If the sampling quality evaluation index meets the resampling conditions, the visualization area corresponding to the target sampling unit in the depth scene image is rendered in the specified fifth visual state.

10. A tracking-type three-dimensional scanning calibration device, characterized in that, The device is based on a calibration system, which includes a calibration structure, a 3D tracker, and a guide display. The 3D tracker includes at least two depth sensors, which are used to collect observation data from the calibration structure. The guide display initially displays a depth scene image corresponding to the work scene, which is divided into multiple sampling units, including: The calibration structure projection module is used to calculate the pose data of the calibration structure in the tracker coordinate system based on the observation data of the calibration structure, so as to project the visualization object corresponding to the calibration structure onto the visualization area corresponding to the target sampling unit in the depth scene image. The guide sign projection module is used to project the dynamic guide sign corresponding to the pose data onto the depth scene image, using the visualized object as a mapping reference, to guide the user to adjust the pose of the calibration structure according to the dynamic guide sign, so as to obtain effective observation data and corresponding effective pose data. The calibration module is used to calibrate the calibration structure based on the effective observation data and the corresponding effective pose data in the multiple target sampling units, and obtain the calibration result.

11. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 9.

Images