A Scan Path Planning and Missed Scan Compensation Method Based on Spatial Tracking

Abstract

The present invention discloses a scanning path planning and missed scan compensation method based on spatial tracking, comprising the following steps: acquiring the posture data of a spatial tracker fixedly connected to a handheld 3D scanner in real time, and synchronously collecting 3D point cloud data; performing coordinate transformation on the data, calculating a comprehensive quality weight, assigning the quality weight to each 3D point, and constructing a 3D coverage map in real time, identifying the actual missed scan area through a spatial data structure and a clustering algorithm; constructing a mixed cost function of the operating ergonomic cost and the scanning task efficiency cost for the identified missed scan area, and dynamically planning an optimal re-scanning path; visualizing the optimal path through a virtual reality device with a 3D virtual path belt and a holographic scanner model, and immersively guiding the operator to complete the re-scan efficiently and accurately; the present invention significantly improves the overall operating efficiency of the operator in scanning data through objective missed scan identification, optimal path planning, and immersive operation guidance.

Description

Technical Field

[0001] This invention relates to the fields of three-dimensional measurement and virtual reality technology, and in particular to a scanning path planning and missed scan compensation method based on spatial tracking. Background Technology

[0002] 3D digitization technology, as a key bridge connecting the physical and digital worlds, plays a crucial role in fields such as reverse engineering, industrial design, quality inspection, cultural relic preservation, and virtual reality. Among these, handheld 3D scanning systems, represented by laser triangulation or structured light technology, have become the mainstream tool for acquiring 3D morphological information of complex curved objects due to their high precision, efficiency, and portability. To overcome the limitations of traditional fixed scanners in terms of measurement range and achieve flexible scanning of certain workpieces (such as automotive body-in-white, aircraft engine blades, and wind turbine blades), the industry commonly adopts a solution that combines handheld scanners with external large-space, high-precision spatial tracking systems. This solution tracks optical markers fixed to the scanner in real time to accurately calculate the scanner's pose in the global coordinate system, thereby seamlessly and automatically registering local point cloud data collected from various perspectives to a unified coordinate system. This eliminates tedious post-processing stitching steps, greatly improving the freedom and overall efficiency of data acquisition.

[0003] However, despite the improvements in flexibility and automation of scanning operations, the inherent shortcomings of existing scanning processes become increasingly apparent in practical applications, especially when dealing with large workpieces with complex structures and numerous features. First, current mainstream scanning operations heavily rely on the operator's personal experience and subjective judgment. Operators must constantly allocate their attention among observing the physical workpiece, maintaining the appropriate scanner posture, and reviewing the point cloud model generated on the computer screen. This separation of human-machine-object interaction leads to extremely high cognitive load. Missed scans often arise from visual fatigue, distraction, or blind spots caused by geometric occlusion of the workpiece, and the identification and compensation process lacks objective standards and systematic guidance. Second, existing technologies lack path planning for compensation in missed scan areas. When an operator discovers a missing data point, their choice of a replacement scanning path is usually arbitrary and suboptimal, relying solely on intuition to move the scanner near the missing area without considering the time cost, operational smoothness, or whether the scanning posture upon reaching the target point is the optimal observation pose for the area's features. This blindness not only reduces replacement scanning efficiency but may also introduce low-quality data due to inappropriate scanning angles or distances, damaging the accuracy of the final model. Furthermore, while existing software systems can provide preliminary visualization of point cloud coverage (such as pseudo-color rendering), this analysis is superficial at the topological level and cannot intelligently distinguish between actual missed scans caused by complex internal structures of the model (such as deep holes and grooves) and external boundaries of the model that have not yet been covered by the scanning process. This topological ambiguity often leads to invalid or incorrect guidance from the system to the operator, interfering with the normal scanning process.

[0004] CN109813219B discloses a method and system for collecting and processing information on the detection, identification and reinforcement of existing structures. This scheme mainly uses real-time rendering of point clouds and overlay with VR scenes to reduce multiple on-site re-measurements and surveys, while reducing model reconstruction work, optimizing reinforcement design work, making construction technology and organization more reasonable, and thus reducing rework. However, it does not consider the smoothness of the scanning path and the optimality of the scanning pose.

[0005] CN104137030A discloses a method for measuring three-dimensional samples using a measuring device incorporating a laser scanning microscope, as well as the measuring device itself. This method primarily utilizes a three-dimensional virtual reality device to generate a three-dimensional virtual space for the measurement space, allowing operations to be selected within this virtual space. It also provides real-time one-way or two-way connections between the measurement space and the virtual space, enabling operations selected in the virtual space to be performed in the measurement space, and data measured in the measurement space to be displayed in the virtual space. While it mentions the combination of three-dimensional scanning and virtual reality technology, its primary purpose is to achieve virtual reality interaction under microscopic measurement and does not involve guiding the scanning path for large objects. Summary of the Invention

[0006] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0007] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a scan path planning and missed scan compensation method based on spatial tracking to solve the problems mentioned in the background art.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a scanning path planning and missed scan compensation method based on spatial tracking, comprising:

[0009] The pose data of the spatial tracker fixed to the handheld 3D scanner is acquired in real time, and the 3D point cloud data output by the 3D scanner is collected simultaneously.

[0010] The coordinate transformation is performed on the three-dimensional point cloud data and the pose data, and a comprehensive quality weight is calculated for each three-dimensional point in the transformed point cloud data set according to a preset quality assessment model.

[0011] Based on the point cloud data set with the comprehensive quality weight, a three-dimensional coverage map is constructed in real time to identify missed areas.

[0012] For the identified missed scan areas, the optimal rescanning path is dynamically planned by constructing a hybrid cost function based on the current pose of the scanner.

[0013] The optimal re-scanning path is visualized and used to guide the operator in performing the re-scanning.

[0014] As a preferred embodiment of the spatial tracking-based scanning path planning and missed scan compensation method of the present invention, the preset quality assessment model includes at least one or more of the following weight combinations:

[0015] Angle weight determined by the angle between the scanner's line of sight and the local normal direction of the target surface;

[0016] Distance weights are determined by the distance between the scanner and the target surface;

[0017] The tracker confidence weight is determined by the confidence level of the pose data output by the spatial tracker;

[0018] Local density weights are determined by the local point cloud density of a 3D point within its original point cloud frame.

[0019] As a preferred embodiment of the spatial tracking-based scanning path planning and missed scan compensation method of the present invention, the method includes: constructing a three-dimensional coverage map to identify missed scan areas, comprising:

[0020] The three-dimensional coverage map is constructed using a hierarchical spatial data structure to distinguish between regions located inside the workpiece model and regions located at the outer boundary of the workpiece model, and the missed areas are identified by a density-based spatial clustering algorithm.

[0021] As a preferred embodiment of the scanning path planning and missed scan compensation method based on spatial tracking described in this invention, the dynamic planning of the optimal scan compensation path includes employing a path planning algorithm based on random sampling and performing priority sampling within the missed scan area.

[0022] As a preferred embodiment of the scanning path planning and missed scan compensation method based on spatial tracking described in this invention, the hybrid cost function is obtained from the operator's ergonomic cost and the scanning task performance cost.

[0023] As a preferred embodiment of the scanning path planning and missed scan compensation method based on spatial tracking described in this invention, the ergonomic cost of the operator is obtained by the second-order finite difference method.

[0024] As a preferred embodiment of the scanning path planning and missed scan compensation method based on spatial tracking described in this invention, the scanning task performance cost is obtained by comprehensively considering visibility cost and field of view center cost.

[0025] As a preferred embodiment of the spatial tracking-based scanning path planning and missed scan compensation method described in this invention, the method includes: guiding the operator to perform supplementary scanning in a visual manner, comprising:

[0026] The optimal scan path is rendered as a three-dimensional virtual path band.

[0027] As a preferred embodiment of the spatial tracking-based scanning path planning and missed scan compensation method described in this invention, it further includes:

[0028] On the optimal scan path, at a predictive position in front of the operator's current position, a holographic scanner model is rendered to indicate the target pose.

[0029] As a preferred embodiment of the spatial tracking-based scanning path planning and missed scan compensation method of the present invention, the method further includes an initialization calibration step before execution, which includes:

[0030] The fixed transformation relationship between the coordinate system of the 3D scanner and the coordinate system of the spatial tracker is calculated by using the hand-eye calibration method.

[0031] The fixed transformation relationship between the coordinate system of the virtual reality device and the coordinate system of the tracking marker points fixed on it is calculated using the head-mounted display device calibration method.

[0032] Compared with existing technologies, the beneficial effects of the invention are:

[0033] 1. By constructing a 3D coverage map in real time, it is possible to identify the real missed areas caused by geometric occlusion or improper operation in an instant and objective manner, avoiding secondary re-operation and fundamentally solving the problem of information feedback lag, thereby improving the success rate and data integrity of a single scanning operation.

[0034] 2. By constructing a hybrid cost function that combines scanning task efficiency and operator ergonomics, the optimal re-scanning path is dynamically planned, ensuring that the entire re-scanning process not only has the shortest path and is smooth and labor-saving, but also that the scanning posture after reaching the target position is optimal, thus guaranteeing the quality of the re-scanning data and reducing the dependence on the operator's skills.

[0035] 3. By using virtual reality equipment, the planned 3D path and other guiding information are directly superimposed on the operator's field of vision, realizing an immersive guidance that is exactly what you see. The operator no longer needs to frequently switch their gaze between the physical workpiece and the external display screen, which greatly reduces the operator's cognitive load, further shortens the operation time, and improves the smoothness and efficiency of the overall scanning process. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0037] Figure 1 This is a flowchart illustrating the overall process of a spatial tracking-based scanning path planning and missed scan compensation method according to an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of missed scan identification under a three-dimensional coverage map, based on a spatial tracking-based scanning path planning and missed scan compensation method according to an embodiment of the present invention.

[0039] Figure 3 This is a gesture prompt diagram for the scan path supplementation method based on spatial tracking, as described in one embodiment of the present invention. Detailed Implementation

[0040] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0041] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0042] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0043] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0044] In the description of the present invention, it should be noted that the terms "upper, lower, inner, and outer" and other references to orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first, second, or third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0045] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0046] Example 1

[0047] Reference Figures 1 to 3This is the first embodiment of the present invention, which provides a scanning path planning and missed scan compensation method based on spatial tracking, including:

[0048] S1. Real-time acquisition of pose data from the spatial tracker fixed to the handheld 3D scanner, and synchronous acquisition of 3D point cloud data output by the 3D scanner.

[0049] It should be noted that before the scanning operation begins, the rigid body transformation calibration between coordinate systems needs to be completed to ensure the accuracy of data fusion;

[0050] Furthermore, the fixed transformation relationship between the coordinate system of the 3D scanner and the coordinate system of the spatial tracker is calculated using the hand-eye calibration method;

[0051] Specifically, a calibration board for the hand-eye calibration method is created. This calibration board uses a 10×10 checkerboard pattern (each square has a side length of 20mm, and it uses an aluminum alloy base with a high-contrast coating). It is fixed in the world coordinate system W, and the pose is represented as follows. ;

[0052] It should be noted that the pose in the present invention includes a translation part and a rotation part, which are respectively composed of a three-dimensional coordinate vector and a unit quaternion;

[0053] Specifically, the operator uses a handheld scanner to scan the calibration plate from 12 different angles (position interval > 0.5m, rotation angle > 30°), with each scan lasting 1-2 seconds. During the scanning process, the position and pose of the calibration plate in the scanner coordinate system S are recorded simultaneously. The pose of the space tracker in the world coordinate system W The position of The corner positions and poses of a chessboard can be detected by calling the `findChessboardCorners` function in OpenCV. It is obtained in real time by a space tracking system, which also includes a space tracker coordinate system T;

[0054] Furthermore, given the above conditions, let the pose of the calibration plate in the world coordinate system W be... (Keep it constant), from two different scan processes i and j, the following transformation chain is obtained:

[0055]

[0056] in, This assumes a fixed relationship of change. By simultaneously solving the two formulas above, and eliminating... ,get:

[0057]

[0058] make Then we obtain the hand-eye calibration equation. ;

[0059] Specifically, at least three sets of non-coplanar pose pairs are collected (i.e. and The optimal X for solving the hand-eye calibration equation is obtained by using the quaternion method and stored. This optimal X is... ;

[0060] Furthermore, using the head-mounted display device calibration method, the fixed transformation relationship between the coordinate system of the virtual reality device and the coordinate system of the tracking marker points fixed on it is calculated;

[0061] Specifically, using the same calibration board as the hand-eye calibration method described above, the head-mounted display device identifies the checkerboard pattern on the calibration board through its built-in camera. During the scanning process, it simultaneously records the pose in the coordinate system of the tracking markers and the pose in the head-mounted display device's own coordinate system. By establishing a transformation chain, the head-mounted display device calibration equation is obtained, and X is solved and stored. This yields a fixed transformation relationship between the virtual reality device's coordinate system and the coordinate system of the tracking markers fixed on it. The specific processing steps are the same as those in the hand-eye calibration method described above, and will not be repeated here.

[0062] In addition, the Network Time Protocol (NTP) or the more precise Time Protocol (PTP) is used to synchronize the time between the host and the control host in the scanner in the spatial tracking system, ensuring that the error of their timestamps is controlled within milliseconds.

[0063] Specifically, after achieving time synchronization, the spatial tracking system outputs the pose of the spatial tracker at a high frequency (120Hz) to obtain pose data. The scanner outputs point cloud frames at a low frequency (15Hz). Approximately 10 per frame 5 There are points, among which Represented as a pose timestamp. Represented as a point cloud timestamp, Represented as a set of point clouds in the scanner coordinate system S;

[0064] S2. Perform coordinate transformation on the 3D point cloud data and pose data, and calculate a comprehensive quality weight for each 3D point in the transformed point cloud data set according to the preset quality assessment model.

[0065] It should be noted that for each output point cloud frame, its timestamp Normally not associated with any pose timestamp Completely overlapping, among which Therefore, it is necessary to find the timestamp in the pose data that corresponds to the point cloud. The two closest poses before and after and ;

[0066] Furthermore, for coordinate transformation, spherical linear interpolation (SLERP) and linear interpolation (Lerp) are used to interpolate the rotation part (i.e., the rotation relationship between the object's own coordinate system and the world coordinate system) and the translation part (the specific coordinates (x, y, z) of the object in space) respectively.

[0067] Specifically, spherical linear interpolation is expressed by the formula:

[0068]

[0069] in, Indicated as in The rotational part of the spatial tracker pose with timestamps is a quaternion, which is represented as follows: (satisfy Similarly, Indicated as in The rotational portion of the spatial tracker pose with timestamps; Represented as interpolation coefficients, ranging from [0,1], used for timestamps. In timestamp range The relative position in the data reflects the interpolation ratio of the point cloud timestamp relative to the pose timestamp; The result is a spherical linear interpolation, representing the timestamp. Rotational quaternions;

[0070] Specifically, linear interpolation can be expressed by the following formula:

[0071]

[0072] in, and With the above and Similarly, represent respectively in timestamp and The translation component of the spatial tracker pose is a translation vector, which is represented as: Through pose The translation components are directly extracted. and The same applies to the translation vector; This is the result of linear interpolation, representing the timestamp. The translation vector;

[0073] Specifically, the above Convert to a rotation matrix and with Combined, update pose to ,in Represented as a rotation matrix for transformation;

[0074] Furthermore, Transforming to the world coordinate system, we get:

[0075]

[0076] in, Expressed as The representation in the world coordinate system;

[0077] It needs to be explained that the reason for using different interpolation methods is that the translation part is usually linear and Euclidean, while the rotation part is usually non-linear and spherical. If linear interpolation is used directly for the rotation part, the unit length will be lost because the unit quaternion usually represents pure rotation. If two unit quaternions are linearly interpolated, the intermediate result will contain not only the unit quaternion but also scaling. This will cause the rendered holographic scanner model to be enlarged or shrunk during the operator guidance process, making the guidance process look very unnatural and causing the operator to easily become fatigued.

[0078] Furthermore, to differentiate between good and bad scanning quality (e.g., point cloud accuracy is lower with grazing angles), each transformed 3D point cloud... An additional quality weight is added, and a comprehensive quality weight is calculated using a preset quality assessment model. This model is preferably a weighted combination of the following four weights:

[0079] (1) Considering the angle weight, it is obtained by the angle between the scanner's line of sight and the local normal direction of the target surface;

[0080] Specifically, through the Principal component analysis (PCA) is performed on the points and their neighbors to estimate the local normal vectors of the surface in real time. Then, the distance from the optical center of the scanner to the nearest point is calculated. The line-of-sight vector is given. Furthermore, the angle weight is defined as the cosine function of the angle between the local normal vector and the line-of-sight vector, resulting in:

[0081]

[0082] in, Represented as angle weights, This is represented as the angle between the line-of-sight vectors. Represented as a local normal vector, Represented as a line-of-sight vector, , , Indicates the position of the scanner's optical center in the world coordinate system;

[0083] Specifically, real-time estimation is achieved by creating a NormalEstimation object in the PLC library and calling computeFromNormalVectorField;

[0084] (2) Considering distance weighting, it is obtained from the distance between the scanner and the target surface;

[0085] Specifically, the scanning distance is based on the line-of-sight vector and takes into account the effective scanning distance of the target surface, resulting in:

[0086]

[0087] in, Represented as distance weights, Represented as scan distance, ;

[0088] (3) Consider the confidence weight of the spatial tracker, which is obtained from the confidence of the pose data output by the spatial tracker;

[0089] Specifically, since spatial tracking systems typically include a confidence level when outputting spatial tracker pose data to reflect the reliability of the current pose data, this confidence level can be directly used as the confidence weight of the spatial tracker. ,in Represented as the confidence weight of the spatial tracker. This represents the confidence level of the pose data of the space tracker.

[0090] (4) Considering the local density weight, it is obtained from the local point cloud density of the three-dimensional point in its original point cloud frame;

[0091] Specifically, within the original point cloud frame, the point cloud density in its surrounding neighborhood is considered. Higher point cloud density indicates more reliable 3D reconstruction quality. This is calculated using the k-nearest neighbor density. get:

[0092]

[0093] in, Represented as local density weights, Expressed as The nearest neighbor; Represented as the set of nearest neighbors; Represented as the number of nearest neighbors;

[0094] Furthermore, the overall quality weight is a combination of the weights in (1) to (4), and the output is generated through a preset quality assessment model;

[0095] It should be noted that traditional methods for determining whether an area has been scanned typically check whether point cloud data has been collected in the corresponding 3D space. However, the fatal flaw of this method is its inability to distinguish between effective coverage and false coverage. Effective coverage refers to the operator scanning the area at the correct angle, optimal distance, and stable posture, acquiring a large number of accurate, low-noise points. False coverage, on the other hand, refers to situations where the operator unintentionally "brushes" the area with the edge of the scanner's line of sight during movement, or scans at a very large grazing angle (nearly parallel to the surface). While a few points may be acquired, these points have extremely large positional errors and high noise levels, making them unsuitable for reverse mapping. Data that is useless for modeling or dimensional analysis is what is known as junk data. Traditional methods would mark both effective and false coverage as scanned, thus considering the task complete. However, in reality, areas with false coverage need to be rescanned. The solution of this invention considers comprehensive quality weights and assigns a quality score to each acquired 3D point cloud. This score represents the key factors affecting data quality. In this way, what is accumulated is not just the number of 3D point clouds, but the total score of point cloud data quality. Therefore, for an area covered by junk data, its accumulated data quality score will be very low, and it can be accurately identified as a missed scan area that needs to be compensated.

[0096] S3. Based on the point cloud data set with comprehensive quality weights, a 3D coverage map is constructed in real time to identify missed areas;

[0097] Furthermore, based on weighted point cloud assemblies, a hierarchical spatial data structure is used to construct a 3D coverage map, and the missed scan areas requiring compensation are accurately identified from it; (Reference) Figure 2 In this context, gray units represent detected missed scan areas, and grid lines represent the division of the 3D coverage map.

[0098] Specifically, the hierarchical spatial data structure can use an octree as the spatial partitioning data structure because it can efficiently represent sparse three-dimensional environments and save a lot of memory compared to a fixed-size voxel grid. In this case, each leaf node (Voxel) of the octree stores an accumulated coverage quality score, and a preloaded workpiece CAD model (STL format) is used to distinguish between the regions where the scan points are located inside the workpiece and the regions located on the outer boundary of the workpiece.

[0099] Specifically, the cumulative coverage quality score is obtained by weighted summation of the weighted point cloud counts;

[0100] It should be noted that for the multiple identified missed areas that need compensation, the coverage quality score setting is mainly used to determine which area to fill in first and how to plan the filling in path.

[0101] Specifically, for the determination of missed scans, a coverage quality threshold needs to be set. This threshold is objectively set based on metrological calibration (i.e., preparing a metrological standard, fixing the scanner on a bracket, scanning the standard at the optimal working distance and a vertical angle close to 90 degrees specified in the scanner specifications, and extracting the peak value during the scanning process to obtain the theoretical maximum coverage quality threshold). By traversing all leaf nodes (Voxels) in the 3D coverage map, if the coverage quality score of a leaf node is lower than the set coverage quality threshold, the leaf node is marked as having poor quality, and its center coordinates are added to a temporary candidate set of missed scan points. For each point in the candidate set of missed scan points, the following determination logic is executed:

[0102] From this point, emit virtual rays along multiple random directions (e.g., 12 directions uniformly sampled on a unit sphere). Calculate the number of intersections between each virtual ray and the preloaded workpiece CAD model. If the ray intersects the model at least once in most directions, then the point can be determined to be "surrounded" by the model, i.e., located inside it; otherwise, the point is determined to be located outside it.

[0103] Furthermore, regarding the coverage quality score, when a new weighted point cloud set arrives, iterate through each point cloud and its weight, then insert the point cloud set into the octree, locate its corresponding leaf node, and update the coverage quality score of that leaf node:

[0104]

[0105] Where CQS represents the coverage quality score, and V represents the leaf node. The weights are represented as the weights of the point cloud set.

[0106] Specifically, candidate missed points located inside will be identified as real missed points and added to a new set of real missed points.

[0107] Furthermore, the density-based spatial clustering algorithm DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is used to identify discrete, spatially adjacent real missed scan points and aggregate them into meaningful and continuous missed scan regions. The advantage of choosing DBSCAN is that it does not require pre-specifying the number of clusters, can effectively identify clusters of arbitrary shapes, and can identify sparse discrete points as noise, which is very consistent with the irregular shape of the missed scan region.

[0108] Specifically, DBSCAN's neighborhood radius defines the distance between adjacent points, and its value needs to be set according to the size of the leaf nodes of the octagon. For example, it can be set to 1.5 times the diagonal length of the leaf node. As for DBSCAN's minimum number of points, it defines the minimum number of points required to form a region. A small value (such as between 3 and 5) can be set to ensure that small, independent missed scan regions can also be successfully identified as a cluster.

[0109] S4. For the identified missed scan areas, combine the scanner's current pose to construct a hybrid cost function to dynamically plan the optimal supplementary scan path;

[0110] Specifically, after identifying one or more missed scan regions, the urgency of each missed scan region is calculated, which is the ratio of the number of missed scan points in the missed scan region to the Euclidean distance from the scanner's current pose to the cluster center of the missed scan region. The urgency of each missed scan region is then sorted in descending order (from high to low), with priority given to repairing large and close missed scan regions.

[0111] It should be explained that after determining the priority missed scan areas, considering the possibility that multiple missed scan areas may have the same priority, it is necessary to find the optimal rescanning path. In traditional path planning, only the shortest or fastest path is usually considered. However, for handheld scanning tasks involving human-machine collaboration, this path planning is often not applicable and requires a high level of operator skill, which not only increases operator fatigue but also reduces rescanning efficiency. The solution of this invention aims to find a rescanning path that can reduce operator fatigue while maintaining scanning efficiency by constructing a hybrid cost function.

[0112] Specifically, the hybrid cost function is derived from the operator's ergonomic cost and the scanning task performance cost. The dynamic programming of the path, based on the sorted missed scan areas, uses the RRT* algorithm to generate a sequence of discrete poses, also known as a waypoint set. The first waypoint in the sequence is the scanner's current pose, and the last waypoint is the target pose. Each waypoint contains a position vector and an attitude quaternion. Assuming the time interval between waypoints is a constant representing the expected speed for guiding the operator's movement, this constant can ultimately be incorporated into the weighting coefficients and therefore can be omitted in the operator's ergonomic cost. The goal of the operator's ergonomic cost is to evaluate the smoothness of the new path segment formed when a new waypoint is connected to an existing waypoint in the tree. Furthermore, the parent node of the waypoint is needed to calculate the rate of change (such as acceleration or jerk).

[0113] Furthermore, the second-order finite difference method is used to approximate the acceleration vector of the path at waypoints. The square of the magnitude of this vector can be regarded as an excellent cost measure. A large acceleration magnitude means a drastic velocity change; however, for a smooth path, the velocity vector change and angular velocity change are slow. Therefore, when a new waypoint connects to an existing waypoint in the tree, it is only necessary to calculate the cost of the "turning" and "attitude twisting" generated at the existing waypoint in the tree.

[0114] Specifically, the cost of "turning" can be expressed by the formula:

[0115]

[0116] in, Indicates a new waypoint. This indicates that there are already waypoints in the tree. This is represented as the parent node of the waypoint. Represented as a position vector; It is expressed as the square of the magnitude of the acceleration vector; Represented as a linear motion smoothness cost function;

[0117] It should be noted that if the three points (the new waypoint, the existing waypoint in the tree, and the parent node of the waypoint) are on a straight line and the distance between them is equal, then the cost of "turning" is 0, which means the lowest cost.

[0118] Specifically, the cost of "attitude twisting" is expressed by the formula:

[0119] in, Represented as attitude quaternions;

[0120] It should be noted that if the attitude changes at a constant angular velocity (e.g., rotating at a constant speed around the same axis), the cost of "attitude torsion" is close to 0, which means the lowest cost.

[0121] Specifically, by adding the "turning" cost and the "posture twisting" cost mentioned above, we can obtain the ergonomic cost of operation.

[0122] Furthermore, if a certain pose on the path is completely obstructed by the workpiece itself, making it impossible to see the target missed area (i.e., the path collides with the obstacle (workpiece model)), then this pose is ineffective for the rescanning task. Therefore, the visibility cost of scanning needs to be considered. The calculation method for visibility cost is similar to that of the angle weight and distance weight mentioned above, aiming to ensure that the scanner maintains the optimal working distance from the target surface, while keeping the scanner's optical axis as perpendicular to the target surface as possible. Based on this, we encourage placing the target point to be scanned (missed point) in the center of the scanner's field of view, forming the field-of-view center cost, which is calculated as follows:

[0123] The target point is projected onto the imaging plane of the scanner, and the Euclidean distance between the target point and the center of the imaging plane is calculated. The closer the target point is to the center of the imaging plane, the smaller the distance will be, and the lower the final cost will be.

[0124] Specifically, the scanning task performance cost can be obtained by weighted summing of the aforementioned visibility cost and field of view center cost.

[0125] S5. Guide the operator to perform the supplementary cleaning in a visual way using virtual reality equipment to present the optimal supplementary cleaning path;

[0126] Furthermore, the planned optimal re-scanning path is sent to the virtual reality device in real time. The virtual reality device uses the aforementioned head device calibration method to take the fixed transformation relationship between the virtual reality device coordinate system and the coordinate system of the tracking marker points fixed on it as input, and guides the operator to perform re-scanning in a precise and visual way.

[0127] Specifically, during this process, the optimal scan path is rendered as an intuitive 3D virtual path strip that floats in the operator's field of view and extends from the current scanner position to the target area.

[0128] Specifically, for the rendered 3D virtual path strip, its visual attributes will change dynamically according to the operator's operation process. For example, when the operator's scanner moves precisely along the path strip, the path strip is displayed in green, and when an offset occurs, the color changes to yellow or red, providing the operator with instant correction feedback. In addition, the texture on the path strip can flow, and the speed of its flow indicates the recommended scanning speed.

[0129] Additionally, on the path, at a predictive location ahead of the operator's current position (e.g., 0.5 seconds in advance), and indicated by a gesture-controlled arrow (see reference). Figure 3 It can render a semi-transparent holographic scanner model that is designed to precisely indicate the target pose (including position and orientation) that the operator needs to reach, suggesting what posture the operator should adopt after arriving at the scanning area;

[0130] It should be noted that by using the rendering path and the holographic scanner model, the operator can be immersed in the process, thereby improving the smoothness and efficiency of the overall scanning process.

[0131] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0132] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0133] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0135] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0136] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A scanning path planning and missed scan compensation method based on spatial tracking, characterized in that, include: The pose data of the spatial tracker fixed to the handheld 3D scanner is acquired in real time, and the 3D point cloud data output by the 3D scanner is collected simultaneously. The coordinate transformation is performed on the three-dimensional point cloud data and the pose data, and a comprehensive quality weight is calculated for each three-dimensional point in the transformed point cloud data set according to a preset quality assessment model. Based on the point cloud data set with the comprehensive quality weight, a three-dimensional coverage map is constructed in real time to identify missed areas. For the identified missed scan areas, the optimal rescanning path is dynamically planned by constructing a hybrid cost function based on the current pose of the scanner. The optimal re-scanning path is visualized and used to guide the operator in performing the re-scanning.

2. The scanning path planning and missed scan compensation method based on spatial tracking as described in claim 1, characterized in that, The preset quality assessment model includes at least one or more of the following weight combinations: Angle weight determined by the angle between the scanner's line of sight and the local normal direction of the target surface; Distance weights are determined by the distance between the scanner and the target surface; The tracker confidence weight is determined by the confidence level of the pose data output by the spatial tracker; Local density weights are determined by the local point cloud density of a 3D point within its original point cloud frame.

3. The scanning path planning and missed scan compensation method based on spatial tracking as described in claim 1, characterized in that, Construct a 3D coverage map to identify missed scan areas, including: The three-dimensional coverage map is constructed using a hierarchical spatial data structure to distinguish between regions located inside the workpiece model and regions located at the outer boundary of the workpiece model, and the missed areas are identified by a density-based spatial clustering algorithm.

4. The scanning path planning and missed scan compensation method based on spatial tracking as described in claim 1, characterized in that, The dynamic programming optimal supplementary scanning path includes employing a path planning algorithm based on random sampling and prioritizing sampling within the missed scanning area.

5. The scanning path planning and missed scan compensation method based on spatial tracking as described in claim 1, characterized in that, The hybrid cost function is derived from the operator's ergonomic cost and the scanning task's performance cost.

6. The scanning path planning and missed scan compensation method based on spatial tracking as described in claim 5, characterized in that, The ergonomic cost of the operation is obtained using the second-order finite difference method.

7. The scanning path planning and missed scan compensation method based on spatial tracking as described in claim 5, characterized in that, The performance cost of the scanning task is obtained by comprehensively considering the visibility cost and the center of the field of view cost.

8. The scanning path planning and missed scan compensation method based on spatial tracking as described in claim 1, characterized in that, Visually guide operators to perform supplementary scanning, including: The optimal scan path is rendered as a three-dimensional virtual path band.

9. The scanning path planning and missed scan compensation method based on spatial tracking as described in claim 8, characterized in that, Also includes: On the optimal scan path, at a predictive position in front of the operator's current position, a holographic scanner model is rendered to indicate the target pose.

10. The scanning path planning and missed scan compensation method based on spatial tracking as described in claim 1, characterized in that, The method also includes an initialization calibration step before execution, which includes: The fixed transformation relationship between the coordinate system of the 3D scanner and the coordinate system of the spatial tracker is calculated using the hand-eye calibration method; the fixed transformation relationship between the coordinate system of the virtual reality device and the coordinate system of the tracking marker points fixed on it is calculated using the head-mounted display device calibration method.

Images