Large-range weak feature scene measurement system based on multi-sensor fusion and structured light scanning

Abstract

The invention discloses a wide-range weak feature scene measurement system based on multi-sensor fusion and structured light scanning, and relates to the technical field of wide-range scene scanning, in the system, an extended Kalman filtering algorithm is adopted, fusion is carried out based on positioning data provided by an inertial measurement module and a positioning sensing module, and a wide-range weak feature scene is obtained. And the distance measurement information provided by the laser distance measurement module is combined to correct the fused positioning data, so that the precision and robustness of subsequent point cloud registration are ensured. Then, according to position information and attitude information of the binocular structured light scanning camera during scene scanning, calculating a relative rotation matrix and a relative displacement matrix of each frame of scanning point cloud data relative to the first frame of scanning point cloud data, and performing coarse registration; in the coarse registration process, all frames of point clouds can be converted into a unified coordinate system by directly utilizing global pose information, and the efficiency of coarse registration is greatly improved. And finally, extracting laser marking point clouds projected by the non-contact laser marking device in each frame of scanning point cloud data, and performing iterative precise registration on each frame of scanning point cloud data by using color features and geometric distribution rules of the extracted laser marking point clouds. In conclusion, high-precision, high-efficiency and high-robustness point cloud registration in a large-scale weak feature scene is realized, and the method can be used for realizing high-precision non-contact measurement.

Description

Technical Field

[0001] The present application relates to the technical field of large-scale scene scanning, and in particular to a large-scale weak-feature scene measurement system based on multi-sensor fusion and structured light scanning. Background Art

[0002] In many fields such as modern industrial manufacturing, aerospace, and construction engineering, the demand for high-precision three-dimensional dimension measurement, quality inspection, and reverse engineering of large objects or scenes is growing rapidly. Non-contact three-dimensional scanning technology, especially structured light scanning technology, has been widely used in the measurement of small and medium-sized objects due to its high measurement accuracy, high efficiency, and relatively low cost. However, when the measurement object is expanded to a large range of scenes (such as aircraft skin, automobile body, large casting and forging blanks, etc.), the field of view of a single scan is limited. Multiple scans must be performed and the collected multi-frame point cloud data must be accurately aligned (stitched) into a unified coordinate system to reconstruct a complete three-dimensional model of the object. At this time, structured light scanning measurement technology faces major challenges.

[0003] In recent years, some studies have attempted to integrate inertial measurement units (IMUs) or simultaneous localization and mapping (SLAM) technologies into structured light scanning systems to assist in the registration process. Rough registration is achieved by fusing sensor data to estimate the motion trajectory of the scanning device. However, this approach primarily addresses initial alignment or navigation issues. Pure IMU integration is prone to cumulative drift errors, making it difficult to meet the requirements of high-precision measurement. SLAM methods based on vision or lidar are also subject to the risk of positioning drift or failure in environments with weaker features, and generally cannot achieve the precise registration accuracy required for industrial measurement. Summary of the Invention

[0004] The purpose of this application is to provide a large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning, which can achieve high-precision, high-efficiency and high-robustness point cloud registration in large-scale weak feature scenes.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning, comprising:

[0007] The multi-sensor joint calibration module is used to collaboratively calibrate the binocular structured light scanning camera, inertial measurement module, positioning sensor module and laser ranging module; the binocular structured light scanning camera is used to scan the target scene; the target scene is a large-scale scene with weak features to be scanned; the inertial measurement module, positioning sensor module and laser ranging module are respectively fixed in position with the binocular structured light scanning camera to achieve three-dimensional spatial positioning of the binocular structured light scanning camera.

[0008] The multi-sensor data fusion module is used to fuse the positioning data of the inertial measurement module and the positioning sensor module using the extended Kalman filter algorithm, and to correct the fused positioning data in combination with the ranging information of the laser ranging module to obtain the three-dimensional spatial coordinates when the binocular structured light scanning camera scans the scene.

[0009] The scanning point cloud data coarse registration module is used to calculate the relative rotation matrix and relative displacement matrix of each frame of scanning point cloud data relative to the first frame of scanning point cloud data based on the three-dimensional spatial coordinates and Euler angles when the binocular structured light scanning camera scans the scene, and align the coordinate system of each frame of scanning point cloud data to the coordinate system of the first frame of scanning point cloud data; the multiple frames of scanning point cloud data all include laser marked point clouds remotely projected by a non-contact laser marking device.

[0010] The scanning point cloud data fine registration module is used to extract the laser marker point cloud from each frame of scanning point cloud data based on the coarse registration, and use the color characteristics and geometric distribution patterns of the laser marker point cloud extracted from different frames of scanning point cloud data to iteratively fine-register each frame of scanning point cloud data. After the fine registration, the scanning point cloud data can be used for non-contact measurement of the target scene.

[0011] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0012] The present application provides a large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning. In this system, after the coordinated calibration of the binocular structured light scanning camera, inertial measurement module, positioning sensor module and laser ranging module, we use the extended Kalman filter algorithm to fuse the positioning data provided by the inertial measurement module and the positioning sensor module, and correct the fused positioning data in combination with the ranging information provided by the laser ranging module, thereby ensuring the accuracy and robustness of the subsequent point cloud registration. Subsequently, based on the position information and posture information of the binocular structured light scanning camera during scene scanning, the relative rotation matrix and relative displacement matrix of each frame of scanning point cloud data relative to the first frame of scanning point cloud data are calculated and coarse registration is performed. The coarse registration process directly uses the global pose information to transform each frame of point cloud into a unified coordinate system, greatly improving the efficiency of coarse registration. Finally, by extracting the laser marking point cloud projected by the non-contact laser marking device in each frame of scanning point cloud data, and using the color characteristics and geometric distribution laws of these extracted laser marking point clouds, each frame of scanning point cloud data is iteratively fine-tuned. In summary, this application achieves high-precision, high-efficiency, and high-robust point cloud registration in large-scale weak-feature scenes, and can be used to achieve high-precision non-contact measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 This is a schematic diagram of the functional modules of a large-scale weak-feature scene measurement system based on multi-sensor fusion and structured light scanning provided in one embodiment of the present application.

[0015] Figure 2 This is a schematic diagram of the installation of a positioning sensor module in a large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning provided in one embodiment of the present application.

[0016] Figure 3 A schematic structural diagram of a large-scale weak-feature scene scanning device based on multi-sensor fusion and structured light scanning provided in another embodiment of the present application.

[0017] Figure 4 A schematic structural diagram of a computer device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0019] In modern industrial manufacturing, aerospace, construction engineering and other fields, there is an increasing demand for high-precision three-dimensional dimension measurement, quality inspection and reverse engineering of large objects or scenes. Non-contact three-dimensional scanning technology, especially structured light scanning technology, has been widely used in the measurement of small and medium-sized objects due to its high measurement accuracy, efficiency and low cost. However, when the measurement object is expanded to a large range of scenes (such as aircraft skin, automobile body, large casting and forging blanks, etc.), structured light scanning measurement technology faces severe challenges. Due to the limited field of view of a single scan, multiple scans must be performed, and the obtained multi-frame point cloud data must be accurately aligned (stitched) to a unified coordinate system in order to reconstruct a complete three-dimensional model of the object. The existing point cloud registration technology is mainly divided into two categories:

[0020] Registration methods based on point cloud features: These methods rely on the geometric features (such as corners and edges) or texture information contained in the point cloud data itself. By extracting and matching these feature points or feature descriptions, the relative pose transformation between frames is calculated. However, in typical large-scale weak feature scenes such as aircraft skins, large smooth surfaces, and rough walls, the object surfaces lack sufficiently rich, stable, and easily distinguishable geometric, texture, and color features. This leads to difficulties in feature extraction, sparse matching point pairs, and high error rates, making such registration methods often inaccurate or even completely ineffective.

[0021] Registration method based on artificial markers: In order to solve the registration problem in weak feature scenes, the industry often adopts the method of sticking a large number of physical markers (such as circular coding points, checkerboard targets) on the surface of the object to be measured. By accurately identifying and positioning these markers, high-precision point cloud registration can be achieved. However, the disadvantages of this method are very obvious: (a) Low efficiency: Manually laying out a large number of markers on a large surface requires a lot of manpower and time; (b) Cumbersome operation: The layout of the marker points needs to be carefully planned, and additional equipment (such as a total station) may be required to measure the coordinates of the marker points; (c) There is a risk of surface damage: The sticking and removal of markers may cause scratches, contamination or damage to precision, coated or fragile surfaces; (d) Limited applicability: For high-temperature, oversized or complex-structured surfaces, laying out markers may be very difficult or impossible.

[0022] In recent years, although some studies have attempted to introduce inertial measurement units (IMUs) or simultaneous localization and mapping (SLAM) technology into scanning systems to assist with registration, achieving coarse registration by fusing sensor data to estimate the motion trajectory of the scanning device, this mainly solves the initial alignment or navigation problem. Pure IMU integration will produce cumulative drift errors, making it difficult to meet the requirements of high-precision measurement. SLAM methods based on vision or lidar also face the risk of positioning drift or failure in weak feature environments, and generally cannot achieve the fine registration accuracy required for industrial measurement.

[0023] Based on the core technical problems of the above-mentioned existing technologies in the application of large-scale weak-feature scene three-dimensional scanning, such as difficulty in registration, insufficient accuracy, low efficiency and possible surface damage, this application proposes a large-scale weak-feature scene measurement system based on multi-sensor fusion and structured light scanning, aiming to solve the problems existing in the above-mentioned existing technologies and achieve high-precision, high-efficiency and high-robustness point cloud registration in large-scale weak-feature scenes.

[0024] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0025] The embodiment of the present application provides a large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning. In an exemplary embodiment, Figure 1 Shown, including:

[0026] The multi-sensor joint calibration module M1 is used to collaboratively calibrate the binocular structured light scanning camera, inertial measurement module, positioning sensor module and laser ranging module; the binocular structured light scanning camera is used to scan the target scene; the target scene is a large-scale scene with weak features to be scanned; the inertial measurement module, positioning sensor module and laser ranging module are respectively fixed in position with the binocular structured light scanning camera to achieve three-dimensional spatial positioning of the binocular structured light scanning camera.

[0027] The multi-sensor data fusion module M2 is used to use the extended Kalman filter algorithm to fuse the positioning data of the inertial measurement module and the positioning sensor module, and to correct the fused positioning data in combination with the ranging information of the laser ranging module to obtain the three-dimensional spatial coordinates when the binocular structured light scanning camera performs scene scanning.

[0028] The scanning point cloud data coarse registration module M3 is used to calculate the relative rotation matrix and relative displacement matrix of each frame of scanning point cloud data relative to the first frame of scanning point cloud data based on the three-dimensional spatial coordinates and Euler angles when the binocular structured light scanning camera scans the scene, and align the coordinate system of each frame of scanning point cloud data to the coordinate system of the first frame of scanning point cloud data; the multiple frames of scanning point cloud data all include laser marked point clouds remotely projected by a non-contact laser marking device.

[0029] The scanning point cloud data fine registration module M4 is used to extract the laser marker point cloud from each frame of the scanning point cloud data based on the rough registration, and use the color characteristics and geometric distribution rules of the laser marker point cloud extracted from different frames of scanning point cloud data to iteratively fine-register each frame of the scanning point cloud data. The scanning point cloud data after fine registration can be used for non-contact measurement of the target scene.

[0030] Specifically in this embodiment, the multi-sensor joint calibration module M1 includes:

[0031] The binocular structured light scanning camera self-calibration unit M11 is used to calibrate the binocular structured light scanning camera using a vision system calibration technique based on perspective geometry, determining and correcting the camera's internal and external parameters. Vision system calibration involves determining and correcting the camera's internal and external parameters to enable it to accurately capture and measure the position, shape, and size of objects. In this embodiment, the binocular structured light scanning camera is calibrated using a vision system calibration technique based on perspective geometry.

[0032] The camera-inertial measurement module calibration unit M12 is used to collect several sets of scanning point cloud data and inertial measurement data, and solve the rotation matrix between the binocular structured light scanning camera and the inertial measurement module.

[0033] Because the scanned point cloud coordinates are based on the binocular camera's coordinate system, there's a rigid transformation between the actual installation location of the inertial measurement unit (IMU) and the origin of the camera coordinate system. Therefore, collaborative calibration of the camera and IMU is necessary to ensure that the attitude data obtained by the IMU can be directly applied to the camera coordinate system. Since the scanning device doesn't need to move quickly, the camera's rolling shutter effect can be ignored. Furthermore, because the IMU is internally self-calibrated, gyroscope bias calculations are unnecessary. Furthermore, since translation can be determined in advance through the installation relationship and is small, its impact is minimal compared to rotation, so only rotation can be estimated.

[0034] Under these constraints, the calibration problem is similar to the hand-eye calibration problem. A calibration board is used to help estimate the position and orientation of the camera in the 3D world. The rotation of the camera will cause a rotation in the original camera data D, as shown in the following equation:

[0035] D c =R c D.

[0036] Among them, the rotation matrix R c is the rotation matrix that describes the rotation change of the original camera data, D c Refers to the final data after rotation transformation. For IMU data collection:

[0037] D c =R cal D cm .

[0038]

[0039] Among them, R cal Is the data D rotated by IMU cm To the final data rotation transformation matrix, that is, the matrix of camera and IMU collaborative calibration that needs to be solved, R m is the rotation matrix read from the IMU. Combining the above formula, we can infer that R c and R m The relationship is shown as follows:

[0040]

[0041] Since the camera has been calibrated, R c and R m As the quantity is known, we only need to solve R cal The equation can be written as Ax = 0, and the value of x can be found by solving the null space of the homogeneous equation. In this embodiment, 300 sets of camera-IMU data are collected as input to the calibration algorithm, and the final camera-IMU calibration matrix R is solved. cal .

[0042] The positioning-inertial measurement module calibration unit M13 is used to use the IMU-UWB coordinate system unified calibration method based on the constraint path on the two-dimensional horizontal plane to solve the installation angle of the inertial measurement module relative to the positioning sensor module coordinate system.

[0043] In one embodiment, the positioning sensor module uses a combination of at least 4 UWB base stations and UWB positioning tags, such as Figure 2 The figure shows the installation diagram of the positioning sensor module. This system deploys six LD150 modules as base stations and fixes them in specific locations to form a base station array. At the same time, one LD150 module is mounted as a tag on a binocular structured light scanning camera. Figure 2The circles represent the locations of the base stations, and the triangles represent the locations of the positioning tags. With base station A as the origin of the world coordinate system, the world coordinates of the other five base stations can be determined. The positioning sensor module uses bilateral ranging to measure the distance from the positioning tag to each base station and then uses a trilateration algorithm to determine the positioning tag's coordinates.

[0044] When UWB positioning algorithms directly calculate the carrier's position, non-line-of-sight (NLOS) errors often occur due to environmental factors, resulting in outliers in the positioning results. To prevent the impact of NLOS errors on coarse registration, UWB-IMU fusion positioning is used to compensate for the original positioning values.

[0045] In order to achieve UWB-IMU fusion positioning, it is first necessary to solve the coordinate transformation problem between the IMU local coordinate system and the UWB base station global coordinate system. This embodiment adopts a unified calibration method of the UWB-IMU coordinate system based on a constraint path, which can accurately obtain the installation angle of the IMU relative to the UWB coordinate system. Due to the existence of gravitational acceleration, the calibration of the Z axis is relatively simple, so in order to simplify the problem, we focus on the calibration of the two-dimensional horizontal plane. Suppose the global coordinate system of the UWB base station is (x, y), the local coordinate system of the IMU is (x′, y′), and there is an unknown rotation angle θ between the two coordinate systems, then the coordinate transformation relationship can be expressed as:

[0046]

[0047] Among them, (a x ′,a y ′) is the acceleration measured in the IMU coordinate system, (a x ,a y ) is the corresponding UWB global coordinate system acceleration.

[0048] First, place the scanning device equipped with the IMU at the known starting point P0 (x0, y0) and move it along a straight line to the end point P1 (x1, y1). The motion vector V in the global coordinate system is:

[0049] V=(x1-x0,y1-y0).

[0050] The motion direction angle θ1 in the global coordinate system is:

[0051] θ1=arctan2(y1-y0,x1-x0).

[0052] Acceleration data a′ obtained by IMU x , a′ y The motion vector obtained by quadratic integration in the IMU coordinate system is shown as follows:

[0053] V′=(∫∫a′ x (t)dt 2 ,∫∫a′ y (t)dt 2 ).

[0054] The direction angle θ2 in the IMU coordinate system is:

[0055] θ2=arctan2(∫∫a′ y (t)dt 2 ,∫∫a′ x (t)dt 2 ).

[0056] The installation angle θ can be calculated as:

[0057] θ=θ1-θ2.

[0058] To improve calibration accuracy, maintain uniform linear motion during operation, avoid excessive shaking, and maintain sufficient rest time at the starting and ending points to allow for zero-velocity updates. Calibration should also be performed in an open area to avoid the influence of UWB signal multipath. Repetitive measurements, performed ten times and averaging the results, can further enhance the reliability of the calibration results. The resulting mounting angle θ is used in the UWB-IMU positioning data fusion process, effectively integrating inertial navigation and UWB positioning.

[0059] Specifically, the positioning-inertial measurement module calibration unit M13 is specifically used to: establish a positioning sensor module coordinate system and an inertial measurement module coordinate system on a two-dimensional horizontal plane respectively; move the binocular structured light scanning camera equipped with an inertial measurement module from a known starting point to a known end point in the positioning sensor module coordinate system, and determine the motion vector and motion direction angle of the binocular structured light scanning camera in the positioning sensor module coordinate system; determine the motion vector and motion direction angle of the binocular structured light scanning camera in the inertial measurement module coordinate system through quadratic integration of the acceleration data obtained by the inertial measurement module; and determine the installation angle of the inertial measurement module relative to the positioning sensor module coordinate system based on the motion direction angle of the binocular structured light scanning camera in different coordinate systems.

[0060] certainly, Figure 1 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different functions. Figure 1 One or at least two components of the system shown.

[0061] The inertial measurement module (IMU) is immune to non-line-of-sight interference and can assist in positioning when the UWB positioning tag is affected by NLOS errors. Considering the long-term accumulation of positioning errors, the IMU's positioning results can also be corrected with the help of UWB measurement data that is not affected by NLOS interference to ensure positioning accuracy. Since loose coupling relies on the IMU for positioning when the number of visible nodes in the UWB positioning system is insufficient, tight coupling can use these nodes to correct the IMU's positioning results. Therefore, in this embodiment, tight coupling is selected and the extended Kalman filter (EKF) is used to achieve positioning data fusion.

[0062] Since UWB base stations are limited by geometric distribution and multipath effects during deployment, the UWB positioning sensor module has poor accuracy on the Z axis. Therefore, when performing UWB-IMU positioning data fusion, only the two-dimensional horizontal plane is targeted, and the Z axis accuracy is subsequently compensated by the first laser ranging sensor.

[0063] The position and velocity of the scanning device on the x and y axes in the UWB coordinate system are obtained by integrating the accelerations as follows:

[0064]

[0065] Among them, P x,k is the x-axis position of the device in the UWB coordinate system at time k, P y,k is the y-axis position of the device in the UWB coordinate system at time k, V x,k is the speed of the device on the x-axis in the UWB coordinate system at time k, V y,k is the speed of the device on the y-axis in the UWB coordinate system at time k, a x,k is the acceleration of the device on the x-axis in the UWB coordinate system at time k, a y,k is the acceleration of the device on the x-axis in the UWB coordinate system at time k, T s is the sampling time interval.

[0066] Converting the above formula into matrix form, the state equation of the EKF system can be obtained as follows:

[0067]

[0068] Where η is the system noise, and its corresponding covariance is Q η , C imu-uwb is the rotation matrix from the IMU coordinate system to the UWB coordinate system. In the two-dimensional perspective, two of the six base stations are repeated. Here, the ranging equation expression of the four UWB base stations and the tag can be obtained as follows:

[0069]

[0070] In the above formula, σ i represents the ranging noise of the i-th UWB base station. Since this embodiment targets nonlinear systems, the traditional Kalman filter cannot achieve good results. Therefore, this embodiment uses an extended Kalman filter for data fusion. Since the ranging equation of the positioning sensor module is nonlinear, the Jacobian matrix is ​​used for linear approximation to obtain the following formula:

[0071]

[0072] Among them, H d P is the matrix obtained by linearly approximating the ranging equation of the positioning sensor module. x,k is the x-axis position of the binocular structured light scanning camera in the positioning sensor module coordinate system at time k, P y,k is the position of the binocular structured light scanning camera on the y-axis in the positioning sensor module coordinate system at time k, (x i ,y i ) is the two-dimensional coordinate of the i-th positioning base station, i = {1, 2, 3, 4}.

[0073] The expressions for the system equation prediction value, prediction covariance, and extended Kalman filter algorithm gain are calculated as follows:

[0074]

[0075] Among them, X k / k-1 is the predicted value of the system equation, P k / k-1 is the predicted covariance, the subscript k / k-1 represents the predicted value at time k based on the information at time k-1, the subscript k-1 represents the posterior estimate at time k-1, Kg k is the gain of the extended Kalman filter algorithm, Q η is the covariance corresponding to the system noise η, R d is the diagonal matrix composed of ranging variance, A and B are the state coefficient matrix and input control matrix corresponding to the extended Kalman filter algorithm, u k-1 is the system input variable, and the corresponding expression is as follows:

[0076]

[0077] Among them, T s is the sampling time interval, C imu-uwb is the rotation matrix from the inertial measurement module coordinate system to the positioning sensor module coordinate system, (a x ′,a y ′,a z ′) is the three-axis acceleration measured in the inertial measurement module coordinate system, and g is the acceleration due to gravity.

[0078] Based on the measurement value of the positioning sensor module, the update correction is performed to obtain the measurement residual, output estimation value and corresponding error estimation matrix, as shown in the following formula:

[0079]

[0080] Among them, e k is the measurement residual, X k / k is the output estimate, P k / k is the corresponding error estimation matrix, d k is the measurement value of the positioning sensor module at time k, d k-1 is the predicted value at time k based on the measurement value of the positioning sensor module at time k-1, and I is the unit matrix.

[0081] The measurement residual e obtained from the above formula is k The difference between the measurement value of the positioning sensor module and the predicted value of the inertial measurement unit integral operation can be used to determine whether the positioning sensor module has generated an NLOS error; the ranging residual judgment threshold is obtained by counting the normal value under normal line of sight and no interference; when the measurement residual e k If the range residual error exceeds the threshold, the positioning sensor module is considered to have generated an NLOS error, and the measurement value is discarded. The update frequency of the positioning fusion EKF is consistent with the IMU data update frequency, selected as 200 Hz. The UWB positioning data update frequency is approximately 10 Hz. During the 0.1-second gap in absolute positioning data, the IMU-derived positioning prediction will not have significant error accumulation.

[0082] Due to the geometric distribution of base stations during deployment and the multipath effect, the Z-axis positioning data error of the UWB positioning system is approximately ±30cm, which significantly affects alignment. Therefore, as an optional embodiment, the laser ranging module includes: a first laser ranging sensor and a second laser ranging sensor; the first laser ranging sensor is used to provide real-time compensation for the Z-axis height data used in the three-dimensional positioning of the positioning sensor module; the installation position of the first laser ranging sensor can be precisely adjusted using the attitude angle data provided by the inertial measurement module to ensure that the measurement direction of the first laser ranging sensor is strictly aligned with the vertical direction; the second laser ranging sensor is used to guide the operator to maintain the binocular structured light scanning camera at the optimal imaging distance for scanning operations through real-time distance feedback; the second laser ranging sensor is installed near the origin of the binocular structured light scanning camera coordinate system.

[0083] Specifically, the fused positioning data is corrected according to the following formula:

[0084] Z real =Z laser cos(φ)cos(θ)+d.

[0085] Among them, Z laser is the device height obtained by the first laser ranging sensor, d is the vertical distance from the positioning sensor module to the first laser ranging sensor, Z real is the corrected Z-axis height data, θ and φ are the pitch angle and roll angle obtained by the inertial measurement module, respectively.

[0086] During coarse registration, αβγ are used to represent the angles of rotation around the XYZ axes. According to the definition of Euler angles, the rotation matrix of the three rotations is as follows:

[0087]

[0088] First read the multi-frame point cloud data and store it in the collection Where N is the total number of frames of point cloud data in the set, and the coordinates and Euler angle data in the sensor data are stored in T i and O i =(α i ,β i ,γ i ). Convert the Euler angle corresponding to each frame of point cloud into the rotation matrix in the IMU coordinate system:

[0089] R i =R z (γ i )R y (β i )R x (α i ).

[0090] Referring to the content of multi-sensor joint calibration, since the point cloud data generated by the scanning camera is in the camera coordinate system, there is a rotation transformation R between the IMU and the camera. cal . For the rotation matrix R in the IMU coordinate system i It needs to be converted to the camera coordinate system:

[0091] R C,i =R cal R i .

[0092] Here, the rotation matrix and displacement vector of the first frame scanning point cloud data are assumed to be R C,1 and T1, the relative rotation matrix and relative displacement matrix of each frame of scanned point cloud data relative to the first frame of scanned point cloud data can be determined according to the following formulas:

[0093]

[0094] T relative,i =T i -T1.

[0095] Among them, Rrelative,i is the relative rotation matrix of the i-th frame scan point cloud data relative to the first frame scan point cloud data, T relative,i is the relative displacement vector of the i-th frame scan point cloud data relative to the first frame scan point cloud data, the superscript T is the transpose of the matrix, R C,i is the rotation matrix of the scanned point cloud data of the i-th frame, T i is the displacement matrix of the scanned point cloud data of the i-th frame.

[0096] The coordinate system of each frame of scanned point cloud data is aligned to the coordinate system of the first frame of scanned point cloud data according to the following formula:

[0097] P′={R relative,i P i +T relative,i |i=1,2,...,N}.

[0098] Among them, P i Scan point cloud data for the i-th frame, is the set of scanned point cloud data, N is the total number of frames of scanned point cloud data, and P′ is the set of all scanned point cloud data after coarse registration.

[0099] The scanning point cloud data fine registration module is specifically used to: on the basis of coarse registration, use the color region growing segmentation algorithm to extract the laser marking point cloud in each frame of scanning point cloud data; for the first frame scanning point cloud data and the second frame scanning point cloud data, use the first frame scanning point cloud data as the target overall point cloud B, and the laser marking point cloud extracted from the first frame scanning point cloud data as the target laser marking point cloud Q, use the second frame scanning point cloud data as the source overall point cloud A, and the laser marking point cloud extracted from the second frame scanning point cloud data as the source laser marking point cloud P; for the source overall point cloud A, source The laser marker point cloud P, the target overall point cloud B, and the target laser marker point cloud Q are iteratively solved using the least squares method to minimize the matching error of the source laser marker point cloud P and the target laser marker point cloud Q through an iterative nearest neighbor search algorithm, obtaining the optimal rotation matrix R and the optimal translation vector T. The optimal rotation matrix and the optimal translation vector are applied to the source overall point cloud A for coordinate transformation, so that the source overall point cloud A and the target overall point cloud B are precisely aligned. After completing the precise alignment of the first and second frames of the scanned point cloud data, the next frame of the scanned point cloud data is used as the source overall point cloud for precise alignment of the next frame. In this way, the final precise alignment is completed, and the registered point cloud data can be spliced ​​to achieve non-contact measurement for this large-scale weak feature scene.

[0100] As a specific embodiment, a non-contact laser marking device includes a conical reflector, a quartz tube, and a linear laser module. The non-contact laser marking device converts a laser beam into a 360° circular light curtain. By fixing the non-contact laser marking device at a specific location in the target scene, it provides features to assist in point cloud registration. During deployment, it is necessary to ensure that at least two intersecting laser line marks exist between two adjacent frames of the point cloud to be registered, in order to meet the three-dimensional constraints of the two-frame point cloud registration.

[0101] In another exemplary embodiment of the present application, there is provided Figure 3 The large-scale weak-feature scene scanning device based on multi-sensor fusion and structured light scanning shown in the figure includes a binocular structured light scanning camera 1 (including an MV-CS023-10GC industrial camera, a 12mm lens, and a DLP4500 digital projector), a UWB positioning tag 2 (an LD150 UWB positioning module), an inertial measurement unit IMU 3 (an HWT9073 IMU attitude sensor), a laser ranging module (a first laser ranging sensor 4-1 and a second laser ranging sensor 4-2), an embedded processor 5 (a BingPi-M2 embedded platform based on the Allwinner Technology T113-S3 processor), and a touch screen fixing plate 6 for interaction (equipped with a touch display screen, integrating device status monitoring, data uploading, and scanning control functions).

[0102] In addition to the scanning equipment, a laser marking projection device (consisting of a laser light source and a conical reflector model ZN-ZJ-01, which can produce colored laser line marks with a specific geometric distribution) must be deployed in advance in the scene to be scanned. It can be placed arbitrarily, and only needs to ensure that clear marks are generated on the surface of the target scene, and it must remain stationary during the scanning process.

[0103] Compared with the existing technology, the large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning provided by this application has the following significant advantages:

[0104] 1. Significantly improved applicability and registration success rate in large-scale weak feature scenes:

[0105] Existing technologies usually rely on the geometric or texture features contained in the point cloud data itself for matching and alignment. In large-scale scenes that lack significant features, such as aircraft skins, tunnel inner walls, and rough walls, feature extraction is difficult and matching is prone to errors, resulting in registration failure or low accuracy. However, the present application integrates the data of the inertial measurement unit (IMU), ultra-wideband (UWB) positioning system, and laser ranging module to accurately calculate the position and orientation of the scanning device corresponding to the single-frame point cloud to be registered in the global coordinate system in real time. The coarse registration process directly uses this global pose information to transform each frame of point cloud into a unified coordinate system, and does not rely on the features of the point cloud itself for initial alignment. Therefore, even in scenes where features are extremely sparse or repetitive, this method can still achieve reliable coarse registration and provide an initial value reference for subsequent fine registration.

[0106] 2. Improved the accuracy and robustness of point cloud coarse registration:

[0107] This application effectively suppresses the cumulative drift error caused by pure IMU integration through the fusion of multi-source sensor data, especially the combination of high-frequency relative motion information provided by IMU and the global absolute position reference provided by UWB, and performs optimal estimation through fusion algorithms such as extended Kalman filtering (EKF). The global positioning capability of UWB ensures the long-term stability of pose estimation under large-scale mobile scanning. At the same time, the optional laser ranging module can specifically improve the positioning accuracy in a specific direction (such as vertical height). High-precision real-time pose estimation is directly converted into high-precision point cloud coordinate transformation, making the coarse alignment result more accurate and more robust to sensor noise and environmental interference (such as some UWB non-line-of-sight signals).

[0108] 3. Significantly improved the efficiency of point cloud coarse registration:

[0109] Traditional coarse registration algorithms based on feature or point pair matching require complex feature extraction, description, and matching searches in overlapping areas, which is computationally intensive and inefficient, especially for large-scale scanning scenarios with large amounts of data. However, this application directly obtains the global pose through multi-sensor data fusion, simplifying the complex feature matching process of the traditional coarse registration method into a direct coordinate transformation, greatly improving the efficiency of coarse registration. Once the sensor data is fused to obtain the pose, the transformation and alignment of the corresponding point cloud frames can be completed quickly, greatly shortening the time required for coarse registration and improving the efficiency of overall 3D reconstruction or measurement.

[0110] 4. Using optical markers to assist, while maintaining high accuracy, it significantly improves the efficiency, convenience, and surface protection of fine registration in scenes with weak features:

[0111] In order to solve the problem that the accuracy after coarse registration is not enough to meet the needs of precision measurement, and the traditional fine registration method performs poorly in weak feature scenes, this application adopts a non-contact laser marking device based on a laser light source and a conical reflector. The non-contact laser marking device can generate laser marks with clear color and geometric features on the surface of the object to be measured, and such marks can be accurately extracted through image processing algorithms. The extracted laser mark point cloud is used as a corresponding feature set with high confidence, and algorithms such as iterative closest point (ICP) are used to perform registration calculations specifically for these mark points, so that accurate inter-frame transformation relationships can be obtained, thereby achieving high-precision and fine alignment of the entire point cloud. Compared with traditional high-precision registration technologies that rely on manual placement of a large number of physical markers on the surface of the object to be measured, the laser marking method of this application generates marks through non-contact projection, avoiding the time-consuming and labor-intensive process of manually pasting, measuring and recording physical markers, significantly improving the efficiency of the overall measurement and the convenience of operation. At the same time, since no physical contact is required, this solution avoids the risk of physical damage or contamination to the surface of the object being measured (especially precise, sensitive or fragile surfaces) that may be caused by physical markers and their operation processes, thereby achieving effective protection of the object being measured while ensuring high precision.

[0112] In summary, this application obtains high-precision global pose through an innovative multi-sensor fusion strategy, and performs point cloud coordinate transformation based on this. At the same time, a non-contact laser marking device based on a laser light source and a conical reflector is used to provide features, thereby achieving high-precision, high-efficiency, and high-robust point cloud alignment in large-scale weak-feature scenes, and realizing high-precision non-contact measurement.

[0113] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it can realize the various functions of a large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning provided in the previous embodiment.

[0114] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0115] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0116] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0117] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0118] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0119] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0120] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0121] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning, characterized by: include: Multi-sensor joint calibration module, used for collaborative calibration of binocular structured light scanning camera, inertial measurement module, positioning sensor module and laser ranging module; The binocular structured light scanning camera is used to scan the target scene; The target scene is a scene to be scanned with a large range and weak features; The inertial measurement module, the positioning sensor module and the laser ranging module are respectively fixed in position with the binocular structured light scanning camera, so as to realize the three-dimensional spatial positioning of the binocular structured light scanning camera; a multi-sensor data fusion module for fusing the positioning data of the inertial measurement module and the positioning sensor module using an extended Kalman filter algorithm, and correcting the fused positioning data in combination with the ranging information of the laser ranging module to obtain the three-dimensional spatial coordinates of the scene scanned by the binocular structured light scanning camera; a scanning point cloud data coarse registration module for calculating, for a plurality of frames of scanning point cloud data obtained by the binocular structured light scanning camera during continuous scanning, a relative rotation matrix and a relative displacement matrix of each frame of scanning point cloud data relative to the first frame of scanning point cloud data based on the three-dimensional spatial coordinates and Euler angles when the binocular structured light scanning camera performs scene scanning, and registering the coordinate system of each frame of scanning point cloud data to the coordinate system of the first frame of scanning point cloud data; the plurality of frames of scanning point cloud data each include a laser-marked point cloud remotely projected by a non-contact laser marking device; The scanning point cloud data fine registration module is used to extract the laser marker point cloud from each frame of scanning point cloud data based on the coarse registration, and use the color characteristics and geometric distribution patterns of the laser marker point cloud extracted from different frames of scanning point cloud data to iteratively fine-register each frame of scanning point cloud data. After the fine registration, the scanning point cloud data can be used for non-contact measurement of the target scene.

2. The large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning according to claim 1 is characterized in that: The multi-sensor joint calibration module includes: A binocular structured light scanning camera self-calibration unit, configured to calibrate the binocular structured light scanning camera using a visual system calibration technique based on perspective geometry, and determine and correct internal and external parameters of the binocular structured light scanning camera; A camera-inertial measurement module calibration unit, configured to collect a plurality of sets of scanning point cloud data and inertial measurement data, and solve the rotation matrix between the binocular structured light scanning camera and the inertial measurement module; The positioning-inertial measurement module calibration unit is used to use an IMU-UWB coordinate system normalization calibration method based on a constrained path on a two-dimensional horizontal plane to solve the installation angle of the inertial measurement module relative to the positioning sensor module coordinate system.

3. The large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning according to claim 2 is characterized in that: The positioning-inertial measurement module calibration unit is specifically used for: Establishing the positioning sensor module coordinate system and the inertial measurement module coordinate system on a two-dimensional horizontal plane respectively; moving the binocular structured light scanning camera equipped with the inertial measurement module from a known starting point to a known end point in the positioning sensor module coordinate system, and determining the motion vector and motion direction angle of the binocular structured light scanning camera in the positioning sensor module coordinate system; determining the motion vector and motion direction angle of the binocular structured light scanning camera in the inertial measurement module coordinate system by quadratic integration based on the acceleration data acquired by the inertial measurement module; The installation angle of the inertial measurement module relative to the positioning sensor module coordinate system is determined based on the movement direction angle of the binocular structured light scanning camera in different coordinate systems.

4. The large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning according to claim 2 is characterized in that: The laser ranging module includes: a first laser ranging sensor and a second laser ranging sensor; the first laser ranging sensor is used to perform real-time compensation for the Z-axis height data in the three-dimensional positioning of the positioning sensor module; the installation position of the first laser ranging sensor can be precisely adjusted through the attitude angle data provided by the inertial measurement module to ensure that the measurement direction of the first laser ranging sensor is strictly aligned with the vertical direction; the second laser ranging sensor is used to guide the operator to maintain the binocular structured light scanning camera at the optimal imaging distance for scanning operations through real-time distance feedback; the second laser ranging sensor is installed at a position close to the origin of the binocular structured light scanning camera coordinate system.

5. The large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning according to claim 4 is characterized in that: Correct the fused positioning data according to the following formula: WITH real =Z laser cos(φ)cos(θ)+d; Among them, Z laser is the height of the device obtained by the first laser ranging sensor, d is the vertical distance from the positioning sensor module to the first laser ranging sensor, Z real is the corrected Z-axis height data, θ and φ are the pitch angle and roll angle obtained by the inertial measurement module, respectively.

6. The large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning according to claim 1 is characterized in that: When the extended Kalman filter algorithm is used to fuse the positioning data of the inertial measurement module and the positioning sensor module, since the ranging equation of the positioning sensor module is nonlinear, the Jacobian matrix is ​​used for linear approximation to obtain the following formula: Among them, H d P is the matrix obtained by linearly approximating the ranging equation of the positioning sensor module. x,k is the x-axis position of the binocular structured light scanning camera in the positioning sensor module coordinate system at time k, P y,k is the position of the binocular structured light scanning camera on the y-axis in the positioning sensor module coordinate system at time k, (x i ,y i ) is the two-dimensional coordinate of the i-th positioning base station, i = {1, 2, 3, 4}; The expressions for the system equation prediction value, prediction covariance, and extended Kalman filter algorithm gain are calculated as follows: Among them, X k / k-1 is the predicted value of the system equation, P k / k-1 is the predicted covariance, the subscript k / k-1 represents the predicted value at time k based on the information at time k-1, and the subscript k-1 represents the posterior estimate at time k-1, Kg k is the gain of the extended Kalman filter algorithm, Q η is the covariance corresponding to the system noise η, R d is the diagonal matrix composed of ranging variance, A and B are the state coefficient matrix and input control matrix corresponding to the extended Kalman filter algorithm, u k-1 is the system input variable, and the corresponding expression is as follows: Among them, T s is the sampling time interval, C imu-uwb is the rotation matrix from the inertial measurement module coordinate system to the positioning sensor module coordinate system, (a x ′,a y ′,a z ') is the three-axis acceleration measured in the inertial measurement module coordinate system, and g is the acceleration due to gravity; The measurement values ​​of the positioning sensor module are updated and corrected to obtain the measurement residual, output estimation value and corresponding error estimation matrix, as shown in the following formula: Among them, e k is the measurement residual, X k / k is the output estimate, P k / k is the corresponding error estimation matrix, d k is the measurement value of the positioning sensor module at time k, d k-1 is the predicted value at time k based on the measurement value of the positioning sensor module at time k-1, and I is the unit matrix.

7. The large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning according to claim 6, characterized in that: Measurement residual e k The difference between the measurement value of the positioning sensor module and the predicted value of the inertial measurement unit integral operation can be used to determine whether the positioning sensor module has generated an NLOS error; the ranging residual judgment threshold is obtained by statistically analyzing the normal value under normal line of sight and no interference; when the measurement residual e k When the value is higher than the ranging residual judgment threshold, it can be considered that the positioning sensor module has generated an NLOS error, and the measurement value of the positioning sensor module is discarded.

8. The large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning according to claim 1 is characterized in that: The relative rotation matrix and relative displacement matrix of each frame of scanned point cloud data relative to the first frame of scanned point cloud data are determined according to the following formulas: T relative,i =T i -T1; Among them, R relative,i is the relative rotation matrix of the i-th frame scan point cloud data relative to the first frame scan point cloud data, T relative,i is the relative displacement vector of the i-th frame scan point cloud data relative to the first frame scan point cloud data, R C,1 is the rotation matrix of the first frame scan point cloud data, the superscript T is the transpose of the matrix, T1 is the displacement matrix of the first frame scan point cloud data, R C,i is the rotation matrix of the scanned point cloud data of the i-th frame, T i is the displacement matrix of the scanned point cloud data of the i-th frame; The coordinate system of each frame of scanned point cloud data is aligned to the coordinate system of the first frame of scanned point cloud data according to the following formula: P′={R relative,i P i +T relative,i |i=1,2,...,N}; Among them, P i Scan point cloud data for the i-th frame, is the set of scanned point cloud data, N is the total number of frames of scanned point cloud data, and P′ is the set of all scanned point cloud data after coarse registration.

9. The large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning according to claim 1, characterized in that: The scanning point cloud data fine registration module is specifically used for: on the basis of coarse registration, using the color region growing segmentation algorithm to extract the laser marking point cloud in each frame of scanning point cloud data; for the first frame scanning point cloud data and the second frame scanning point cloud data, using the first frame scanning point cloud data as the target overall point cloud, and the laser marking point cloud extracted from the first frame scanning point cloud data as the target laser marking point cloud, using the second frame scanning point cloud data as the source overall point cloud, and the laser marking point cloud extracted from the second frame scanning point cloud data as the source laser marking point cloud; for the source overall point cloud, the source laser marking point cloud, the target overall point cloud and the target laser marking point cloud, using the iterative nearest neighbor search algorithm to iteratively solve the source laser marking point cloud and the target laser marking point cloud using the least squares method to minimize the matching error, and obtain the optimal rotation matrix and the optimal translation vector; applying the optimal rotation matrix and the optimal translation vector to the source overall point cloud for coordinate transformation, so that the source overall point cloud and the target overall point cloud are precisely registered.

10. The large-scale weak feature scene measurement system based on multi-sensor fusion and structured light scanning according to claim 1, characterized in that: The non-contact laser marking device includes: a conical reflector, a quartz tube and a straight line laser module; the non-contact laser marking device is used to convert a laser beam into a 360° circular light curtain. By fixing the non-contact laser marking device at a specific position in the target scene, feature assistance is provided for the target scene to perform point cloud registration.

Images