Depth point cloud correction and defect detection method and system for inclination angle compensation

Abstract

The invention discloses a depth point cloud correction and defect detection method and system for inclination angle compensation, relates to the technical field of industrial visual inspection and three-dimensional point cloud data processing, and is used for solving the technical problem that the defect detection precision is influenced by geometric distortion of depth point cloud caused by attitude disturbance of an acquisition platform. The method comprises the following steps: synchronously acquiring an original depth image and an attitude quaternion, and executing two-stage core correction by a point cloud geometric correction engine: firstly, performing inverse rotation transformation according to the attitude quaternion, and preliminarily correcting a point cloud orientation; then, depth projection geometric reverse mapping is executed on each point critically, and a high-fidelity corrected point cloud is generated by calculating a projection included angle and performing inversion compensation on depth value non-uniform compression caused by inclination. According to the method, on the basis of the physical imaging principle, refined mathematical correction of single-frame point cloud geometric distortion is achieved, and the accuracy and robustness of surface defect detection under the dynamic disturbance working condition are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of industrial visual inspection and 3D point cloud data processing technology, specifically to a method and system for depth point cloud correction and defect detection with tilt compensation. Background Technology

[0002] With the deepening advancement of industrial intelligence and digital twin technologies, 3D perception methods based on deep point clouds have demonstrated irreplaceable technological value in several key fields, including precision manufacturing, bridge and tunnel health monitoring, wind turbine blade inspection, and rail transit operation and maintenance. These methods acquire geometric information of the object's surface through high-density spatial sampling, providing fundamental data support for subsequent morphology reconstruction, dimensional measurement, and even defect identification.

[0003] In real-world, complex operating conditions, data acquisition platforms (such as mobile robots, drones, or handheld scanning devices) often experience attitude shifts due to terrain undulations, mechanical vibrations, or operational disturbances, with changes in pitch and roll angles being particularly pronounced. Although these tilt disturbances are small, they can directly cause unexpected rotations of the depth sensor coordinate system relative to the world coordinate system. This results in systematic compression, stretching, or misalignment of the acquired point cloud in local areas, severely compromising the fidelity of the original surface geometry and consequently causing fatal interference to defect detection algorithms that rely on high-precision geometric features.

[0004] Existing technologies attempt to compensate by introducing attitude sensing units. Patent CN111912345B proposes a self-compensating laser scanning device and method for coal discharge volume of a hydraulic support. In a fully mechanized coal mining face, an angle sensor monitors the movement angle of the rear connecting rod of the support in real time, and drives the lidar to perform mechanical pitch leveling, thereby maintaining the stability of the scanning beam relative to the coal flow direction. This solution effectively improves the repeatability of coal pile volume estimation in specific enclosed scenarios. Its core logic lies in converting attitude disturbances into structural motion parameters and achieving active correction of the front-end acquired attitude through a physical actuator.

[0005] The invention patent with announcement number CN120593714B addresses collaborative mapping tasks involving multiple UAVs along railway lines. It integrates six-degree-of-freedom pose information acquired from various airborne position and attitude systems to perform global registration and coordinate normalization of multi-view point clouds, ensuring geometric consistency in the extraction of the track centerline. While this method improves the spatial consistency of point clouds on a macro scale, its compensation mechanism essentially serves the purpose of multi-source data fusion or overall modeling accuracy, rather than providing refined mathematical reconstruction for local depth distortion caused by tilt angles within a single frame of point cloud.

[0006] Mechanical leveling strategies rely heavily on the kinematic model of specific equipment and can only handle a single tilt component in a preset direction. They cannot adapt to the general correction requirements under arbitrary attitude disturbances. More importantly, their compensation effect occurs before data acquisition, and no post-processing correction is performed on the acquired point cloud data itself. As a result, once the leveling mechanism responds late or there is an installation error, the distorted data is solidified and difficult to trace and repair.

[0007] While pose-based point cloud registration methods possess some versatility, their compensation granularity is typically measured in whole frames or flight strips, neglecting the non-uniform influence of tilt angle on local point cloud depth values. For example, under tilt, the depth error of points at different distances on the same plane exhibits a non-linear distribution. If only rigid body transformation is applied without considering the inverse mapping of depth projection geometry, significant deformation residuals will still remain. Consequently, this incompletely eliminated geometric distortion directly propagates to the downstream defect detection module: surface depressions may be misjudged as normal undulations, minute cracks may be lost due to point cloud sparsity, and even regular structures may trigger false alarms due to distortion. Existing technologies separate pose compensation and defect identification into two independent stages. The former pursues macroscopic coordinate alignment, while the latter relies on ideal point cloud input. The lack of a closed-loop feedback mechanism for geometric fidelity optimization between the two results in severely insufficient robustness of the entire detection process when facing dynamic disturbances. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for tilt-compensated depth point cloud correction and defect detection, solving the technical problem in the prior art where geometric distortion of depth point clouds is caused by attitude disturbance of the acquisition platform, which in turn affects the accuracy and robustness of defect detection.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A tilt-compensated depth point cloud correction and defect detection method is applicable to depth sensor modules that are not horizontally mounted or have attitude deviations. This method performs geometric correction on the acquired raw point cloud data and achieves high-precision surface defect detection. By immediately performing refined coordinate correction based on a physical imaging model after point cloud data generation, and directly feeding the correction results into a defect identification module focused on extracting surface microstructure features, the entire detection process maintains high geometric fidelity and high discrimination accuracy even under dynamic disturbance conditions.

[0010] The tilt-compensated depth point cloud correction and defect detection includes a depth sensor module, an inertial measurement unit, a central collaborative controller, a point cloud geometry correction engine, a surface feature analysis module, and a defect decision output unit.

[0011] The depth sensor module acquires the raw depth image of the surface of the object being measured. Its output format is a two-dimensional array D(u,v), where u and v represent the horizontal and vertical pixel coordinates of the image plane, respectively, and D(u,v) represents the distance value of the corresponding pixel along the optical axis of the sensor. The inertial measurement unit is fixedly mounted on the rigid base of the depth sensor module and is used to output the triaxial acceleration and triaxial angular velocity data at the acquisition time in real time. It also generates the current attitude quaternion by fusing the data through an internal Kalman filter. The quaternion represents the rotation relationship of the depth sensor coordinate system relative to the preset world coordinate system; the central coordinating controller synchronously receives the depth image and attitude quaternion through the SPI bus and transmits them to the point cloud geometry correction engine and the surface feature analysis module, respectively.

[0012] The point cloud geometry correction engine first backprojects the original depth image D(u,v) based on the intrinsic parameter matrix K of the depth sensor to generate an uncorrected 3D point cloud. ,in f x f and fᵧ are the focal length parameters of the depth sensor in the x and y directions, respectively. and These are the coordinates of the principal point of the image; for example, in this embodiment, Pixels Pixels; then, the point cloud geometry correction engine constructs the corresponding 3×3 rotation matrix R based on the pose quaternion q, and calculates its inverse matrix R⁻¹; next, coordinate transformation is performed on each 3D point P0. The point cloud P1 is obtained after preliminary correction. Based on this, the point cloud geometric correction engine further introduces a depth projection geometric inverse mapping model to perform secondary correction for the non-uniform compression effect of depth values ​​caused by sensor tilt: for any point Its original depth reading Z0 in the tilted state is compared with its real-world depth. The following relationship exists between them: ,in Let be the angle between the projection of this point and the optical axis in the sensor coordinate system, and its value is given by... Determined; therefore, the point cloud geometry correction engine solves... Obtain the final correction depth According to the trigonometric identities This solution is equivalent to calculating the point The Euclidean distance to the origin of the coordinate system, i.e. Subsequently, the point coordinates were updated accordingly. This generates a high-fidelity corrected point cloud P2.

[0013] In a preferred embodiment of the present invention, the point cloud geometry correction engine employs a lookup table method to accelerate trigonometric function calculations when performing the aforementioned inverse mapping: an angle-cosine reciprocal mapping table is pre-constructed during the system initialization phase, wherein... The range of values ​​is The step size is 0.1°, with a total of 851 entries, stored in on-chip SRAM; during runtime, the calculated... Convert to angle value. Located in the interval Inside, according to value pairs Linear interpolation is performed to obtain a high-precision approximation; if α is greater than or equal to 85°, then the value is directly taken. A fixed value is used to avoid the situation where α approaches 90° due to... The method avoids the instability caused by values ​​approaching zero. It avoids the computational delays introduced by floating-point division and trigonometric function calls.

[0014] The surface feature analysis module receives the corrected point cloud P2 and performs local geometric feature extraction. P2 is downsampled using a voxel mesh, with the voxel side length set to δ=2mm to balance computational efficiency and detail preservation. A k-nearest neighbor graph is constructed within each non-empty voxel, k=16, and the principal curvature, Gaussian curvature, and mean curvature of each point are calculated based on the covariance matrix. Furthermore, the surface feature analysis module introduces a normal consistency constraint mechanism: for each point p∈P2, its normal vector is calculated. And perform a dot product with the normal vectors of its k nearest neighbors. If the mean of the dot product is lower than a threshold... If the point is found to be in a geometrically discontinuous region, it is marked as a potential defect candidate point. Simultaneously, the surface feature analysis module also calculates the local point density. , defined as the number of neighborhood points within a unit sphere, if Points with a density less than 60% of the global average density are also marked as sparse outliers.

[0015] The defect decision output unit receives the candidate point set output by the surface feature analysis module and performs fusion and discrimination by combining multi-dimensional features. The defect decision output unit incorporates a lightweight convolutional neural network model. Its input is a local point cloud block centered on each candidate point with a radius of r=15mm, which is normalized and converted into a 32×32×32 voxel grid. The network structure contains three 3D convolutional layers with kernel sizes of 5×5×5, 3×3×3, and 3×3×3, and channel numbers of 32, 64, and 128 respectively, followed by a global max pooling layer and two fully connected layers. The output consists of three labels: normal surface, dent defect, and crack defect. During the training phase, synthetic datasets and real labeled data are used for joint optimization. The synthetic data generation method is as follows: on the surface of a standard CAD model, dent defects (using Gaussian functions to simulate depth distribution, diameter 2–20mm, depth 0.1–2mm) or crack defects (using B-spline curves to simulate paths, length 5–50mm, width 0.2–1mm) are injected at randomly selected locations. Subsequently, simulated tilt perturbations (pitch ±15°, roll ±10°) were applied to the model, and a point cloud distortion model was applied: for each point, the projection angle θ was calculated based on its original coordinates and the perturbation angle, and then... Update the depth values ​​and generate a distorted point cloud, where This represents the distortion depth.

[0016] Furthermore, the central collaborative controller is equipped with a hardware timestamp synchronization unit, which sends synchronization trigger signals to the depth sensor module and the inertial measurement unit through GPIO pins to ensure that the time deviation of the data acquisition between the two does not exceed ±50μs. The timestamp synchronization unit is implemented using FPGA and integrates a high-precision counter and a phase-locked loop to ensure that the multi-source sensor data are strictly aligned within the microsecond time window.

[0017] In another preferred embodiment of the present invention, the system further includes an adaptive attitude verification module, which operates within the central co-controller and is used to evaluate the reliability of the attitude output of the inertial measurement unit online. This module continuously monitors the integral drift of the angular velocity. If it exceeds the preset threshold This triggers the attitude reset process: Pause the point cloud correction engine and start the vision-inertial tightly coupled initialization algorithm. The vision-inertial tightly coupled initialization algorithm starts from the point cloud corresponding to the current original depth image. In this process, the RANSAC algorithm is used to fit one or more large planes; assuming that the largest plane should be a horizontal plane in the world coordinate system, the normal of this plane is calculated. ;according to With respect to the Z-axis of the world coordinate system The relationship is used to calculate the current pitch angle. With roll angle The estimated value; the solution process is as follows: Let the normal vector of the dominant plane obtained by fitting be... Pitch angle Calculated as Roll angle Calculated as Among them, a The function is a four-quadrant arctangent function, ensuring the attitude angle range covers ±180°. The calculated... and The angular velocity was corrected to zero bias by comparing it with the IMU integration results.

[0018] Finally, these visual estimates are compared with the IMU integration results to correct the IMU's angular velocity zero bias. This verification mechanism effectively prevents the accumulation of attitude errors caused by IMU zero bias drift during long-term operation.

[0019] At the physical implementation level, the depth sensor module employs a time-of-flight depth sensor with an effective resolution of 640×480, a depth measurement range of 0.3m to 5.0m, and a depth accuracy better than ±1mm at a distance of 1m. The inertial measurement unit uses a six-axis IMU chip manufactured using MEMS technology, with a gyroscope zero-bias instability of 0.5° / hr and an accelerometer noise density of 100μg / √Hz. The central co-controller is built on an ARM Cortex-A72 quad-core processor with a main frequency of 1.8GHz, equipped with 4GB LPDDR4 memory and 64GB eMMC storage. The point cloud geometry correction engine and surface feature analysis module are implemented in C++ and use the NEONSIMD instruction set for vectorization acceleration. The neural network inference of the defect decision output unit is executed through a dedicated NPU coprocessor with a computing power of 4 TOPSINT8 and a power consumption of less than 2W.

[0020] The specific execution flow of the method is as follows: After power-on, the initialization of each module is completed, including loading of the depth sensor internal components, IMU zero bias calibration, LUT table generation, and loading of the neural network model; Once in continuous working mode, the central coordinating controller periodically sends a synchronization trigger signal, and the depth sensor module and inertial measurement unit synchronously collect data. The original depth image and pose quaternion are fed into the point cloud geometry correction engine, which performs three-step correction: back projection, inverse rotation transformation and inverse depth mapping, and outputs a high-fidelity point cloud. After voxel downsampling, the high-fidelity point cloud is input into the surface feature analysis module to extract curvature, normal consistency and point density features, and screen out candidate defect regions. The defect decision output unit performs local voxelization and neural network inference on each candidate region, outputs the final defect type and spatial location, and uploads it to the monitoring terminal via Ethernet interface.

[0021] By deeply fusing attitude information with a depth imaging geometric model, point-by-point depth correction of a single-frame point cloud under arbitrary tilt angle perturbation is achieved, effectively eliminating non-uniform geometric distortion caused by platform tilt. Based on this, the constructed surface feature analysis and defect discrimination module directly applies to the high-fidelity point cloud, significantly improving the detection rate of micron-level depressions and sub-millimeter-level cracks, while greatly reducing the false alarm rate. The entire system adopts a hardware synchronization and embedded optimization design, enabling real-time processing on mobile robot or drone platforms with a frame rate of no less than 15fps (640×480 resolution), meeting the needs of online inspection in industrial settings.

[0022] As another embodiment of the present invention, when a known reference plane exists in the application scenario, the system can enable the plane constraint optimization mode: after the point cloud geometry correction engine completes the initial correction, the surface feature analysis module automatically detects the dominant plane in the scene and fits its normal vector. ;like The angle with the Z-axis of the world coordinate system is greater than the preset tolerance. The error signal is then fed back to the point cloud geometry correction engine, which fine-tunes the pose quaternion q to ensure that the dominant plane in the corrected point cloud is strictly parallel to the XY plane. This closed-loop optimization further improves the geometric fidelity in structured scenes.

[0023] Compared with the prior art, the present invention has the following beneficial effects: Traditional methods simply treat attitude perturbations as rigid body rotation and only perform coordinate alignment, neglecting the essence of the depth sensor's measurement mechanism. The output depth value is not the true spatial distance of the target point, but rather its projected length along the optical axis. This invention clearly isolates and processes projection distortion. Through a series correction of inverse rotation transformation and inverse depth mapping, it not only unifies the coordinate system but also restores the depth values ​​systematically compressed due to tilt point by point, achieving a physically accurate reconstruction of the point cloud geometry.

[0024] This invention initially corrects the overall spatial orientation of the point cloud by performing back projection based on a pinhole model and inverse transformation based on a pose rotation matrix on the original depth image. Subsequently, a crucial depth inverse mapping operation is introduced. Based on the orientation angle of each point relative to the optical axis, mathematical inversion is used to compensate for the depth shortening effect caused by projection, generating a high-fidelity point cloud with physical consistency. This avoids nonlinear geometric distortions such as local stretching and compression caused by tilt, providing a reliable data foundation for all subsequent detailed analyses.

[0025] Since subsequent curvature calculations, normal estimations, and other local geometric feature analyses are entirely based on the aforementioned high-fidelity point cloud, the extracted features reflect the true physical properties of the measured surface. Normal consistency judgment accurately distinguishes between real edges and distortion artifacts, and point density analysis effectively identifies genuine sparse regions. This ensures that the selection criteria for defect candidate points are based on solid geometric realism, significantly improving the accuracy of candidate regions and greatly reducing false alarms and missed detections caused by data distortion.

[0026] This invention enables the network to inherently possess robustness against residual distortion by actively injecting simulated tilt distortion data during the model training phase. When high-quality point cloud blocks are input after the aforementioned correction process, the network can focus more on identifying the essential geometric and topological features of defects such as cracks and depressions, rather than adapting to distorted data artifacts, thereby achieving a synergistic improvement in discrimination accuracy and robustness under complex working conditions.

[0027] This invention achieves its goals through an end-to-end closed-loop correction chain based on physical imaging principles. By deeply integrating attitude perception, physical model inversion, and intelligent discrimination, it possesses the ability to fundamentally resist attitude disturbances, ultimately achieving a fundamental breakthrough in consistently outputting high-precision and high-reliability 3D defect detection results even under dynamic and non-ideal acquisition conditions. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0029] Figure 1 This is an overall flowchart of the method described in this invention.

[0030] Figure 2 This is a connection diagram of the system described in this invention. Detailed Implementation

[0031] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0032] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0033] Example 1: See Figure 1 and Figure 2The overall architecture of this embodiment consists of a depth sensor module, an inertial measurement unit (IMU), a central collaborative controller, a point cloud geometry correction engine, a surface feature analysis module, and a defect decision output unit. Each module achieves a closed-loop process from raw data acquisition to high-precision defect identification through a hardware and software collaborative mechanism.

[0034] First, the initialization process is executed. The depth sensor module loads the pre-calibrated intrinsic parameter matrix K, which has the following form: ; in, Pixels Pixels , This corresponds to the optical center and focal length parameters of the time-of-flight depth sensor module at a resolution of 640×480.

[0035] The inertial measurement unit (IMU) performs a zero-bias calibration procedure: it acquires 10 seconds of raw accelerometer and gyroscope data while stationary, calculates the average value, and stores it as the initial zero-bias compensation value in an internal register. Simultaneously, the point cloud geometry correction engine constructs an angle-cosine reciprocal mapping table in the on-chip SRAM. The range of values ​​for α is The step size is 0.1°, with a total of 851 entries, each stored in Q15 fixed-point format. The approximate value is used for fast trigonometric function lookup operations in subsequent deep inverse mapping. The defect decision output unit loads the weight file of the trained 3D convolutional neural network model from the eMMC storage. The model has an input dimension of 32×32×32 voxel grid and outputs three-class classification results.

[0036] After initialization, the system enters continuous operation mode. The FPGA synchronization unit built into the central coordinating controller sends synchronization trigger signals to the depth sensor module and the inertial measurement unit via GPIO pins, with a trigger period of 66.7ms (corresponding to a 15fps frame rate). Upon receiving the trigger signal, the depth sensor module immediately captures a 640×480 resolution depth image D(u,v), where... This represents the distance of pixel (u, v) along the sensor's optical axis, in millimeters, with an effective measurement range of 300mm to 5000mm. The inertial measurement unit latches its internal state at the same trigger moment and outputs triaxial acceleration. With triaxial angular velocity It also uses its built-in Extended Kalman Filter (EKF) to fuse acceleration and angular velocity data to generate the current attitude quaternion. ,in For the real part, The part is imaginary and satisfies the unit quaternion constraint. This quaternion represents the rotational relationship of the depth sensor coordinate system relative to the preset world coordinate system (Z-axis vertically upward). The FPGA synchronization unit ensures that the timestamp deviation between the depth image and the pose quaternion does not exceed ±50μs, thereby guaranteeing strict alignment of multi-source data within a microsecond-level time window.

[0037] The central coordinating controller receives the two data streams via the SPI bus and transmits them to the point cloud geometry correction engine and the surface feature analysis module, respectively. The point cloud geometry correction engine first performs a back-projection operation on the original depth image D(u,v) to generate an uncorrected 3D point cloud. Each valid pixel (u, v) has its coordinates calculated using the following formula: ; ; ; in, For pixel coordinates, For the corresponding depth value (unit: millimeters). , The depth sensor is respectively in Focal length in direction (unit: pixels). The principal point coordinates (unit: pixels). For each valid pixel, its depth value satisfies... .

[0038] This yields point clouds. The Cartesian coordinates of each point are determined. Subsequently, the point cloud geometry correction engine constructs a corresponding 3×3 rotation matrix R based on the pose quaternion q. The conversion formula from quaternion to rotation matrix is: ; Next, calculate the inverse matrix of R. Since R is an orthogonal matrix, its inverse is equal to its transpose, i.e. .right Each point in Perform coordinate transformation: ; Point cloud after preliminary correction The point cloud has eliminated the global coordinate offset caused by the overall rotation of the platform.

[0039] Building upon this, the point cloud geometry correction engine further performs depth projection geometry inverse mapping to compensate for the non-uniform compression effect of depth values ​​caused by sensor tilt. This effect stems from the fact that depth sensors can only measure distances along their optical axis. When the sensor pitches or rolls, the depth readings of the same physical point at different viewpoints will exhibit systematic deviations due to the projection angle. Specifically, for any point... , The three-dimensional coordinates after inverse rotation transformation, and their projection angles relative to the optical axis in the sensor coordinate system. Defined as: ; In an ideal scenario, real-world depth Compared with the original depth reading The following conditions must be met: ; However, due to It is an intermediate result after inverse rotation transformation. The component still retains its original depth. The compression characteristics of this feature mean that the true depth needs to be recovered through iterative or direct solutions. .

[0040] This implementation method uses the direct analysis method: ; Using trigonometric identities We can obtain: ; This result indicates that, under this model, the corrected depth That is, a point The Euclidean distance in the sensor coordinate system. It should be noted that the above derivation is based on the following premises: the inertial measurement unit and the depth sensor are rigidly connected, and there is only a rotational relationship between their coordinate systems, with no relative translation; and the origin of the world coordinate system coincides with the origin of the sensor coordinate system. In practical applications, if there is installation offset or translational movement, translation compensation needs to be performed before or after the aforementioned inverse rotation transformation; this is an scalable scenario for the present invention.

[0041] The specific compensation method is as follows: Define a fixed translation vector between the sensor coordinate system and the IMU coordinate system. Before performing the inverse rotation transformation, the point cloud coordinates are subtracted. Alternatively, in the world coordinate system, the attitude output by the IMU can be applied to the translated point. For example, for point P0, the compensated coordinates are... Then apply the inverse transformation of the rotation matrix.

[0042] Therefore, the final correction depth It can be directly calculated as follows: ; Subsequently, to maintain the consistency of the projection of points onto the image plane, scaling is required. and Quantity: ; Thus, the final corrected point cloud is obtained. This process ensures that the depth value of each point reflects its true radial distance in the world coordinate system while maintaining its relative position in the image plane, thereby maximizing geometric fidelity.

[0043] As a preferred embodiment of the present invention, the above calculations... The process is accelerated by using a table lookup method. During runtime, Calculated in radians and then converted to degrees. Obtained from the LUT table through linear interpolation The Q15 fixed-point approximation is then converted to a floating-point number for computation. This strategy reduces the single-point correction time from approximately 120ns to 45ns, achieving a full-frame (307,200 points) correction time of less than 15ms on the ARM Cortex-A72 platform.

[0044] Correction point cloud It is then fed into the surface feature analysis module. This module first performs voxel mesh downsampling, with voxel side lengths... Set to 2mm.

[0045] In the specific implementation, the point cloud space is divided into a cubic mesh with a side length of 2mm. For all points falling into the same mesh, their geometric center is retained as a representative point, thereby reducing the subsequent computational load. After downsampling, the point cloud density is about 1 / 8 of the original, but sub-millimeter-level surface details are still preserved.

[0046] Within each non-empty voxel, the module constructs a k-nearest neighbor graph, where k=16. For each point... The 16 nearest neighbors of a given point are searched by Euclidean distance, forming a local neighborhood set N(p). Based on this set, the covariance matrix is ​​calculated. : ; in for The center of mass. For Eigenvalue decomposition yields three eigenvalues. and its corresponding eigenvectors. Principal curvature Defined as: ; Gaussian curvature mean curvature These curvature indices are used to characterize the unevenness and flatness of local surfaces.

[0047] Furthermore, the surface feature analysis module introduces a normal vector consistency constraint mechanism. For each point p, its normal vector... Take as the minimum eigenvalue The corresponding eigenvectors. Calculate. Its normal vectors to its 16 nearest neighbors Mean of the dot product: ; like If the point is located in a geometrically discontinuous region (such as an edge, crack, or depression boundary), it is identified as a potential defect candidate point. Simultaneously, the module calculates the local point density. Defined as having point p as center and radius as... The number of points contained within the sphere.

[0048] like ,in If the average point density of the entire point cloud is given (unit: points / mm³), it is marked as a sparse outlier, which may be caused by occlusion, missing reflections, or surface detachment.

[0049] The defect decision output unit receives the above candidate point set. For each candidate point... Extract a local point cloud patch centered on it with a radius r = 15 mm. .Will Normalization to A cubic space is divided into a 32×32×32 voxel grid. Each voxel is marked as 1 if it contains at least one point, and 0 otherwise, forming... A binary voxel occupancy map. This map is used as input to a lightweight 3D convolutional neural network.

[0050] The network structure consists of the following layers: a first 3D convolutional layer (kernel 5×5×5, channels 32, stride 2), a second 3D convolutional layer (kernel 3×3×3, channels 64, stride 1), a third 3D convolutional layer (kernel 3×3×3, channels 128, stride 1), followed by a global max pooling layer (outputting a 128-dimensional vector), then two fully connected layers (256-dimensional → 3-dimensional), and finally outputting three types of probabilities through Softmax: normal surface, dent defect, and crack defect.

[0051] A hybrid dataset was used during the training phase: synthetic data was based on standard industrial part CAD models (such as bearing housings and flanges), with depressions of 2–20 mm in diameter and 0.1–2 mm in depth, or cracks of 5–50 mm in length and 0.2–1 mm in width randomly injected into their surfaces; each synthetic point cloud was superimposed with simulated tilt perturbations (pitch ±15°, roll ±10°), and training samples were generated using the aforementioned point cloud distortion model. Real data consisted of 12,000 labeled defect samples collected from factory sites. The model employed a cross-entropy loss function, the Adam optimizer, and a learning rate of 1e-4. After 200 training epochs, the validation set accuracy reached 96.7%.

[0052] In another preferred embodiment of the present invention, the system integrates an adaptive attitude verification module. This module continuously monitors the integral drift of the IMU angular velocity. The calculation method is as follows: angular velocity Perform time integration to obtain the change in attitude angle, and then compare it with the attitude angle estimated visually. If If the IMU zero-bias drift exceeds the limit, the attitude reset process is triggered. At this time, the system pauses the point cloud correction engine and starts the vision-inertial tightly coupled initialization algorithm: using the point cloud before correction... The large-area plane detected in the image (fitted by RANSAC) is assumed to be a horizontal plane in the world coordinate system (Z = constant). The current pitch angle is then calculated. With roll angle This is used to update the IMU's bias parameters. This mechanism effectively suppresses attitude accumulation errors during long-term operation.

[0053] At the physical implementation level, the depth sensor module uses a Sony IMX556PLR time-of-flight sensor, operating at 15fps at 640×480 resolution, with depth noise of ±0.8mm (1σ) at a distance of 1m. The inertial measurement unit uses a Bosch BMI088 six-axis IMU, with gyroscope zero-bias instability of 0.5° / hr and accelerometer noise density... The central co-controller is based on the NXP.MX8MPlusSoC, featuring a quad-core ARM Cortex-A72 at 1.8GHz, 4GB LPDDR4 memory, 64GB eMMC, and an integrated dedicated NPU coprocessor (4 TOPSINT8, 1.8W TDP). The point cloud geometry correction engine and surface feature parsing module are implemented in C++17, with key loops calling the NEONSIMD instruction set for vectorization, such as parallel processing of backprojection and rotation operations on 4 floating-point numbers in a 128-bit register. Defect decision inference is executed by the NPU, with a single-frame inference time of 8.2ms.

[0054] To further improve geometric fidelity in structured scenes, the system supports a planar constraint optimization mode. When the application scene has a known reference plane (such as a conveyor belt or desktop), the surface feature analysis module performs point cloud analysis. Automatic detection of dominant planes: The RANSAC algorithm is used to fit the largest cluster of planes in the point cloud to obtain their normal vectors. .like Angle with the Z-axis of the world coordinate system Then calculate the attitude error quaternion. And fine-tune the original pose quaternion The feedback is sent to the point cloud geometry correction engine to re-execute the correction. This closed-loop optimization ensures that the dominant plane is strictly parallel to the XY plane in the corrected point cloud, and the plane fitting residual is reduced from an average of 1.8 mm to 0.3 mm.

[0055] The system described in this invention is used to detect defects in a batch of aluminum alloy castings in a simulated production line environment. The castings have artificially created dents (5mm in diameter, 0.3mm in depth) and cracks (10mm in length, 0.4mm in width) on their surfaces. The detection platform is mounted on the end effector of a six-degree-of-freedom robotic arm, performing random pitch (±12°) and roll (±8°) disturbances. The system continuously acquires 100 frames of point cloud data at 15fps and performs full-process processing.

[0056] Comparative Example 1: Using the same hardware platform, but disabling the point cloud geometry correction engine, the original point cloud P_0 is directly input into the surface feature analysis module.

[0057] Comparative Example 2: Using the traditional approach, only global rotation correction is performed (i.e., only... (No deep inverse mapping correction is performed).

[0058] Comparative Example 3: Defect detection was performed using commercial point cloud processing software (such as StatisticalOutlierRemoval+SACMODEL_PLANE from the PCL library) in conjunction with a fixed attitude assumption.

[0059] The performance evaluation metrics for detection include: dent detection rate, crack detection rate, false alarm rate, and single-frame processing latency. The results are shown in Table 1 below. Table 1: Data shows that this invention significantly improves the detection capability of minute defects under dynamic disturbance conditions, while keeping the false alarm rate below 2%. Comparative Example 1, due to the lack of any correction, suffers from severe point cloud geometric distortion, leading to a large number of real defects being misjudged as normal undulations; Comparative Example 2, although correcting for global rotation, does not compensate for the depth compression effect, resulting in an underestimation of the depression depth in edge regions, affecting detection; Comparative Example 3 relies on static assumptions, leading to a sharp decline in performance in dynamic scenes and high processing latency.

[0060] Furthermore, in long-term operational stability testing, the system operated continuously for 8 hours, recording the dominant plane fitting residuals every 30 minutes. In the embodiment where the adaptive attitude verification module was enabled, the residuals remained below 0.5 mm; while in the variant where the verification module was disabled, the residuals rose to 2.1 mm after 4 hours, demonstrating that the verification mechanism effectively suppressed IMU drift.

[0061] Existing technologies, whether mechanical leveling or global registration, rely on rigid body kinematics for compensation logic, implicitly treating depth sensors as ideal devices measuring real distances. Essentially, they only address the overall rotation of the coordinate system, completely ignoring the fundamental principles of depth sensor imaging: the raw depth image D(u,v) output by each pixel represents the distance of the target point along the sensor's optical axis, i.e., the projected component of the real spatial distance. When the sensor tilts, this depth projection geometry causes a non-uniform compression effect in the measured value, and the degree of compression varies with the projection angle. This distortion in point clouds is exacerbated by the increase in depth. This is the physical root cause of local geometric distortion in point clouds, and traditional rigid body transformations are powerless to address it because they do not address the distortion of the depth value itself.

[0062] This invention decomposes "attitude perturbation" into two physically independent effects that require cascaded correction in data processing, and constructs a corresponding two-level correction architecture accordingly. The point cloud geometry correction engine performs an inverse rotation transformation based on the rotation matrix R derived from the attitude quaternion q to correct the overall coordinate system offset. The point cloud geometry correction engine further performs a depth projection geometry inverse mapping. This step is not based on empirical adjustments, but strictly adheres to the projection model. The mathematical inversion is performed by solving... The true depth Z2 is recovered, and the point coordinates are updated to the corrected point cloud P2 accordingly. The above process reverses the process of each point from its compressed measurement position to its true radial position in world space, completing the geometric reconstruction from the sensor observation space to the objective physical space.

[0063] The attitude quaternion q provided by the inertial measurement unit of this invention is no longer only used for coordinate rotation, but is also used in the point cloud geometry correction engine to solve for the projection angle of each point. The key input. This fundamental correction allows the subsequent surface analysis based on local geometric features such as curvature and normal consistency in the surface feature analysis module to be performed on the corrected point cloud P2 with true geometric fidelity, rather than on distorted data artifacts. The improvement in defect detection accuracy and robustness ultimately achieved by the defect decision output unit is not obtained by increasing the complexity of subsequent recognition algorithms, but by thoroughly "purifying" the geometric authenticity of its input data at the front end through the point cloud geometry correction engine.

[0064] This invention transforms attitude information from a traditional coordinate alignment parameter into a core parameter driving the inverse depth projection geometry mapping, thereby designing a two-stage cascaded point cloud geometry correction method involving inverse rotation transformation and inverse depth mapping within the point cloud geometry correction engine. This eliminates the inherent interference of dynamic conditions on the accuracy of 3D vision measurements.

[0065] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for depth point cloud correction and defect detection with tilt compensation, characterized in that, The method comprises the following steps: Step 1: obtaining an original depth image of a surface of an object to be measured by a depth sensor module, each pixel point in the original depth image having a distance value along the direction of the optical axis of the sensor; Step 2: outputting a real-time attitude quaternion of the acquisition time by an inertial measurement unit fixed on a rigid base of the depth sensor module, the attitude quaternion representing the rotational relationship between the depth sensor coordinate system and a preset world coordinate system; Step 3: performing back projection on the original depth image according to the intrinsic matrix of the depth sensor to generate an uncorrected three-dimensional point cloud; Step 4: constructing a corresponding rotation matrix according to the attitude quaternion and calculating the inverse matrix thereof; Step 5: performing coordinate transformation on each point in the uncorrected three-dimensional point cloud using the inverse matrix of the rotation matrix to obtain a preliminary corrected point cloud; Step 6: performing depth projection geometric inverse mapping on each point in the preliminary corrected point cloud to calculate the projection angle of each point in the preliminary corrected point cloud relative to the optical axis in the sensor coordinate system, and solving the real depth according to the projection angle, and then updating the three-dimensional coordinates of each point in the preliminary corrected point cloud to generate a corrected point cloud, for the non-uniform compression effect of the depth value caused by the inclination of the sensor; Step 7: performing local geometric feature extraction on the corrected point cloud to screen out potential defect candidate points; Step 8: performing defect type discrimination on the local point cloud block based on the potential defect candidate points by a lightweight three-dimensional convolutional neural network model to output the defect category and spatial position.

2. The method of claim 1, wherein: In the depth projection geometric inverse mapping, the calculation of the cosine reciprocal of the projection angle is realized by using a lookup table method; in the initialization stage, an angle-cosine reciprocal mapping table is constructed in advance, and in the running stage, the corresponding cosine reciprocal approximate value is obtained by looking up and interpolating according to the calculated projection angle.

3. The method of claim 1, wherein, The local geometric feature extraction comprises: voxel grid downsampling of the corrected point cloud; constructing a nearest neighbor graph in each non-empty voxel; calculating the principal curvature, Gaussian curvature and average curvature of each point based on covariance matrix analysis; calculating the dot product mean of each point and its nearest neighbor normal vector, and if the mean is lower than a set threshold, it is marked as a potential defect candidate point; and calculating the local point density, and if the local point density is lower than a set proportion of the global average point density, it is marked as a sparse abnormal point.

4. The method of claim 1, wherein, The input of the lightweight three-dimensional convolutional neural network model is a three-dimensional voxel occupancy grid formed by normalizing the local point cloud block centered on each candidate point with a fixed radius; the network structure comprises a plurality of three-dimensional convolutional layers, pooling layers and fully connected layers in sequence, for outputting a defect category label.

5. The method of claim 1, wherein, The data acquisition of the depth sensor module and the inertial measurement unit is controlled by a central cooperative controller through a hardware synchronization trigger signal to realize the synchronization of the acquisition time of the two; the synchronization trigger signal is generated by a hardware timestamp synchronization unit realized by a field programmable gate array.

6. The method of claim 1, wherein, Further comprising an adaptive pose verification step: continuously monitor the angular velocity integral drift of the inertial measurement unit, if the drift exceeds a preset threshold, pause the point cloud correction, start the visual-inertial tightly coupled initialization algorithm, use the planar structure in the depth image to back-propagate the current attitude angle, and correct the zero offset parameters of the inertial measurement unit accordingly.

7. The method of claim 1, wherein, When there is a known reference plane in the application scene, enable the plane constraint optimization mode: after generating the corrected point cloud, automatically detect the dominant plane in the scene and fit its normal vector; if the angle between the normal vector and the reference axis of the world coordinate system is greater than the set tolerance, feedback an error signal to fine-tune the attitude quaternion, so that the dominant plane in the corrected point cloud satisfies the preset spatial geometric relationship.

8. A tilt-compensated depth point cloud correction and defect detection system, characterized in that, Comprise: a depth sensor module for acquiring an original depth image of the surface of the measured object; an inertial measurement unit fixedly installed on the rigid base of the depth sensor module for real-time output of attitude quaternion at the collection time; a central cooperative controller for synchronous reception of the original depth image and the attitude quaternion; a point cloud geometry correction engine for back-projection of the original depth image to generate an uncorrected point cloud according to the depth sensor intrinsic matrix; constructing a rotation matrix according to the attitude quaternion and calculating its inverse matrix; performing inverse rotation transformation on the uncorrected point cloud to obtain a preliminary corrected point cloud; and performing inverse depth projection geometry mapping on the preliminary corrected point cloud to calculate the real depth and update the point coordinates to generate a corrected point cloud; a surface feature analysis module for local geometric feature extraction and screening of potential defect candidate points from the corrected point cloud; a defect decision output unit with a built-in lightweight three-dimensional convolutional neural network model for defect type discrimination of local point cloud blocks of the potential defect candidate points and output of results.

9. The tilt-compensated depth point cloud correction and defect detection system of claim 8, wherein: The point cloud geometry correction engine calls a pre-stored angle-cosine reciprocal mapping table and obtains an approximate value of the projection angle cosine reciprocal by interpolation when performing the inverse depth projection geometry mapping.

10. The tilt-compensated depth point cloud correction and defect detection system of claim 8, wherein, The surface feature analysis module is configured to: down-sample the corrected point cloud in a voxel grid; construct a nearest neighbor graph in each non-empty voxel; calculate the principal curvature, Gaussian curvature and mean curvature of each point based on a covariance matrix; introduce a normal consistency constraint mechanism, and mark a point as a candidate point when the average dot product of the normal vectors of the point and its nearest neighbor points is below a set threshold; and calculate the local point density, and mark a sparse abnormal point when the local point density is below a set proportion of the global average density.

Images