A binocular structured light edge-preserving smoothing three-dimensional reconstruction method based on confidence guidance

Abstract

This invention discloses a confidence-guided binocular structured light edge-preserving smoothing 3D reconstruction method for welding inspection and machine vision measurement. The invention utilizes a three-frequency, four-step phase shift and an improved multi-frequency heterodyne algorithm to obtain a clean texture map and a modulation confidence map. Initial disparity is generated through sub-pixel bilinear scattering, and a workpiece mask is constructed based on the maximum connected component, filling only noise holes and preserving the true structure of the workpiece. A confidence-adaptive edge-preserving smoothing algorithm jointly guided by modulation, texture, and spatial distance is proposed, directly optimized in the 2D disparity map domain, replacing the time-consuming KD-Tree search and filtering in the traditional 3D point cloud domain. This invention can save 99.8% of filtering time, significantly reducing the overall process time while maintaining sub-pixel-level reconstruction accuracy, meeting the real-time and high-precision requirements of industrial weld inspection and precision component dimensional measurement.

Description

Technical Field

[0001] This invention belongs to the field of 3D measurement and machine vision technology, specifically relating to a binocular stereo vision 3D reconstruction method based on surface structured light projection, and particularly to a fast parallax smoothing and point cloud generation method with edge preservation characteristics. Background Technology

[0002] Binocular 3D reconstruction technology based on structured light is widely used in industrial inspection, reverse engineering, robot guidance and other fields. It mainly projects an coded grating onto the object being measured, while using a binocular camera to acquire images of deformed stripes and calculate the absolute phase to complete stereo matching. Then, the three-dimensional shape of the object is restored using the principle of triangulation.

[0003] In practical applications, due to factors such as uneven reflection of the measured surface, ambient light interference, and camera sensor noise, absolute phase maps are prone to local jumps and noise. When generating the original disparity map using stereo matching, numerous invalid holes and outliers exist. Traditional reconstruction methods often employ a reconstruction-then-filter approach, first generating a noisy, dense 3D point cloud, and then post-processing it using point cloud filtering algorithms, such as PCL library statistical filtering and radius filtering. The main drawbacks are: firstly, the point cloud filtering process involves spatial KD-Tree search, which is computationally complex and time-consuming, failing to meet the needs of industrial online inspection; secondly, the filtering process lacks texture and confidence guidance, easily misjudging workpiece edges and fine structures as noise and smoothing them away, resulting in loss of morphological details.

[0004] Therefore, how to abandon time-consuming 3D point cloud post-processing workflows, achieve efficient edge-preserving smoothing in the disparity map domain, accurately repair disparity noise holes without damaging the actual structure of the workpiece, and improve real-time processing while preserving the corner contact edges and detailed features of the workpiece, has become a key technical problem that urgently needs to be solved in the field of binocular structured light 3D reconstruction, while ensuring the accuracy of 3D reconstruction and preserving the corner contact edges and detailed features of the workpiece. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a confidence-guided binocular structured light edge-preserving and smoothing 3D reconstruction method to solve the problems of existing binocular structured light 3D reconstruction relying on post-processing of 3D point clouds such as filtering, denoising, and smoothing, which results in excessive time consumption and insufficient real-time performance. At the same time, there are technical problems such as the lack of effective guidance for parallax smoothing, which causes blurring of workpiece corner contact edges and loss of details, and the easy filling of real holes in the workpiece by parallax, which destroys the original shape.

[0006] To address the above problems, this invention proposes a confidence-guided binocular structured light edge-preserving and smoothing 3D reconstruction method, comprising the following:

[0007] Complete the binocular camera calibration and stereo correction. Using the distortion correction mapping table, perform stereo correction on the left and right absolute phase maps, clean texture maps, and modulation confidence maps to achieve epipolar alignment among the three.

[0008] Multi-frequency phase-shift phase calculations are performed on the structured light grating images acquired by the left and right cameras to obtain a continuous, untangled absolute phase map. Sparse matching point pairs between the left and right images are obtained through phase matching. The sub-pixel precision floating-point disparity values ​​obtained from the sparse matching points are then distributed to their four neighboring integer pixel nodes using bilinear weighted scattering to generate an initial dense disparity map.

[0009] Simultaneously, a clean texture map without stripe interference is extracted, and a structured light modulation confidence map is generated, providing a dual guideline of texture and confidence for subsequent parallax optimization.

[0010] The effective region is obtained by threshold screening based on the modulation confidence map, and the effective region is analyzed by connected component analysis to extract the largest connected component as the workpiece mask. Under the constraint of the workpiece mask, the neighborhood iterative hole filling operation is performed, which only fills the zero-value disparity pixels generated by reconstruction noise in the mask, while the real holes inherent in the workpiece itself that do not belong to the workpiece mask due to low modulation are not processed, so as to completely preserve the original structural features of the workpiece.

[0011] In the two-dimensional disparity map domain, a confidence-adaptive edge-preserving smoothing filter is used to optimize the disparity map. The convolution kernel of the filter is formed by multiplying and fusing the spatial distance weight, the texture-guided edge-preserving weight calculated based on the pure texture map, and the confidence suppression weight based on the modulation confidence map. The filtering process is only performed within the effective disparity region and does not spread to the background region.

[0012] The beneficial effects of this invention are: First, by moving the core optimization process forward to the two-dimensional disparity map domain, it directly replaces all the steps of three-dimensional point cloud post-processing, reducing filtering time by more than 99.8%, improving the real-time performance of three-dimensional reconstruction, and making it suitable for industrial online high-speed inspection scenarios.

[0013] Second, the confidence-adaptive edge-preserving smoothing algorithm guided by modulation and texture can better preserve the workpiece edge contour, bevel angle, and fine structure while suppressing parallax noise, thus solving the problems of edge blurring, lack of detail, and insufficient noise smoothing in traditional smoothing algorithms from the source.

[0014] Third, the reconstruction process achieves the purpose of repairing parallax holes by strictly constraining the workpiece mask. It does not fill real holes or damage the original structure of the workpiece. It takes into account short reconstruction time, high accuracy and high morphological fidelity, and is suitable for scenarios with extremely high requirements for structural integrity, such as weld inspection and precision parts measurement. Attached Figure Description

[0015] Figure 1 This is a flowchart of the algorithm of the present invention.

[0016] Figure 2 Point cloud data without any filtering processing

[0017] Figure 3 For point cloud data processed with a masked median-mean filter

[0018] Figure 4 Point cloud data processed using point cloud filtering and processing time

[0019] Figure 5 The point cloud data after filtering using this method and the processing time.

[0020] Figure 6 The image shows weld information processed using point cloud filtering (left) and the filtering method described in this paper (right). Detailed Implementation

[0021] This embodiment selects the corner weld positioning and bevel inspection before welding automotive parts as the application scenario. The test objects include two metal steel plates with a little oil and certain reflective properties to be welded. The main goal of reconstruction is to obtain the weld trajectory, steel plate flatness, bevel angle and other dimensional parameters before reconstruction, so as to provide the welding robot with information such as the arc starting position and posture.

[0022] The computer used in this embodiment is the Honor MagicBook Pro 2019 with integrated graphics, and the CPU model is AMD Ryzen 54600H. The hardware uses a 1440×1080 pixel CMOS industrial binocular camera (focal length 8mm) and a 1440×1080 pixel DLP projection optical engine, with the binoculars arranged horizontally and symmetrically, and the baseline distance is about 92mm. A checkerboard calibration board with 11×8 inner corner points and a square side length of 10mm is used.

[0023] Step 1: Perform binocular system calibration and stereo correction. Use a checkerboard or circular calibration board to calibrate the left and right cameras, obtaining the camera intrinsic parameter matrix. and distortion coefficient Then, the binoculars are calibrated to obtain the rotation matrix between the left and right cameras. Translation vector The Bouguet algorithm is used to calculate the stereo calibration mapping table. and This is used for epipolar alignment in subsequent images, and a disparity-depth mapping matrix for 3D reconstruction is also established. By pre-storing the correction parameters, unnecessary recalculation can be avoided.

[0024] Step 2: Calculate the absolute phase, modulation, and clean texture of the structured light. The projector uses a three-frequency, four-step phase-shifting method to project a pattern of a four-step phase-shifting grating with periods T1, T2, and T3 of 140, 134, and 129, respectively. For the frequency... The intensities of the four images captured by the camera can be expressed as:

[0025] By directly utilizing the orthogonal properties of four-step phase shifting, fringe modulation is canceled out through summation and averaging, resulting in a pure grayscale texture image of the object without any fringe interference. The formula is as follows:

[0026] Next, the wrap-around phase at each frequency is solved using the four-step phase shift formula. Subsequently, a phase expansion was performed using the multi-frequency heterodyne method to obtain an absolute phase map covering the entire field. and The improved three-frequency heterodyne method, which uses weighted fusion of three-frequency phases to complete phase expansion in this embodiment, is formulated as follows:

[0027] Simultaneously, the modulation index, reflecting the signal-to-noise ratio of the image, is calculated. Its formula is:

[0028] Finally, by taking the mean or minimum value of the three frequency modulation intensities, the initial confidence plot required for the guided filter is constructed as follows:

[0029] Step 3: Epipolar correction and sub-pixel disparity scattering matching with mask constraints. The left and right phase maps, clean texture map, and confidence map are simultaneously remapped using a stereo correction mapping table to ensure epipolar alignment. After finding corresponding points through left and right epipolar matching, disparity points with sub-pixel accuracy are calculated. .

[0030] A major improvement of this invention is the introduction of a bilinear scattering algorithm, which smoothly "scatters" the disparity value of a floating-point coordinate to the surrounding four integer pixel nodes according to bilinear weights, thus eliminating the mesh artifact problem that often occurs in 3D reconstruction.

[0031] The initial disparity generation formula, which is weighted cumulative normalization, is:

[0032] The scattering weights are as follows: ; ; ; .

[0033] Then, binarization is performed based on scattering weights and confidence lower bounds, and the absolute contour mask of the workpiece under test is generated using the maximum connected component extraction algorithm, clearing the background area to zero.

[0034] Under mask constraints, noise holes within the effective area of ​​the workpiece are filled through multiple neighborhood iterations, without filling actual holes in the workpiece such as weld grooves, thus preserving the original structure of the weld.

[0035] The mask-constrained neighborhood iterative hole-filling formula is:

[0036] Neighborhood weight The formula is executed only in the region where the mask is 1, and it is iterated multiple times.

[0037] Step 4: Confidence and texture-guided edge-preserving disparity smoothing. This step avoids the traditional time-consuming post-processing of 3D point cloud KD-Tree filtering, directly performing core smoothing optimization in the disparity map domain, which is the core innovation of this invention. For the original disparity map... To address noise and holes, a joint weighted filtering model for smoothing the disparity values ​​of the center pixel and its neighboring pixels is presented:

[0038] in For pixels The local neighborhood window, in this embodiment, has a window radius of 5; the normalization coefficient is... .

[0039] The mathematical definition of each weight in formula (1) is as follows:

[0040] Spatial distance weight It is based on the Euclidean distance between pixels and uses Gaussian kernel attenuation:

[0041] Spatial scale parameters in the embodiments Set to 5.0 to control the smoothing range.

[0042] Texture-guided edge protection weight This is also key to achieving edge preservation. Introducing the original grayscale image from the left camera. Calculate the center point With neighboring points Absolute difference in grayscale:

[0043] Therefore, texture weight is defined as: In the embodiment, the texture scale parameter is set to... .

[0044] when When it is large (i.e.) (Located on the other side of the object's edge) Approaching 0, thus cutting off the smooth kernel's propagation across the edge, and strictly preserving the sharp edges at depth discontinuities.

[0045] Confidence weight of regulation Based on the confidence map extracted in step 2, low-quality regions are suppressed:

[0046] in The modulation confidence weight for each pixel. This is the original modulation value corresponding to the pixel.

[0047] Pixels with lower modulation levels, such as those in shadows or overexposed areas, have lower phase resolution reliability and are assigned lower weights, effectively preventing abnormal noise from spreading to the surrounding area.

[0048] To avoid the performance overhead of repeatedly calculating the exponential function, this invention employs a pre-computed lookup table technique:

[0049] 1. Spatial weight: Pre-calculated radius All offsets within the window correspond Store in a one-dimensional lookup table ;

[0050] 2. Texture weight: Gray-scale difference Pre-calculate all possibilities Store in a one-dimensional lookup table ;

[0051] 3. Adjustment confidence weight: Multiplication instead of division.

[0052] Using the acceleration methods described above, filtering only requires table lookups and multiplications. Combined with OpenMP's multi-threaded parallel traversal of image pixels, the time required for disparity map smoothing can be reduced to the hundreds of milliseconds.

[0053] Step 5: Generate a 3D point cloud. Combine the smoothed disparity map with the mapping matrix obtained in Step 1, and use inverse perspective projection transformation to recover the 3D coordinates of the pixels. 3D coordinates satisfy:

[0054] After normalization, the true 3D point coordinates are obtained:

[0055] After implementation and verification, the confidence and texture-guided parallax smoothing method adopted in this embodiment saves about 99.8% of the computation time in the smoothing filtering part, while achieving the same effect as traditional point cloud filtering in terms of noise suppression and weld edge sharpness preservation.

[0056] The above description is merely one embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and technology of the present invention should be included within the protection scope of the present invention.

Claims

1. A confidence-guided binocular structured light edge-preserving smoothing three-dimensional reconstruction method, characterized in that, Includes the following steps: Step 1: Binocular camera calibration and stereo correction: Acquire images of a checkerboard or symmetrical circular calibration board, complete single-target calibration of the left and right cameras, complete binocular calibration based on the calibration corner points, obtain the camera intrinsic parameter matrix, distortion coefficients, binocular rotation matrix and translation matrix, and obtain the disparity depth mapping matrix Q and distortion correction mapping table through stereo correction. Step 2: Structured light phase calculation: A four-step phase shift algorithm is executed on the multi-frequency grating images acquired by the left and right cameras to calculate the three-frequency wrapping phase map. Simultaneously, a modulation confidence map and a shadow mask are generated. The three-frequency phases are weighted and fused by an improved multi-frequency heterodyne algorithm to complete the parallel phase expansion and obtain the continuous absolute phase maps of the left and right cameras. Step 3: Phase matching and initial disparity generation: Phase matching is performed on the left and right absolute phase maps after stereo correction to obtain sparse matching point pairs. The sparse disparity is distributed to each pixel of the image using the bilinear scattering method to generate an initial dense disparity map. Step 4: Confidence-guided edge-preserving parallax smoothing: Based on the modulation confidence, an effective mask is generated. The maximum connected workpiece region is extracted through connected component analysis to filter background noise. The neighborhood iteration method with mask constraints is used to fill parallax holes. Only the reconstructed noise zero-value holes in the effective area of ​​the workpiece are repaired. The real holes of the workpiece itself are excluded from the hole-filling range because of insufficient modulation and not belonging to the connected component of the workpiece, thus completely preserving the original structural features of the workpiece. Median filtering is applied to the disparity map after hole filling, and then confidence-adaptive edge-preserving smoothing guided by modulation and texture is performed. Disparity edge-preserving optimization is completed by fusion calculation of spatial weight, grayscale weight, and confidence weight. Step 5: 3D reconstruction and point cloud cleaning. The optimized disparity map is reprojected into 3D coordinates using the disparity depth mapping matrix Q. Invalid points with abnormal values ​​or exceeding the range are filtered out to obtain a clean 3D point cloud.

2. The binocular structured light edge-preserving smoothing 3D reconstruction method based on confidence guidance according to claim 1 is characterized in that the four-step phase shifting algorithm in step 2 uses parallel acceleration to calculate three-frequency wrapped phase row by row and normalizes the phase value to [0, 2 ], and the pixels with modulation lower than the preset threshold are regarded as shadows to generate a binary shadow mask.

3. The confidence-guided binocular structured light edge-preserving smooth 3D reconstruction method according to claim 1 is characterized as follows: in step 2, the improved multi-frequency heterodyne algorithm configures frequency parameters T1=140, T2=134, and T3=129, and traverses the image pixels in parallel loop, and then weighted fuses the three-frequency phases to complete phase unwrapping, thereby eliminating phase entanglement and obtaining a continuous absolute phase map.

4. The confidence-guided binocular structured light edge-preserving smooth 3D reconstruction method according to claim 1 is characterized as follows: in step 3, the bilinear scattering method decomposes the floating disparity points into four integer pixels in the neighborhood, then calculates and assigns weights according to the fractional part of the coordinates, and then accumulates the disparities according to the weights to unify the total weights. In this way, sparse matching points can be transformed into dense initial disparity maps.

5. The confidence-guided binocular structured light edge-preserving smooth 3D reconstruction method according to claim 1 is characterized as follows: in step 4, the effective mask is generated based on the modulation threshold. It extracts the maximum area region of the workpiece as the workpiece mask through connected component analysis. The disparity value of non-workpiece regions is forcibly cleared to eliminate background interference.

6. The confidence-guided binocular structured light edge-preserving smooth 3D reconstruction method according to claim 1 has the following features: in step 4, the neighborhood iterative hole-filling algorithm with mask constraints will only iteratively fill the zero-value disparity pixels generated by reconstruction noise within the workpiece mask, and will not handle the holes that exist in the workpiece itself.

7. The confidence-guided binocular structured light edge-preserving smoothing 3D reconstruction method according to claim 1 is characterized as follows, wherein the adaptive smoothing filter in step 4 specifically includes: Spatial weight calculation: Based on the Euclidean distance between neighboring pixels and the center pixel, a spatial Gaussian weight lookup table is constructed; Pure texture guided weight calculation: Based on the pure texture image of the left camera extracted in step 2, the brightness difference between the neighboring pixels and the center pixel is calculated, and the texture similarity weight is quickly obtained through a preset lookup table to achieve the corner contact edge preservation characteristic; Confidence weight calculation: Based on the modulation image obtained after multi-frequency heterodyne deconvolution, the modulation confidence weight of each pixel is calculated to suppress low-quality regions. Mask protection mechanism: The product of the above three weights is fused as the filter convolution kernel, and neighborhood iterative smoothing is only performed on the effective pixels within the mask, strictly prohibiting the disparity value from spreading to the background black hole region outside the mask.

8. The confidence-guided binocular structured light edge-preserving and smoothing 3D reconstruction method according to claim 1 is characterized as follows: in step 5, point cloud cleaning and culling are performed when the absolute value of the z-axis is greater than... Invalid points with empty, infinite, or all zero values ​​can be cleaned and saved as PCD or XYZ text format.

Images