3D Reconstruction Method Based on Multi-channel Polarization Binocular Image Fusion
By using a multi-channel polarized binocular image fusion method, the problems of point cloud data voids and mismatches in 3D reconstruction of highly reflective and weakly textured surfaces were solved, achieving efficient and accurate 3D reconstruction results.
Patent Information
- Application Number
- CN202311118525.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-01
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-09-01
AI Technical Summary
Existing binocular stereo vision technology is not suitable for 3D reconstruction of highly reflective and weakly textured surfaces, resulting in point cloud data gaps and mismatches.
A multi-channel polarization binocular image fusion method is adopted. Multi-channel polarization images are acquired through a polarization binocular camera, overexposed images are removed, camera calibration and epipolar correction are performed, outliers are detected and removed using mathematical statistics methods, and 3D reconstruction is performed by weighting the disparity data according to polarization characteristics and combining it with a disparity post-processing algorithm.
It achieves complete 3D reconstruction of highly reflective and weakly textured surfaces, improves the accuracy and completeness of point cloud data, and overcomes the shortcomings of binocular imaging technology.
Smart Images

Figure CN117496043B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, specifically to a three-dimensional reconstruction method based on multi-channel polarization binocular image fusion. Background Technology
[0002] Binocular stereo vision is a technology that uses two cameras at different spatial locations to acquire images of the same scene and calculates the absolute depth information of the scene based on the disparity between the two images. Compared to other 3D imaging methods such as photometric stereo, coded structured light, and laser line scanning, binocular stereo vision requires relatively simple equipment, does not require an active light source, can acquire absolute depth information of the scene, and has high point cloud generation efficiency. It has become a mainstream 3D imaging method and is widely used in fields such as face recognition, autonomous driving, industrial production, augmented reality, and reverse engineering. The main process of 3D imaging based on binocular stereo vision includes camera calibration, image acquisition, epipolar correction, stereo matching, and 3D reconstruction. Among these, stereo matching is the key and challenging aspect of binocular stereo vision. Stereo matching, also known as disparity estimation, measures the correlation between the pixel to be matched and candidate pixels by surface texture features, thereby selecting corresponding feature points and calculating the disparity of that point in the two images. The depth information is obtained using the principle of triangulation. Because the stereo matching algorithm requires the surface of the object to have strong texture features, existing binocular stereo vision technologies are not suitable for 3D reconstruction of highly reflective, weakly textured surfaces.
[0003] Polarization imaging is a novel optical imaging technique that, by integrating polarization filter chips into a camera and setting rotating polarizers in the optical path, can image reflected light from a scene with different polarization states, thereby acquiring the scene's polarization information. Since the polarization state of reflected light at a given point is closely related to the material, texture, and other surface properties of the micro-facet at that point, polarization imaging can significantly enhance the representation of detailed surface texture features. By exploring the intrinsic relationship between polarization information and surface texture, and utilizing the "strong light weakening, weak light enhancement" characteristics of polarization imaging, it is hoped that the challenge of binocular imaging technology being unable to adapt to the 3D reconstruction of highly reflective, weakly textured surfaces can be overcome.
[0004] Domestic research has already explored joint polarization imaging and binocular stereo vision technology. Chinese patent document CN111899290B, authorized in 2022, discloses a 3D reconstruction method combining joint polarization and binocular vision. This method spatially fits the initial polarization reconstruction result and the registered binocular depth map, uses the surface data from the polarization reconstruction to fill in the missing parts of the binocular depth map, and then uses the completed binocular depth map to correct the polarization reconstruction surface. This iterative correction process addresses the data loss in the binocular depth map. Compared to traditional point cloud optimization methods, such as algorithms for removing small connected components and median filtering smoothing algorithms, combining polarization methods to fill point cloud holes is more physically meaningful and stable. However, the aforementioned research does not consider the differences in imaging results across different polarization channels. For point cloud holes and mismatches on highly reflective, weakly textured surfaces in binocular stereo vision technology, the complementarity of binocular 3D reconstruction surfaces with different polarization channels holds promise for providing a more robust solution. Summary of the Invention
[0005] The purpose of this invention is to overcome the above-mentioned technical deficiencies and propose a three-dimensional reconstruction method based on multi-channel polarization binocular image fusion, thereby solving the technical problem that existing binocular stereo vision technology cannot be applied to the three-dimensional reconstruction of highly reflective and weakly textured surfaces.
[0006] To achieve the above-mentioned technical objectives, in a first aspect, the present invention provides a three-dimensional reconstruction method based on multi-channel polarization binocular image fusion, comprising the following steps:
[0007] A multi-channel polarized binocular image set of a target scene is acquired using a polarized binocular camera. The multi-channel polarized binocular image set includes a left-eye polarization intensity image set and a right-eye polarization intensity image set.
[0008] Remove polarization intensity images from the multi-channel polarized binocular image set whose overexposed area exceeds a set threshold.
[0009] The left eye polarization intensity image set and the right eye polarization intensity image set are arranged and combined to obtain multiple pairs of binocular images with different polarization channel combinations. The camera is calibrated based on the polarization binocular images. Epipolar correction and stereo matching are performed on each polarization binocular image to obtain the disparity data cube.
[0010] Outliers in the disparity data column are detected and removed using mathematical statistics methods. Based on the polarization characteristics of the data source, different weights are assigned to the remaining disparity data in the disparity data column, and the weighted average result is used as the fused disparity value.
[0011] The fused disparity value is calculated pixel by pixel to obtain a fused disparity image based on a multi-channel polarized binocular image;
[0012] The fused disparity image is post-processed using a disparity post-processing algorithm to obtain a final disparity image. Based on the final disparity image, 3D reconstruction is performed to obtain 3D point cloud data of the target scene.
[0013] Compared with existing technologies, the beneficial effects of the 3D reconstruction method based on multi-channel polarization binocular image fusion provided by this invention include:
[0014] The present invention discloses a 3D reconstruction method based on multi-channel polarized binocular image fusion. First, a multi-channel polarized binocular image set is acquired using a polarized binocular camera. Overexposed polarized images with an area exceeding a set threshold are removed from the image set. The left and right polarized image sets are arranged and combined to obtain stereo matching disparity images with different polarization channel combinations, thereby constructing a 3D data cube of image disparity. Through this method, each image pixel can obtain multiple disparity values, corresponding to the results of stereo matching calculations of different polarization channel image combinations. Outliers in the disparity sequence are detected and removed using mathematical statistics methods. Different weights are assigned to the remaining disparity data according to the polarization characteristics of the data source, and the weighted average is used as the fused disparity value. The fused disparity value is calculated pixel-by-pixel to obtain a fused disparity image based on the multi-channel polarized binocular images. A series of disparity optimization algorithms are used to optimize the disparity image. Finally, 3D reconstruction is performed based on the final disparity image to obtain scene point cloud data.
[0015] To overcome the problem of point cloud data gaps caused by highly reflective and weakly textured surfaces in binocular stereo vision technology, this invention fully considers the complementarity between binocular 3D reconstruction surfaces with different polarization channels. It extracts the disparity value series obtained from stereo matching of multi-channel polarized binocular images pixel by pixel, eliminates mismatched disparities based on mathematical statistics, and weights and calculates the fused disparity value, thereby obtaining a complete and accurate 3D reconstruction result. By exploring the intrinsic relationship between polarization information and surface texture, and utilizing the "strong light weakening, weak light enhancement" characteristics of polarization imaging technology, it is expected to overcome the challenge that binocular imaging technology cannot adapt to 3D reconstruction of highly reflective and weakly textured surfaces.
[0016] According to some embodiments of the present invention, acquiring a multi-channel polarized binocular image set of a target scene using a polarized binocular camera includes the following steps:
[0017] The left eye of the polarization binocular camera acquires a set of polarization intensity images of the target scene at more than three different polarization angles, and the right eye of the polarization binocular camera acquires a set of polarization intensity images of the target scene at more than three different polarization angles.
[0018] Based on the left and right eye polarization intensity image sets, the left and right polarization degree images, left and right polarization azimuth angle images, and left and right non-polarization intensity images are calculated. The intensity value i at pixel (u,v) of the acquired polarization intensity image is related to the polarization angle θ. j Relevant, satisfying the formula:
[0019]
[0020] In the formula: S0, S1, and S2 are Stokes parameters. By acquiring more than three polarization images with different polarization angles, the intensity value i can be solved by constructing a linear equation system according to formula (1); S0, S1, S2, polarization degree ρ, and polarization azimuth angle are also included. non-polarization intensity i un Relational satisfaction formula:
[0021] i un (u,v)=0.5·S0(u,v).
[0022] According to some embodiments of the present invention, the left-eye polarization intensity image set and the right-eye polarization intensity image set are arranged and combined to obtain multiple pairs of polarization binocular images with different polarization channel combinations, including the following steps:
[0023] The left eye polarization intensity image set has N1 remaining images, and the right eye polarization intensity image set has N2 remaining images. The left eye polarization intensity image set and the right eye polarization intensity image set are arranged and combined to obtain N1×N2 pairs of binocular images with different polarization channel combinations. Adding the left and right polarization degree images, left and right polarization azimuth angle images, and left and right non-polarization intensity images, a total of N1×N2+3 pairs of polarization binocular images are obtained.
[0024] According to some embodiments of the present invention, camera calibration is performed on the polarization binocular camera, and epipolar correction and stereo matching are performed on each binocular image to obtain disparity data cube, including the following steps:
[0025] By photographing single-plane checkerboard patterns in different poses, the intrinsic parameter matrix, rotation matrix, translation matrix, and distortion coefficients of the polarization binocular camera are calculated to perform camera calibration.
[0026] The imaging planes of the left and right cameras are rotated by half to control the error caused by the reprojection of the left and right images, thereby ensuring that the same feature points are located in the same row of the left and right images, so as to perform epipolar correction on the binocular images.
[0027] The correlation between candidate pixels and pixels to be matched is measured, the corresponding feature points are selected, and the disparity of the corresponding feature points in the left and right eye images is calculated for stereo matching to obtain the disparity data cube.
[0028] According to some embodiments of the present invention, mathematical statistics methods are used to detect and remove outliers in a disparity data column. An outlier is defined as an element whose converted median absolute deviation (CMAD) differs from the median of the disparity data column by more than three times. The CMAD expression is as follows:
[0029]
[0030] Where: D is the disparity data column, D i This is the i-th data point in the disparity data column.
[0031] According to some embodiments of the present invention, different weights are assigned to the remaining disparity data in the disparity data column based on the polarization characteristics of the data source, and the weighted average result is used as the fused disparity value, including the following steps:
[0032] The remaining disparity data in the disparity data column are assigned different weights according to the polarization characteristics of the data source. The disparity calculated from the binocular polarization intensity image has a weight of W1, and the disparity calculated from the binocular polarization degree image, the binocular polarization azimuth angle image, and the binocular non-polarization intensity image has a weight of W2.
[0033] Fusion disparity value d b The result of the weighted average of the disparity data column D and the weight data column W is expressed as:
[0034]
[0035] According to some embodiments of the present invention, a disparity post-processing algorithm is used to perform disparity post-processing on the fused disparity image to obtain a final disparity image, including the following steps:
[0036] The disparity post-processing algorithm includes: small connected component removal algorithm, hole filling algorithm, and median filtering algorithm;
[0037] The small connected component removal algorithm includes: determining, pixel by pixel, whether the absolute disparity difference between a point and its neighboring pixels is less than a preset disparity difference threshold T. d If so, then the point is determined to be connected to its neighboring pixels. If the number of connected pixels in the connected region obtained by the above method is less than a preset threshold T, then... n If so, the connected region is determined to be a small connected region, and the disparity value of the connected region is invalid;
[0038] The hole-filling algorithm includes: determining the validity of the disparity value of each pixel; if invalid, searching for the closest valid pixel in the upward, downward, leftward, and rightward directions, with the corresponding disparity values denoted as d. left d up d right d down Based on the pixel distance between the effective pixels and the pixels to be filled, a Gaussian function is used to convert the distance into weights for disparity values, denoted as w. left w up w right w down The disparity value of the target pixel satisfies the following formula:
[0039]
[0040] The median filtering algorithm process includes: establishing an N×N window centered on the pixel, with the median of all pixel disparity values within the window as the new disparity value, and performing pixel-by-pixel calculations to obtain a new disparity image, which is the final disparity image.
[0041] According to some embodiments of the present invention, three-dimensional point cloud data of a target scene is obtained by three-dimensional reconstruction based on the final parallax image, including the following steps:
[0042] Given the disparity value d at pixel (u,v) in the left image, the spatial coordinates (X,Y,Z) in the world coordinate system satisfy the following equation:
[0043]
[0044] In the formula: c xl c yl Let c be the coordinates of the left camera principal point in the left image coordinate system. xr c yr Let u, v, and c be the coordinates of the principal point of the right camera in the right image coordinate system; f be the lens focal length; b be the distance between the optical centers of the left and right cameras; where u, v, and c are the coordinates of the principal point of the right camera in the right image coordinate system. xl c yl c xr c yr The units for , d, and f are pixels, while the units for X, Y, Z, and b are mm.
[0045] In a second aspect, the present invention provides a three-dimensional reconstruction system based on multi-channel polarization binocular image fusion, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the three-dimensional reconstruction method based on multi-channel polarization binocular image fusion as described in any one of the first aspects.
[0046] Thirdly, the present invention provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a three-dimensional reconstruction method based on multi-channel polarization binocular image fusion as described in any one of the first aspects.
[0047] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0048] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein the abstract drawings are to be completely consistent with one of the drawings in the specification:
[0049] Figure 1 This is a flowchart of the three-dimensional reconstruction method based on multi-channel polarization binocular image fusion according to the present invention;
[0050] Figure 2 This is a flowchart of the algorithm for arranging and combining left and right polarization intensity image sets, epipolar correction, and stereo matching according to the present invention.
[0051] Figure 3 This is a diagram of an experimental apparatus according to an embodiment of the present invention;
[0052] Figure 4 This is a multi-channel polarization binocular image set according to an embodiment of the present invention;
[0053] Figure 5 These are left and right eye polarization degree images, polarization azimuth angle images, and non-polarization intensity images, according to an embodiment of the present invention.
[0054] Figure 6 This is a schematic diagram illustrating the determination of disparity data cube and outlier values according to an embodiment of the present invention;
[0055] Figure 7 This is a curve showing the change of the weights of the effective disparity data to be fused as a function of the distance between the data and the pixels of the hole to be filled, according to an embodiment of the present invention.
[0056] Figure 8 This is a process of processing a numerical matrix of a certain region using a small connected component elimination algorithm and a hole filling algorithm, according to an embodiment of the present invention.
[0057] Figure 9 This is a comparison image of the three-dimensional reconstruction results based on the fusion of intensity image, single-channel polarization image and multi-channel polarization image according to an embodiment of the present invention;
[0058] Figure 10This is a three-dimensional reconstruction result of point cloud data based on plane fitting using the RANSAC algorithm according to an embodiment of the present invention;
[0059] Figure 11 This is a comparison diagram of target depth curves at several cross sections according to an embodiment of the present invention. Detailed Implementation
[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0061] It should be noted that although functional modules are divided in the system diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0062] The embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0063] Reference Figure 1 , Figure 1 This is a flowchart of a three-dimensional reconstruction method based on multi-channel polarization binocular image fusion according to an embodiment of the present invention; the three-dimensional reconstruction method based on multi-channel polarization binocular image fusion includes, but is not limited to, steps 110 to 160.
[0064] Step 110, Acquire multi-channel polarization binocular images: Use a polarization binocular camera to acquire a set of multi-channel polarization binocular images of the target scene. The set of multi-channel polarization binocular images includes a set of polarization intensity images of the left eye and a set of polarization intensity images of the right eye.
[0065] Step 120, Remove overexposed polarization images from the set: Remove polarization intensity images from the multi-channel polarization binocular image set whose overexposed area is greater than a set threshold;
[0066] Step 130: Arrange and combine the left and right polarization images to obtain multiple parallax images: Arrange and combine the left eye polarization intensity image set and the right eye polarization intensity image set to obtain multiple pairs of binocular images with different polarization channel combinations. Perform camera calibration based on the polarization binocular images, and perform epipolar correction and stereo matching on each pair of polarization binocular images to obtain the parallax data cube.
[0067] Step 140, Multi-disparity image fusion processing based on mathematical statistics: outliers in the disparity data column are detected and removed using mathematical statistics methods. Different weights are assigned to the remaining disparity data in the disparity data column according to the polarization characteristics of the data source. The weighted average result is used as the fused disparity value.
[0068] Step 150, post-processing of disparity image: calculate the fused disparity value pixel by pixel to obtain the fused disparity image based on the multi-channel polarized binocular image;
[0069] Step 160, 3D Reconstruction: The fused parallax image is post-processed using a parallax post-processing algorithm to obtain the final parallax image. Based on the final parallax image, 3D reconstruction is performed to obtain the 3D point cloud data of the target scene.
[0070] In one embodiment, the 3D reconstruction method based on multi-channel polarization binocular image fusion includes the following steps: acquiring a set of multi-channel polarization binocular images of the target scene using a polarization binocular camera, the set of multi-channel polarization binocular images including a set of left-eye polarization intensity images and a set of right-eye polarization intensity images; removing polarization intensity images from the multi-channel polarization binocular image set whose overexposed area is greater than a set threshold; arranging and combining the left-eye polarization intensity image set and the right-eye polarization intensity image set to obtain multiple pairs of polarization binocular images with different polarization channel combinations; and performing camera calibration based on the polarization binocular images, step by step... Epipolar correction and stereo matching are performed on a pair of binocular images to obtain a cube of disparity data. Outliers in the disparity data column are detected and removed using mathematical statistics methods. Different weights are assigned to the remaining disparity data in the disparity data column according to the polarization characteristics of the data source, and the weighted average result is used as the fused disparity value. The fused disparity value is calculated pixel by pixel to obtain a fused disparity image based on multi-channel polarized binocular images. A disparity post-processing algorithm is used to perform disparity post-processing on the fused disparity image to obtain the final disparity image. 3D reconstruction is performed based on the final disparity image to obtain the 3D point cloud data of the target scene.
[0071] The process of acquiring a multi-channel polarized binocular image set of a target scene using a polarized binocular camera includes the following steps: acquiring a set of left-eye polarization intensity images of the target scene at three or more different polarization angles using the left eye of the polarized binocular camera, and acquiring a set of right-eye polarization intensity images of the target scene at three or more different polarization angles using the right eye of the polarized binocular camera; for example, acquiring polarization intensity images of the scene at 0°, 30°, 60°, 90°, 120°, and 150° using the left and right eyes respectively.
[0072] Based on the left and right eye polarization intensity image sets, the left and right polarization degree images, left and right polarization azimuth angle images, and left and right unpolarized intensity images are calculated. The intensity value i at pixel (u,v) of the acquired polarization intensity image is related to the polarization angle θ. j Relevant, satisfying the formula:
[0073]
[0074] In the formula: S0, S1, and S2 are Stokes parameters. By acquiring more than three polarization images with different polarization angles, the intensity value i can be solved by constructing a linear equation system according to formula (1); S0, S1, S2, polarization degree ρ, and polarization azimuth angle are also included. non-polarization intensity i un Relational satisfaction formula:
[0075] i un (u,v)=0.5·S0(u,v).
[0076] The polarization intensity images in the multi-channel polarized binocular image set with overexposed areas exceeding a set threshold are removed. The threshold can be a specific number of pixels or defined as the proportion of tolerable overexposed pixels in the entire image. In this embodiment of the invention, polarization intensity images with overexposed pixels exceeding 22% of the total number of pixels in the foreground mask are removed.
[0077] The left and right eye polarization intensity image sets are arranged and combined to obtain multiple pairs of binocular images with different polarization channel combinations, including the following steps:
[0078] Reference Figure 2 , Figure 2 This is an algorithm flowchart for arranging and combining left and right polarization intensity image sets, epipolar correction, and stereo matching according to an embodiment of the present invention.
[0079] Step 131: After removing overexposed pixels, the left eye polarization intensity image set has N1 images remaining, and the right eye polarization intensity image set has N2 images remaining. The left eye polarization intensity image set and the right eye polarization intensity image set are arranged and combined to obtain N1×N2 pairs of polarized binocular images with different polarization channel combinations. Adding the left and right polarization degree images, left and right polarization azimuth angle images, and left and right non-polarization intensity images, a total of N1×N2+3 pairs of polarized binocular images are obtained.
[0080] Camera calibration is performed based on polarized binocular images. Epipolar correction and stereo matching are performed on each pair of binocular images to obtain disparity data. The steps include:
[0081] By capturing single-plane checkerboard patterns in different poses, the intrinsic parameter matrix, rotation matrix, translation matrix, and distortion coefficients of the polarization camera are calculated, so as to calibrate the camera based on the polarization binocular images. The camera calibration method is Zhang Zhengyou calibration method, but other methods such as Tsai two-step method and layered stepwise self-calibration can also be used.
[0082] Step 132: Rotate the imaging planes of the left and right cameras by half each to control the error caused by the reprojection of the left and right images, so that the corresponding feature points are located in the same row of the left and right images, so as to perform epipolar correction on the binocular images; the epipolar correction method is the Bouguet method, or other epipolar correction methods such as Fusiello can also be used.
[0083] Step 133: Measure the correlation between candidate pixels and pixels to be matched, select corresponding feature points, and calculate the disparity of the corresponding feature points in the left and right eye images for stereo matching to obtain a disparity data column; the stereo matching method is the SGM algorithm, but other stereo matching algorithms such as BM, AD-Census, and PatchMatch can also be used.
[0084] Outliers in the disparity data column were detected and removed using mathematical statistics methods. Outliers were defined as elements whose converted median absolute deviation (CMAD) differed from the median of the disparity data column by more than three times. The CMAD expression is as follows:
[0085]
[0086] Where: D is the disparity data column, D i This is the i-th data point in the disparity data column.
[0087] Based on the polarization characteristics of the data source, different weights are assigned to the remaining disparity data in the disparity data column, and the weighted average result is used as the fused disparity value, including the following steps:
[0088] Based on the polarization characteristics of the data source, the remaining disparity data in the disparity data column are assigned different weights. The disparity calculated from the binocular polarization intensity image has a weight of W1, and the disparity calculated from the binocular polarization degree image, the binocular polarization azimuth angle image, and the binocular non-polarization intensity image has a weight of W2.
[0089] Fusion disparity value d b The expression for the weighted average of the disparity data column D and the weight data column W is:
[0090]
[0091] The final disparity image is obtained by performing disparity post-processing on the fused disparity image using a disparity post-processing algorithm, including the following steps:
[0092] Parallax post-processing algorithms include: small connected component removal algorithm, hole filling algorithm, and median filtering algorithm;
[0093] The small connected component removal algorithm process includes: determining, pixel by pixel, whether the absolute disparity difference between a point and its neighboring pixels is less than a preset disparity difference threshold T. dIf so, then the point is determined to be connected to its neighboring pixels. If the number of connected pixels in the connected region obtained by the above method is less than a preset threshold T, then... n If the connected region is small connected region, then the disparity value of the connected region is invalid.
[0094] The hole-filling algorithm process includes: determining the validity of the disparity value of each pixel; if invalid, searching for the closest valid pixel in the up, down, left, and right directions, with the corresponding disparity values denoted as d. left d up d right d down Based on the pixel distance between the effective pixels and the pixels to be filled, a Gaussian function is used to convert the distance into weights for disparity values, denoted as w. left w up w right w down The disparity value of the target pixel satisfies the following formula:
[0095]
[0096] The median filtering algorithm process includes: establishing an N×N window centered on the pixel, with the median of all pixel disparity values within the window as the new disparity value, and performing pixel-by-pixel calculations to obtain a new disparity image, which is the final disparity image.
[0097] The 3D point cloud data of the target scene is obtained by 3D reconstruction based on the final disparity image, including the following steps:
[0098] Given the disparity value d at pixel (u,v) in the left image, the spatial coordinates (X,Y,Z) in the world coordinate system satisfy the following equation:
[0099]
[0100] In the formula: c xl c yl Let c be the coordinates of the left camera principal point in the left image coordinate system. xr c yr Let u, v, and c be the coordinates of the principal point of the right camera in the right image coordinate system; f be the lens focal length; b be the distance between the optical centers of the left and right cameras; where u, v, and c are the coordinates of the principal point of the right camera in the right image coordinate system. xl c yl c xr c yr The units for , d, and f are pixels, while the units for X, Y, Z, and b are mm.
[0101] Reference Figures 3 to 11 , Figure 3 This is a diagram of an experimental apparatus according to an embodiment of the present invention; Figure 4This is a multi-channel polarization binocular image set according to an embodiment of the present invention; Figure 5 These are left and right eye polarization degree images, polarization azimuth angle images, and non-polarization intensity images, according to an embodiment of the present invention. Figure 6 This is a schematic diagram of a disparity data column and outlier determination according to an embodiment of the present invention; Figure 7 This is a curve showing the change of the weights of the effective disparity data to be fused as a function of the distance between the data and the pixels of the hole to be filled, according to an embodiment of the present invention. Figure 8 This is a process of processing a numerical matrix of a certain region using a small connected component elimination algorithm and a hole filling algorithm, according to an embodiment of the present invention. Figure 9 This is a comparison image of the three-dimensional reconstruction results based on the fusion of intensity image, single-channel polarization image and multi-channel polarization image according to an embodiment of the present invention; Figure 10 This is a three-dimensional reconstruction result of point cloud data based on plane fitting using the RANSAC algorithm according to an embodiment of the present invention; Figure 11 This is a comparison diagram of target depth curves at several cross sections according to an embodiment of the present invention.
[0102] To overcome the problem of point cloud data voids caused by highly reflective and weakly textured surfaces in binocular stereo vision technology, this invention fully considers the complementarity between binocular 3D reconstruction surfaces with different polarization channels and proposes a 3D reconstruction method based on multi-channel polarization binocular image fusion. By extracting the stereo matching disparity value series of multi-channel polarization binocular images pixel by pixel, and using mathematical statistics to eliminate mismatched disparities and weighted calculation of the fused disparity value, a 3D reconstruction result with complete surface and accurate shape is obtained.
[0103] In one embodiment, the 3D imaging target is an aluminum alloy workpiece, which is composed of two smooth beveled surfaces and has high reflectivity and weak texture. The binocular camera used is constructed from two Daheng Image ME2C-503-23GM industrial digital cameras. Two rotatable FLP25-VIS-M thin-film linear polarizers are placed in front of the binocular camera to form a polarized binocular camera. A flat panel light source is used as the unpolarized illumination source to adjust the ambient light brightness. The experimental setup is as follows: Figure 3 As shown. The three-dimensional reconstruction of this aluminum alloy workpiece specifically includes the following steps:
[0104] Step 1: Acquire a multi-channel polarized stereo image set of the scene using a polarized stereo camera. The specific implementation includes the following sub-steps:
[0105] Step 1.1: Using a polarization binocular camera, acquire polarization intensity images of the scene at 0°, 30°, 60°, 90°, 120°, and 150° with the left and right eyes respectively. The polarization binocular image dataset is as follows: Figure 4 As shown, the polarization camera is in the form of a rotatable polarizer mounted in front of the camera lens;
[0106] Step 1.2: Based on the set of left and right multi-channel polarization intensity images, calculate the left and right polarization degree images, left and right polarization azimuth angle images, and left and right unpolarized intensity images. The intensity value i at pixel (u, v) of the acquired polarization intensity image is related to the polarization angle θ. j Regarding, satisfying equation (1):
[0107]
[0108] In the formula: S0, S1, and S2 are Stokes parameters, which can be obtained by acquiring more than three polarization images with different polarization angles and constructing a linear equation system according to equation (1); S0, S1, S2, polarization degree ρ, and polarization azimuth angle are also included. non-polarization intensity i un The relation satisfies equation (2):
[0109] i un (u,v)=0.5·S0(u,v) (2)
[0110] The calculated left and right polarization degree images, left and right polarization azimuth angle images, and left and right unpolarized intensity images are as follows: Figure 5 As shown, they will also undergo stereo matching, and the resulting disparity values will be used for disparity fusion.
[0111] Step 2: Remove polarization intensity images from the multi-channel polarized binocular image set whose overexposed area is greater than a set threshold. In this embodiment of the invention, polarization intensity images whose overexposed pixel count exceeds 22% of the total number of pixels in the foreground mask are removed. The exposure of polarization intensity images in the left and right channels is shown in Table 1.
[0112] Table 1. Exposure ratio of polarization intensity images of the left and right channels in the embodiments of the present invention.
[0113] 0° 30° 60° 90° 120° 150° left eye 26.3% 17.7% 10.5% 11.9% 21.3% 28.4% right eye 23.4% 19.2% 12.9% 11.7% 17.1% 22.4%
[0114] Step 3: Arrange and combine the filtered left and right polarization intensity images. The left eye polarization intensity image set has 4 images remaining, and the right eye polarization intensity image set has 4 images remaining, resulting in 16 pairs of binocular images with different polarization channel combinations. Add the left and right polarization degree images, left and right polarization azimuth angle images, and left and right non-polarization intensity images, for a total of 19 pairs of binocular images. After camera calibration, perform epipolar correction and stereo matching on each pair of images to obtain the disparity data column.
[0115] Step 3.1: The camera calibration method is Zhang Zhengyou's calibration method. By shooting single-plane checkerboard patterns in different poses, the intrinsic parameter matrix, rotation matrix, translation matrix and distortion coefficient of the binocular camera are calculated, as shown in Table 2.
[0116] Table 2 Calibration parameters of the binocular camera in the embodiments of the present invention
[0117]
[0118] Step 3.2: The epipolar correction method is the Bouguet method, which rotates the imaging planes of the left and right cameras by half each, so that the error caused by the reprojection of the left and right images is minimized, thereby realizing that the same feature points are located in the same row of the two images;
[0119] Step 3.3: The stereo matching method is the semi-global matching (SGM) algorithm, which selects corresponding feature points by measuring the correlation between candidate pixels and pixels to be matched, and calculates the disparity of the point in the two images;
[0120] Step 4: Based on the disparity data column obtained in Step 3, each pixel has 19 disparity values. Outliers in the disparity data column are detected and removed using mathematical statistics methods. Different weights are assigned to the remaining disparity data according to the polarization characteristics of the data source. The weighted average of these weighted averages is used as the fused disparity value. A schematic diagram of the disparity data column and outlier determination is shown below. Figure 6 As shown;
[0121] Step 4.1: Outliers are defined as elements of the Converted Median Absolute Deviation (CMAD) that differ from the median by more than 3 times, and the CMAD satisfies equation (3):
[0122]
[0123] Where: D is the disparity data column, D i This is the i-th data point in the disparity data column.
[0124] Step 4.2: The disparity data in the disparity data column are assigned different weights according to the polarization characteristics of the data source. The disparity calculated from the binocular polarization intensity image has a weight of 1, and the disparity calculated from the binocular polarization degree image, the binocular polarization azimuth angle image, and the binocular non-polarization intensity image has a weight of 2.
[0125] Step 4.3: Fuse disparity values d b The result of the weighted average of the disparity data column D and the weight data column W satisfies equation (4):
[0126]
[0127] To better explain the algorithm flow in step 4, let's take a pixel in the image as an example. Its disparity sequence, outlier determination, fusion weight, and fusion disparity are shown in Table 3.
[0128] Table 3. Disparity sequence, outlier determination, fusion weight, and fusion disparity value for a specific pixel (unit: pixel)
[0129]
[0130]
[0131] As can be seen from the data in Table 3, some disparity values obtained from stereo matching of left and right polarized images are invalid (nan), such as numbers 5, 10, and 14, or their values deviate significantly from the disparity values calculated from other combinations, such as numbers 9, 13, and 19. Therefore, using only single-channel polarized binocular images for stereo matching can easily lead to data holes or noise. However, according to the method proposed in this invention, by obtaining a disparity data series based on multi-channel polarized binocular images and removing invalid and outlier values through mathematical statistics, more robust and accurate disparity data can be obtained.
[0132] Step 5: Calculate the fused disparity value pixel by pixel to obtain the fused disparity image based on the multi-channel polarized binocular image.
[0133] Step 6: The fused disparity image is further processed using a disparity post-processing algorithm, including small connected component removal, hole filling, median filtering, etc., and the scene's 3D point cloud data is obtained by 3D reconstruction based on the final disparity image.
[0134] Step 6.1: The small connected component elimination algorithm proceeds as follows: for each pixel, determine whether the absolute disparity difference between a point and its neighboring pixels is less than a preset disparity difference threshold T. d If so, then the point is determined to be connected to its neighboring pixels. If the number of connected pixels in the connected region obtained by the above method is less than a preset threshold T, then... n If the region is determined to be a small connected region, the disparity value of the region is invalid. In this embodiment of the invention, T is set. d Set T to 1. n The value is 100, and the range of neighboring pixels is defined as a four-connected region;
[0135] Step 6.2: The hole-filling algorithm process is as follows: For each pixel, determine whether the disparity value of that pixel is valid. If invalid, search for the closest valid pixel in the upward, downward, left, and right directions respectively, and denot the corresponding disparity values as d. left d up d right d down Based on the pixel distance between the effective pixels and the pixels to be filled, a Gaussian function is used to convert the distance into weights for disparity values, denoted as w. left w up w right w downThe disparity value of the target pixel satisfies equation (5):
[0136]
[0137] The relationship between the weights of the effective pixel disparity values and their pixel spacing with the pixels to be filled satisfies equation (6):
[0138]
[0139] The curves showing the change of weights with pixel spacing for different values of σ are as follows: Figure 7 As shown, in this embodiment of the invention, σ is 12. To better explain the algorithm flow of steps 6.1 and 6.2, Figure 8 The image shows the changes in a 6×6 region of a parallax image after processing with small connected component removal and hole filling algorithms. It can be seen that the parallax image is smoother after the parallax post-processing algorithm of this invention.
[0140] Step 6.3, the median filtering algorithm process is as follows: an N×N window is established with the pixel as the center, the median value of all pixel disparity values in the window is the new disparity value, and a new disparity image is obtained by performing the operation pixel by pixel in the above manner. In this embodiment of the invention, N is 5.
[0141] Step 6.4, the 3D reconstruction algorithm flow is as follows: if the disparity value d at pixel (u,v) in the left image is known, then its spatial coordinates (X,Y,Z) in the world coordinate system satisfy equation (7):
[0142]
[0143] In the formula: c xl c yl Let c be the coordinates of the left camera principal point in the left image coordinate system. xr c yr Here, f represents the coordinates of the principal point of the right camera in the right image coordinate system; f is the lens focal length; b is the distance between the optical centers of the left and right cameras; in this embodiment of the invention, c xl For 1210.7 pixels, c yl For 1180.8 pixels, c xr It is 1271.9 pixels, c yr The f is 1162.6 pixels, the f is 4543.9 pixels, and the b is 50 mm.
[0144] The above steps yield a 3D reconstruction result based on multi-channel polarization binocular image fusion. This result is then compared with 3D reconstruction results based on intensity binocular images and single-channel polarization binocular images. Figure 9As shown, due to the presence of large areas of overexposed pixels in the intensity image, accurate disparity values cannot be obtained through stereo matching, resulting in significant distortion of the 3D reconstructed surface. Both methods based on single-channel polarized binocular images and methods based on multi-channel polarized binocular image fusion can reconstruct approximately accurate 3D surfaces. However, the 3D reconstruction results based on single-channel polarized binocular images have mismatched regions, as shown in the dashed box in the figure. The 3D reconstruction method based on multi-channel polarized binocular image fusion proposed in this paper fully utilizes the complementary information of different polarization channels, thereby restoring the surface morphology of the target more completely.
[0145] To quantitatively evaluate the advantages of the proposed 3D reconstruction method, the RANSAC algorithm was used to perform planar fitting on the 3D point cloud data obtained from multi-channel polarization image fusion. The resulting 3D reconstruction was used as a benchmark for comparison with other methods. The 3D reconstruction results fitted by the RANSAC algorithm are shown below. Figure 10 As shown.
[0146] The mean square error (MSE) was introduced as an evaluation metric. The mean square errors of point cloud data reconstructed from intensity images, point cloud data reconstructed from single-channel polarization images, and point cloud data reconstructed from multi-channel polarization image fusion compared with the benchmark are shown in Table 4.
[0147] Table 4. MSE (unit: mm) for different reconstruction methods
[0148]
[0149] A 3D reconstructed surface based on multi-channel polarization image fusion can be obtained, with its MSE improved by 75.3% and 1896% compared to the 3D reconstructed surfaces based on single-channel polarization images and intensity images, respectively.
[0150] Figure 11 The images show the depth curves of the 3D reconstructed surface at several cross-sections obtained based on intensity images, single-channel polarization images, multi-channel polarization image fusion, and plane fitting methods. It can be seen that the depth curve of the 3D reconstructed surface based on intensity images at the shown cross-sections differs significantly from the fitted curve. The depth curve of the 3D reconstructed surface based on single-channel polarization images at row=1100 has a region with a large deviation from the fitted curve. The depth curve of the 3D reconstructed surface based on multi-channel polarization image fusion at the shown cross-sections is always close to the fitted curve.
[0151] In summary, the method proposed in this invention, based on binocular stereo vision, fully utilizes the information differences of different polarization channels to achieve robust 3D reconstruction of highly reflective and weakly textured surfaces. Compared with binocular stereo vision methods based on intensity images or single-channel polarization images, the 3D reconstructed surface obtained using the method proposed in this invention has a more accurate and complete target surface shape.
[0152] In one embodiment, a three-dimensional reconstruction system based on multi-channel polarization binocular image fusion includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the three-dimensional reconstruction method based on multi-channel polarization binocular image fusion as described above.
[0153] The processor and memory can be connected via a bus or other means.
[0154] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0155] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0156] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described terminal embodiment, such that the processor performs the three-dimensional reconstruction method based on multi-channel polarization binocular image fusion in the above-described embodiment.
[0157] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0158] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.
[0159] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A three-dimensional reconstruction method based on multi-channel polarization binocular image fusion, characterized in that, Includes the following steps: A multi-channel polarized binocular image set of a target scene is acquired using a polarized binocular camera. The multi-channel polarized binocular image set includes a left-eye polarization intensity image set and a right-eye polarization intensity image set. Remove polarization intensity images from the multi-channel polarized binocular image set whose overexposed area is greater than a set threshold. The remaining left eye polarization intensity image set and the right eye polarization intensity image set are arranged and combined to obtain multiple pairs of polarization binocular images with different polarization channel combinations. The camera is calibrated according to the polarization binocular images, and epipolar correction and stereo matching are performed on each polarization binocular image to obtain the disparity data cube. Outliers in the disparity data column are detected and removed using mathematical statistics methods. Based on the polarization characteristics of the data source, different weights are assigned to the remaining disparity data in the disparity data column, and the weighted average result is used as the fused disparity value. The fused disparity value is calculated pixel by pixel to obtain a fused disparity image based on a multi-channel polarized binocular image; The fused disparity image is post-processed using a disparity post-processing algorithm to obtain a final disparity image. Based on the final disparity image, 3D reconstruction is performed to obtain 3D point cloud data of the target scene. The method of acquiring a multi-channel polarized binocular image set of the target scene using a polarized binocular camera includes the following steps: The left eye of the polarization binocular camera acquires a set of polarization intensity images of the target scene at more than three different polarization angles, and the right eye of the polarization binocular camera acquires a set of polarization intensity images of the target scene at more than three different polarization angles. Based on the left and right eye polarization intensity image sets, left and right polarization degree images, left and right polarization azimuth angle images, and left and right non-polarization intensity images are calculated. The acquired polarization intensity images are located at the pixel level. Intensity value at With polarization angle Relevant, satisfying the formula: ; In the formula: To obtain the Stokes parameter, acquire at least three polarization images at different polarization angles. Then, construct a system of linear equations based on equation (1) to solve for the intensity value. ; and polarization degree ρ Polarization azimuth angle φ Non-polarization intensity Relational satisfaction formula: ; The step of calibrating the polarized binocular camera, performing epipolar correction and stereo matching on each binocular image to obtain disparity data cube, includes the following steps: By photographing single-plane checkerboard patterns in different poses, the intrinsic parameter matrix, rotation matrix, translation matrix, and distortion coefficients of the polarization binocular camera are calculated to perform camera calibration. The imaging planes of the left and right cameras are rotated by half to control the error caused by the reprojection of the left and right images, thereby ensuring that the same feature points are located in the same row of the left and right images, so as to perform epipolar correction on the binocular images. The correlation between candidate pixels and pixels to be matched is measured, the corresponding feature points are selected, and the disparity of the corresponding feature points in the left and right eye images is calculated for stereo matching to obtain the disparity data cube. The step of assigning different weights to the remaining disparity data in the disparity data column based on the polarization characteristics of the data source, and using the weighted average result as the fused disparity value, includes the following steps: Based on the polarization characteristics of the data source, different weights are assigned to the remaining disparity data in the disparity data column. The disparity calculated from the binocular polarization intensity image has the following weights: The disparity calculated from the binocular polarization degree image, the binocular polarization azimuth image, and the binocular unpolarized intensity image is... Weights ; Fusion disparity values The result of a weighted average of the disparity data column D and the weight data column W: .
2. The three-dimensional reconstruction method based on multi-channel polarization binocular image fusion according to claim 1, characterized in that, The left-eye polarization intensity image set and the right-eye polarization intensity image set are arranged and combined to obtain multiple pairs of polarization binocular images with different polarization channel combinations, including the following steps: The remaining left eye polarization intensity image set The remaining images in the right eye polarization intensity image set The images are obtained by arranging and combining the left eye polarization intensity image set and the right eye polarization intensity image set. For binocular images with different polarization channel combinations, plus left and right polarization degree images, left and right polarization azimuth angle images, and left and right unpolarized intensity images, the total... For polarized binocular images.
3. The three-dimensional reconstruction method based on multi-channel polarization binocular image fusion according to claim 1, characterized in that, Outliers in the disparity data column were detected and removed using mathematical statistics methods. An outlier was defined as an element whose converted median absolute deviation (CMAD) differed from the median of the disparity data column by more than three times. The CMAD expression is as follows: ; Where: D is the disparity data column, For the first column of disparity data i Data points.
4. The three-dimensional reconstruction method based on multi-channel polarization binocular image fusion according to claim 1, characterized in that, The fused disparity image is post-processed using a disparity post-processing algorithm to obtain the final disparity image, including the following steps: The disparity post-processing algorithm includes: small connected component removal algorithm, hole filling algorithm, and median filtering algorithm; The small connected component removal algorithm includes: determining, pixel by pixel, whether the absolute disparity difference between a pixel and its neighboring pixels is less than a preset disparity difference threshold. If so, then the point is determined to be connected to its neighboring pixels. If the number of connected pixels in the connected region obtained by the above method is less than a preset threshold, then... If so, the connected region is determined to be a small connected region, and the disparity value of the connected region is invalid; The hole-filling algorithm includes: determining the validity of the disparity value of each pixel; if invalid, searching for the closest valid pixel in the upward, downward, leftward, and rightward directions, and recording the corresponding disparity values as follows. Based on the pixel distance between the effective pixels and the pixels to be filled, a Gaussian function is used to convert the distance into weights for disparity values, denoted as follows: The disparity value of the target pixel satisfies the following formula: ; The median filtering algorithm process includes: establishing a system centered on the pixel. N × N A new disparity image is obtained by performing pixel-by-pixel calculations on a window of a certain size, with the median value of all pixel disparity values within the window being the new disparity value. This new disparity image is the final disparity image.
5. The three-dimensional reconstruction method based on multi-channel polarization binocular image fusion according to claim 4, characterized in that, Based on the final disparity image, 3D reconstruction is performed to obtain 3D point cloud data of the target scene, including the following steps: Known left image pixels disparity value at d Then in the spatial coordinates of the world coordinate system ( X , Y , Z The following equation must be satisfied: ; In the formula: The coordinates of the left camera principal point are given in the left image coordinate system. The coordinates of the principal point of the right camera in the right image coordinate system; f The focal length of the lens; b The distance between the optical centers of the left and right cameras; where, The unit is pixel. X , Y , Z , b The unit is mm.
6. A three-dimensional reconstruction system based on multi-channel polarization binocular image fusion, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the three-dimensional reconstruction method based on multi-channel polarization binocular image fusion as described in any one of claims 1 to 5.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the three-dimensional reconstruction method based on multi-channel polarization binocular image fusion as described in any one of claims 1 to 5.
Citation Information
Patent Citations
A 3D Reconstruction Method Combining Polarization and Binocular Vision
CN111899290B
Stereo matching three-dimensional reconstruction method based on dynamic programming
CN106228605A
Underwater point cloud acquisition method based on binocular camera
CN111563921A
