A three-dimensional reconstruction method based on binocular stereo matching

Abstract

The application provides a three-dimensional reconstruction method based on binocular stereo matching, and relates to the technical field of binocular stereo vision. The method combines binocular stereo matching algorithm, triangulation algorithm and surface texture mapping, and can realize surface reconstruction of scene objects in different environments. After stereo calibration is performed on a binocular camera, the internal and external parameters of the binocular camera are obtained, stereo correction is performed, then an image disparity map of the corresponding scene object is generated according to an optimized semi-global stereo matching algorithm, after the disparity information is obtained, the point cloud information converted by the object disparity is triangulated on the surface according to the triangulation algorithm, and finally three-dimensional surface reconstruction of the object is realized through the texture mapping technology. The application can reduce the influence of noise in the disparity map generation process, i.e. the semi-global stereo matching algorithm, improve the accuracy of the disparity information, and complete the three-dimensional surface reconstruction task of the object with little sacrifice of time performance.

Description

Technical Field

[0001] This invention relates to the field of binocular stereo vision technology, and in particular to a three-dimensional reconstruction method based on binocular stereo matching. Background Technology

[0002] 3D reconstruction technology involves rebuilding accurate digital models of real-world 3D objects or scenes in a computer environment, enabling analysis and processing. The accuracy of the generated model is fundamental to its further processing and interaction within the computer environment, especially crucial in the creation of virtual worlds. Currently, model creation in virtual environments relies heavily on modeling software, which is time-consuming and labor-intensive. If the accuracy issue of image-based reconstruction can be resolved, image-based reconstruction technology will become a primary means of creating virtual environments. 3D reconstruction using video or images is a major branch of computer-generated 3D models. This method primarily involves using single-view or multi-view images of a 3D model or scene, combined with camera parameters and corresponding algorithms, to further reconstruct the object's 3D information in a computer.

[0003] Binocular stereo vision is also widely used in measurement, and this method outperforms other methods in terms of efficiency, accuracy, and cost. In today's society, the rapid development of the metaverse and virtual reality concepts relies heavily on the construction of virtual environments. As a crucial means of acquiring 3D models, binocular vision-based 3D model reconstruction algorithms require significant effort. A major challenge in binocular vision-based 3D reconstruction is matching the left and right images; the accuracy of the resulting disparity map directly impacts the reconstruction result. Continuous research into binocular vision-based 3D reconstruction algorithms and improvements to address current accuracy issues will further enhance reconstruction performance, offering new possibilities for future industrial product design and manufacturing, as well as applications in other industries. Summary of the Invention

[0004] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a 3D reconstruction method based on binocular stereo matching. This method addresses the problems existing in stereo matching technology and reconstruction-related triangulation algorithms in binocular vision, and then combines texture mapping technology to complete the 3D surface reconstruction of models in different scenes. This improves the noise resistance of the stereo matching process, reduces the generation of illegal triangles in the triangulation algorithm, and thus further improves the accuracy and effect of 3D reconstruction.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A 3D reconstruction method based on binocular stereo matching includes the following steps:

[0007] Step 1: Take pictures of the calibration board from different angles using the left and right cameras through the binocular camera acquisition program. After segmenting the left and right images, complete the stereo calibration of the binocular camera using the calibration toolbox based on Zhang Zhengyou calibration method in Matlab.

[0008] Step 2: Perform epipolar correction on the calibrated binocular camera to ensure that the left and right images are coplanar and aligned after correction;

[0009] Step 3: Use the calibrated binocular camera to take pictures of the actual scene, and perform different image preprocessing based on the image quality.

[0010] Step 4: Perform stereo correction on the processed image, then select a semi-global stereo matching algorithm to perform stereo matching on the image to obtain the disparity information of the image. Generate the actual disparity value of each point in the object through the disparity map of the left and right cameras, and calculate the three-dimensional spatial coordinates of the corresponding point based on the disparity value to obtain the point cloud data of the corresponding scene.

[0011] Step 5: After segmentation and filtering, the optimized point cloud data is obtained. Then, the point cloud information obtained from the disparity map is triangulated using a triangulation algorithm.

[0012] Step 6: Apply texture mapping to the surfaced results to restore the texture information of the object to be reconstructed and obtain the reconstruction result.

[0013] Furthermore, the image preprocessing in step 3 includes the following steps:

[0014] Step 3-1: Grayscale conversion;

[0015] For indoor scene model data, the grayscale conversion method built into OpenCV is used for image grayscale conversion; for outdoor scenes, the average value method is selected for image grayscale conversion, and the grayscale value of the corresponding pixel is directly represented by the average value of the three RGB channels.

[0016] Step 3-2: Filtering;

[0017] A 3x3 convolution kernel is selected as the template for the mean filtering stage of the outdoor scene model to perform filtering operations on the grayscale image; a 5x5 convolution kernel is selected as the template for the median filtering stage of the indoor scene model to perform filtering operations on the grayscale image.

[0018] Step 3-3: Image sharpening;

[0019] To compensate for the blurred image boundaries and contours caused by median and mean filtering, image sharpening is needed to clarify the boundaries. The Log operator is selected for sharpening. First, a Gaussian filter is used to sharpen the image, and then a Laplacian operation is performed on the filtering result. The image information obtained after edge processing is applied to the original image to enhance the edge information of the original image.

[0020] Steps 3-4: Adaptive histogram equalization;

[0021] Adaptive histogram equalization alters image contrast by redistributing brightness through the calculation of local histograms; it adjusts overly bright or dark areas in real-world datasets to enhance local details.

[0022] Furthermore, the specific process of the semi-global stereo matching algorithm in step 4 is as follows:

[0023] Step 4-1: Use the multi-neighborhood minimum error gray-level mean algorithm to calculate the Census cost;

[0024] Step 4-2: Combine the AD cost and the improved Census algorithm to calculate the matching cost;

[0025] Step 4-3: Perform cost aggregation to establish connections between adjacent pixels;

[0026] Step 4-4: Perform disparity calculation and disparity optimization;

[0027] Steps 4-5: Convert the disparity map into point cloud data using the depth map.

[0028] Furthermore, step 4-1 includes the following steps when calculating the Census cost:

[0029] Step 4-1-1: Starting from the gray value of the center point, diverge outward from 8 paths within the selected Census area, with path numbers represented by 0-7;

[0030] Step 4-1-2: Calculate the sum of squared errors between the gray values ​​of the pixels on each path within the region and the center pixel. Sort the pixels from smallest to largest and select the four paths with the smallest sum of squared errors. Then, take the average gray value of each pixel on the corresponding path to replace the original gray value of the center point.

[0031] Step 4-1-3: Obtain the Census value of the center pixel. Convert the comparison result between the gray values ​​of the pixels in the neighborhood of the matching pixel and the center pixel into a 01 bit string. If the gray value of the current neighboring pixel is greater than the gray value of the center pixel, then the point is recorded as 1; otherwise, the point is recorded as 0. Assuming the selected neighborhood window size of the center pixel is n*m, the final comparison result will be a (n*m-1) bit string without the center point. This bit string represents the Census value of the current center point.

[0032] Step 4-1-4: Solve for the Hamming distance between the corresponding Census values, and the obtained value is represented as the Census cost of that point.

[0033] Furthermore, in step 4-2, when performing the fusion calculation of the two costs, the results of the two costs are normalized. After normalization, the results are added together, and the cost range obtained is [0, 2].

[0034] The formula for calculating the fusion cost is shown in equation (1);

[0035] C(p, d) = ρ(C census (p, d), λ cnesus )+ρ(C AD (p, d), λ AD (1)

[0036] in, C(p, d) represents the fusion cost of pixel p with disparity d. census For the corresponding Census value, C AD c is the corresponding AD arithmetic value, λ is the control parameter, and when both c and λ are positive, the function value range is [0, 1].

[0037] Furthermore, the energy function of the cost aggregation part in step 4-3 is shown in equation (2);

[0038]

[0039] In the above formula, the first term represents the data term, which refers to the sum of the matching costs of all pixels when the disparity map is d; N p The first term represents the pixels in the surrounding neighborhood of pixel p; the last two terms are smoothing terms, representing the remaining pixels in the parallax continuity region (|D) of the surrounding neighborhood of pixel p. p -D q |=1) and discontinuous regions (|D) p -D q The penalty term corresponding to |>1) has a smaller penalty coefficient P1 and a larger penalty coefficient P2, and P2 needs to be adjusted according to the brightness difference between adjacent pixels; T[] represents the neighboring pixels that satisfy the conditions in parentheses;

[0040] The energy value for each path direction is solved using dynamic programming, as shown in equation (3):

[0041]

[0042] The first term represents the initial matching cost of pixel p with a disparity value of d; the second and third terms are smoothing terms; the penalty cost of the second smoothing term includes no penalty for disparities of d on the path, a P1 penalty for continuous regions, and a P2 penalty for discontinuous regions, and the minimum cost among the three is taken as the penalty term of the second term; the third smoothing term ensures the new path cost L through constraints. r Not exceeding a certain upper limit;

[0043] L r (pr,d) represents the minimum cost when the disparity of the previous pixel pr is d.

[0044] L r (pr,d-1) is the minimum cost when the disparity of the previous pixel pr is d-1.

[0045] L r (pr,d+1) is the minimum cost when the disparity of the previous pixel pr is d+1.

[0046] The minimum cost when the disparity of the previous pixel pr is any other value i;

[0047] The minimum cost when taking different disparity values ​​k for the previous pixel;

[0048] After obtaining the path cost of each single path, select an appropriate multi-path to accumulate the cost, as shown in equation (4).

[0049]

[0050] The final disparity value of pixel p is the disparity value with the minimum accumulated cost for each path; after performing the calculation on all pixels in the image, the disparity value of the entire image is obtained.

[0051] Furthermore, in step 4-4, the disparity calculation stage adopts the winner-take-all algorithm, which selects the disparity corresponding to the minimum cost after aggregation of a single pixel as the optimal disparity of that point; the disparity optimization stage uses the methods of mismatch elimination and noise suppression for optimization.

[0052] Furthermore, in steps 4-5, the disparity image is converted into a depth image according to equation (5), and the depth information in the image is used to obtain the corresponding point cloud data through coordinate calculation;

[0053]

[0054] Where D is depth, d is parallax, B is baseline length, f is focal length, and X... 0l X 0r These represent the column coordinates of the principal point in the left and right views, respectively.

[0055] Furthermore, the triangulation algorithm described in step 5 is an improved version of Lawson's algorithm based on the point-by-point insertion method. The steps of the improved algorithm based on Lawson's algorithm include:

[0056] Step 5-1: Construct a large triangle containing all data points in the point set;

[0057] Step 5-2: Insert a point into the triangle and connect that point to the three vertices of the triangle to form three new triangles;

[0058] Step 5-3: Perform empty circumcircle detection on each of the newly generated triangles, and simultaneously use LOP for local optimization;

[0059] Step 5-4: Insert the points in the point set one by one, repeating steps 5-2 and 5-3 until all data points have been inserted;

[0060] Step 5-5: Delete the initially constructed large triangle;

[0061] Steps 5-6: In the application of the Lawson algorithm, by limiting the maximum length of the triangle edges in the triangulation network, illegal and meaningless triangles formed in the triangulation network are removed, and the subdivision result is further optimized.

[0062] The beneficial effects of the above technical solution are as follows: The 3D reconstruction method based on binocular stereo matching provided by this invention employs different preprocessing methods during the image acquisition stage, considering different lighting conditions and image information in different scenarios (indoor and outdoor). In the binocular stereo matching stage, it abandons the traditional semi-global stereo matching algorithm based on mutual information, and improves the semi-global algorithm's weak anti-interference ability and poor performance in areas with repetitive textures by fusing an improved Census algorithm with AD cost. Secondly, after obtaining disparity information through stereo matching, in the triangulation stage of reconstruction, to reduce the problem of too many illegal triangles in the triangulation results of point cloud data, constraints are added during the triangulation process to optimize the results. Finally, material mapping is performed on the results of the three-step optimization algorithm, effectively restoring the details of the object to be reconstructed. Experiments show that this invention can improve the overall reconstruction results of the model obtained by the binocular 3D reconstruction method to a certain extent while optimizing each reconstruction module. Attached Figure Description

[0063] Figure 1 A flowchart illustrating a 3D reconstruction method based on binocular stereo matching provided for a specific embodiment of the present invention;

[0064] Figure 2 A physical image of an image acquisition platform for three-dimensional surface reconstruction provided for a specific embodiment of the present invention;

[0065] Figure 3 A physical image of the chessboard calibration plate provided for a specific embodiment of the present invention;

[0066] Figure 4 The images show actual calibration plates captured by binocular cameras at different angles, as provided in a specific embodiment of the present invention. Among them, Figure (4a) is an image of the calibration plate captured by the left camera, and Figure (4b) is an image of the calibration plate captured by the right camera.

[0067] Figure 5 The image corner search diagram of the left and right cameras of the camera on the same calibration board is provided for a specific embodiment of the present invention;

[0068] Figure 6 A schematic diagram of the relative positions of the binocular camera and the calibration plate is provided for a specific embodiment of the present invention; wherein, Figure (6a) is a side angle schematic diagram of the positional relationship, and Figure (6b) is a top view of the relative positions;

[0069] Figure 7 A histogram of bi-target fixed-weight projection error provided for a specific embodiment of the present invention;

[0070] Figure 8The binocular corrected left and right images provided for a specific embodiment of the present invention;

[0071] Figure 9 The images shown are the results of image sharpening under the action of various operators provided in specific embodiments of the present invention; wherein, Figure (9a) is the original image, Figure (9b) is the Roberts sharpening result, Figure (9c) is the Prewitt sharpening result, Figure (9d) is the Sobel sharpening result, Figure (9e) is the Laplacian sharpening result, Figure (9f) is the Canny sharpening result, and Figure (9g) is the LOG sharpening result.

[0072] Figure 10 The following are the results of point cloud segmentation and filtering provided for specific embodiments of the present invention; wherein, Figure (10a) is the result of point cloud segmentation without the segmentation, and Figure (10b) is the result of segmenting useless background and filtering;

[0073] Figure 11 The following are comparison diagrams of the partitioning results with and without added constraints, provided for specific embodiments of the present invention; wherein, Figure (11a) is the partitioning result without added constraints, and Figure (11b) is the partitioning result with added constraints.

[0074] Figure 12 The following are binocular three-dimensional reconstruction results provided for specific embodiments of the present invention; wherein, Figure (12a) is a reconstruction model of an indoor scene and Figure (12b) is a reconstruction model of an outdoor scene. Detailed Implementation

[0075] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0076] like Figure 1 As shown, a three-dimensional reconstruction method based on binocular stereo matching in this embodiment includes the following steps:

[0077] Step 1: Use the binocular camera acquisition program to capture images of the calibration board from different angles using the left and right cameras. After segmenting the left and right images, use the calibration toolbox based on Zhang Zhengyou's calibration method in Matlab to complete the stereo calibration of the binocular camera.

[0078] The ultimate goal of 3D reconstruction based on binocular cameras is to obtain a disparity map through stereo matching, then use triangulation to determine the relative positions of points on the object's surface from the camera, thereby obtaining the object's 3D point cloud data. This data is then processed using a subdivision algorithm to reconstruct the object's surface stereo information. However, this is under ideal conditions with the binocular camera. Typically, binocular cameras exhibit two different types of distortion: radial distortion and tangential distortion. Therefore, stereo calibration and correction are necessary before using a binocular camera to ensure it operates under ideal conditions.

[0079] This embodiment uses Zhang Zhengyou's calibration method as its theoretical basis, and utilizes MATLAB to perform offline calibration of the binocular camera. Two sets of cameras with identical parameters are selected, and a binocular parallel optical axis structure is constructed using a support. The two cameras are parallel with a fixed baseline length (physical measurement distance approximately 176mm), and both have a resolution of 640*480. The camera pitch angle is adjustable. Figure 2 As shown. A 7*10 checkerboard calibration plate was selected. To prevent physical deformation of the checkerboard due to external factors during the calibration process, it was fixed to a sturdy flat surface. Each checkerboard square measures 30mm*30mm. Figure 3 As shown.

[0080] In this embodiment, the position of the binocular camera is fixed, and the angle and distance of the calibration board are adjusted multiple times. An image acquisition program written in VS2019 and OpenCV is used to capture images of the calibration board. According to calibration theory, too few or too many calibration images will have a negative impact on the calibration results. In order to make the calibration results more accurate, this embodiment captures a total of 19 sets of left and right images. Images that cause excessive errors can be removed later.

[0081] After obtaining the images captured by the binocular camera, since the images simultaneously contain left and right images of the calibration checkerboard pattern, it is necessary to crop the captured left and right images separately. The cropping steps are as follows: When the image acquisition program reads the camera frame by frame, the current image is saved by pressing a key, and the width and height of the camera image are obtained. A custom rectangle is used to segment the left and right images. The coordinates of the custom cropping rectangle start from the (0,0) point in the pixel coordinate system, that is, the upper left corner of the image, and its size is half the width of the camera image and the entire height. The same image containing the left and right images is divided into corresponding left and right images. Then, the segmented images are named according to the rules and stored in the corresponding left and right image folders, such as... Figure 4 As shown.

[0082] In MATLAB, select the stereo camera calibration program and add the segmented left and right image folders to the specified location in the program. Since the physical size of the calibration board's checkerboard is 30*30 mm, manually set the checkerboard size to 30 mm. Run the calibration program and select any image. It can be observed that the corner points at the intersection of the black and white checkerboard are detected, and the detected corner points coincide with the actual corner points in the image. Figure 5 As shown. If significant errors occur during calibration, some images with large deviations can be removed to improve the calibration results. After calibration, the intrinsic and extrinsic parameters of the stereo camera will be obtained, and the relative positions of the camera and multiple calibration images will be as shown. Figure 6 As shown.

[0083] The reprojection error corresponding to each image during the calibration process is as follows: Figure 7 As shown, the average reprojection error of the calibration experiment was 0.11 pixels, which is within a reasonable range and can be used for subsequent experiments. If the reprojection error is too large during the actual calibration process, the image pair causing the excessive error can be removed and recalibrated. Besides the influence of hardware equipment, the excessive reprojection error is partly caused by an excessively large angle between the camera and the calibration object during shooting. Generally, a shooting angle range of 0–45° yields better results.

[0084] After calibration using the toolbox in MATLAB software, the intrinsic parameters of the left and right cameras of the stereo camera, as well as the rotation and translation matrices of the camera, will be obtained. The parameters of the stereo camera are shown in Table 1.

[0085] Table 1. Parameters of the binocular camera

[0086]

[0087] Step 2: Perform epipolar correction on the calibrated binocular camera so that the left and right images can be coplanar and aligned after correction.

[0088] After obtaining the intrinsic parameters, rotation, and translation matrices of the stereo camera through calibration, the calibrated camera parameters are used to perform epipolar correction on the images. This is to reduce search time and improve matching efficiency during subsequent stereo matching. This embodiment uses VS2019 combined with OpenCV to write an image correction program, ensuring that the left and right images are coplanar and row-aligned after correction. The correction images are the checkerboard left and right images used during camera calibration. Figure 8 As shown, it can be observed that the corresponding matching points in the image before correction do not meet the epipolar constraint conditions, while the matching points in the image after correction are all located on the same straight line.

[0089] Step 3: Use the calibrated binocular camera to take pictures of the actual scene, and perform different image preprocessing operations to improve the image quality.

[0090] When reconstructing a real 3D scene model, the acquisition and segmentation procedures used in camera calibration are required. After acquiring the left and right images of the scene, preprocessing is necessary to obtain better reconstruction results. To address model data reconstruction under different environments, this embodiment selects indoor and outdoor models for acquisition and reconstruction. In the image preprocessing stage, different preprocessing workflows are designed for different datasets.

[0091] Step 3-1: Grayscale Conversion: This embodiment selects different grayscale conversion methods. For indoor model data, where the lighting is relatively uniform, the grayscale conversion method built into OpenCV is used for image grayscale conversion. For outdoor scenes, where the lighting varies greatly, in order to balance the brightness information as much as possible, the average value method is selected for image grayscale conversion, and the grayscale value of the corresponding pixel is directly represented by the average value of the three RGB channels.

[0092] Step 3-2: Filtering methods: Select a 3*3 convolution kernel as the template for the mean filtering stage of the outdoor scene model to perform filtering operations on the grayscale image; select a 5*5 convolution kernel as the template for the median filtering stage of the indoor scene model to perform filtering operations on the grayscale image.

[0093] Step 3-3: Image Sharpening: To compensate for the blurred image boundaries and contours caused by median and mean filtering, image sharpening is needed to clarify the boundaries. Applying the image information obtained after edge processing to the initial image can sharpen the edge information of the original image. This embodiment selects various sharpening effects for different scenes, and the sharpening operators used include Roberts, Prewitt, Sobel, Laplacian, Canny, and LoG operators. The LoG operator is characterized by first using a Gaussian filter to sharpen the image, and then performing a Laplacian operation on the filtered result. Taking an indoor image as an example, the image preprocessing is performed using the aforementioned sharpening methods, and the resulting image edge results are as follows. Figure 9 As shown, it can be observed that under the processing of various operators, the Log operator preserves the image edge information and the rest of the image information does not have too many noise points, which facilitates subsequent superposition with the original image.

[0094] Steps 3-4: Histogram equalization is primarily used for image enhancement. However, traditional equalization methods offer limited improvement if the image contains excessively bright or dark areas. Adaptive histogram equalization, by calculating the local histogram of the image and redistributing brightness, alters the image's contrast. In this embodiment, it adjusts excessively bright or dark areas in the actual dataset to enhance local details.

[0095] Step 4: Perform stereo correction on the processed image, then select the semi-global stereo matching algorithm SGBM to obtain the disparity information of the image, generate the actual disparity value of each point in the object through the disparity map of the left and right cameras, and calculate the three-dimensional spatial coordinates of the corresponding point based on the disparity value to obtain the point cloud data of the corresponding scene.

[0096] Step 4-1: To address the issue of traditional Census relying too heavily on the grayscale value of the center pixel, the traditional Census method is optimized during the cost calculation stage of stereo matching. A multi-neighborhood minimum error grayscale mean algorithm is used to calculate the Census cost, reducing noise interference. The specific method is as follows:

[0097] Starting from the center point's grayscale value, eight paths radiate outwards within the selected Census region, numbered 0-7. The sum of squared errors between the grayscale values ​​of pixels along each path and the center point's pixel is calculated. These paths are sorted from smallest to largest, and the four paths with the smallest sum of squared errors are selected. The average grayscale value of each pixel along this path is used to replace the original center point's grayscale value. The Census value of the center point pixel is obtained. The comparison result between the grayscale values ​​of pixels in the neighborhood of the matching pixel and the center pixel is converted into a 0 / 1 bit string. If the grayscale value of the current neighboring pixel is greater than that of the center point, the point is marked as 1; otherwise, it is marked as 0. Assuming the selected center point's neighborhood window size is n*m, the final comparison result will be an (n*m-1) bit string excluding the center point. This bit string represents the current center point's Census value. The Hamming distance between the corresponding Census values ​​is calculated, and the resulting value represents the Census cost of that point.

[0098] Step 4-2: In this embodiment, the AD cost and the improved Census algorithm are fused to calculate the matching cost. The AD cost calculation method can preserve the details and texture information of the image, while Census can compensate for the influence of noise and lighting when using AD cost alone. In the cost calculation stage, the improved method performs well in areas with repetitive textures.

[0099] When performing the cost fusion calculation, the results of the two algorithms have different scales. The AD algorithm calculates the brightness difference, which ranges from [0, 255], while the Census algorithm calculates the number of different bit values ​​in the bit string generated by the corresponding Census value. Therefore, the two cost results need to be normalized. After normalization, the results are added together, and the resulting cost range should be [0, 2].

[0100] The formula for calculating the fusion cost is shown in formula (1).

[0101] C(p, d) = ρ(C census (p, d), λ cnesus )+p(C AD (p, d), λ AD (1)

[0102] in C(p, d) represents the fusion cost of pixel p with disparity d. census For the corresponding Census value, C AD c is the corresponding AD arithmetic value, λ is the control parameter, and when both c and λ are positive, the function value range is [0, 1].

[0103] Step 4-3: Although multiple cost fusion calculation methods were used in the cost calculation stage, only local correlation was considered. If the initial disparity obtained from the cost calculation is used directly to generate a disparity map, a good matching effect cannot be obtained. Therefore, it is necessary to aggregate the obtained initial disparity values. Cost aggregation is actually to establish the relationship between adjacent pixels. Aggregated cost values ​​can more accurately represent the correlation between pixels with the same name.

[0104] The semi-global stereo matching (SGBM) algorithm uses a global algorithm optimization method for disparity estimation. The global energy function is defined by a data term and a smoothing term: the data term refers to the overall matching cost of the disparity corresponding to the disparity in the disparity map, and the smoothing term refers to the penalty for disparity discontinuities between adjacent pixels, usually to increase the energy value with the help of the penalty term. The energy function of the cost aggregation part of the SGBM algorithm is shown in Equation (2).

[0105]

[0106] The first term in the formula represents the data term, which is the sum of the matching costs of all pixels when the disparity map is d. The last two terms are smoothing terms, which are the surrounding neighbors N of pixel p. pThe remaining pixels within the brackets are penalized in the continuous and discontinuous regions of disparity. The penalty coefficient P1 is relatively small, while P2 is relatively large. This can better preserve the discontinuous regions of disparity in the actual scene, and P2 needs to be adjusted according to the brightness difference between adjacent pixels. T[] represents the neighboring pixels that satisfy the conditions in the brackets.

[0107] Finding the optimal solution for this optimization function is an NP-complete problem. In SGBM, the scan line method, also known as unidirectional dynamic programming, is used to solve the corresponding problem. This method uses the aggregation of multiple one-dimensional paths to approximate the two-dimensional optimal solution. The matching cost of a pixel within the disparity range is aggregated in one dimension for each path, and the resulting cost is accumulated and recorded as the aggregate cost of that pixel.

[0108] The path cost of pixel p along a single path r, i.e. the energy value under a single path, is solved as shown in equation (3):

[0109]

[0110] The first term represents the initial matching cost of pixel p with a disparity value of d; the second and third terms are smoothing terms, where the penalty cost of the second term includes no penalty for disparity d along the path, a P1 penalty for continuous regions, and a P2 penalty for discontinuous regions, and the minimum cost among the three is taken as the penalty term of the second term; the third term ensures the new path cost L through constraints. r It shall not exceed a certain upper limit.

[0111] After obtaining the path cost of each individual path, a suitable multi-path is selected for cost accumulation. Typically, there are three path aggregation methods: 4-path, 8-path, and 16-path. The larger the number of paths, the closer the aggregation result is to the two-dimensional optimum; however, the more paths involved in the calculation, the longer the computation time. In actual scene reconstruction, path selection needs to be based on reconstruction requirements. In this embodiment, considering both reconstruction accuracy and computational efficiency, 8-path aggregation is selected from the real-world scene dataset, as shown in equation (4).

[0112]

[0113] The final disparity value of pixel p is the disparity value with the minimum accumulated cost for each path; after performing the calculation on all pixels in the image, the disparity value of the entire image is obtained.

[0114] Step 4-4: The disparity calculation stage employs a winner-takes-all algorithm. For a single pixel, the disparity corresponding to the minimum cost after pixel aggregation is selected as the optimal disparity for that point. After the initial disparity map is generated, it needs to be optimized using various constraints. To address various erroneous matching points caused by noise, occlusion, weak textures, or repetitive textures during the disparity calculation process, and to process some disparity holes and mismatched points, erroneous disparities are eliminated to improve matching accuracy. This embodiment uses methods of erroneous match elimination and noise suppression for optimization, generating an optimized disparity map with better results.

[0115] Steps 4-5: Parallax refers to the difference between the column coordinates of the corresponding pixel in the left view and the column coordinates of the corresponding pixel in the right view for a pair of points of an object in the left and right images, usually represented by d. Depth D refers to the Z coordinate value of a pixel in the current view camera coordinate system, which is a spatial unit. Point cloud data refers to the set of three-dimensional points of an object in three-dimensional space. The parallax image is converted into a depth image according to formula (5), and the depth information in the image can be used to obtain the corresponding point cloud data through coordinate operations.

[0116]

[0117] Where D is depth, d is parallax, B is baseline length, f is focal length, and X... 0l and X 0r These are the column coordinates of the principal point in the left and right views, respectively.

[0118] Step 5: Use the triangulation algorithm to triangulate the point cloud information obtained from the disparity map.

[0119] Step 5-1: Two types of data need to be processed separately in the initial point cloud: one type is useless information in the scene, such as non-overlapping parts of the left and right images, or useless background information during the reconstruction of the scene model. For this type of data, triangulation and reconstruction are not required later. For performance reasons, it can be segmented from the point cloud of the target region during the point cloud processing stage. The other type is noise point information. Due to various errors during reconstruction, the point cloud data at key locations of the scene model will contain some noise points that need to be processed, including outliers, abnormal protrusions, and depressions. If such points are not processed, problems such as broken surfaces or uneven surfaces will occur during subsequent segmentation and surface reconstruction. A point cloud processing method that can be selected for this type of noisy data is filtering and denoising, smoothing irregular point cloud parts. This embodiment uses the Adirondack dataset as an example, and the point cloud images before and after segmentation are as follows: Figure 10 As shown.

[0120] Step 5-2: After obtaining the optimized point cloud data through segmentation and filtering, the surface of the 3D model can be reconstructed using a triangulation algorithm. Triangulation requires connecting discrete point cloud data into multiple triangles to achieve the purpose of model surface reconstruction.

[0121] This embodiment selects the point-by-point insertion method for triangulation. The Lawson algorithm in the point-by-point insertion method is simple to implement, and the steps are as follows:

[0122] (1) Construct a large triangle containing all data points in the point set.

[0123] (2) Insert a point into it and connect the point to the three vertices of the triangle to form three new triangles.

[0124] (3) Perform empty circumcircle detection on the newly generated triangles and use LOP for local optimization.

[0125] (4) Insert the points in the point set one by one, repeating steps (2) and (3) until all data points have been inserted.

[0126] (5) Delete the initially constructed large triangle.

[0127] When constructing a triangulation using the Lawson algorithm, if the point cloud data contains inner loops, the resulting triangulation will include invalid triangles. This paper, in applying the Lawson algorithm, limits the maximum length of the sides of the triangles in the triangulation network to remove invalid and meaningless triangles, further optimizing the triangulation results. A comparison before and after adding constraints is provided. Figure 11 As shown.

[0128] Step 6: Apply texture mapping to the surface-mapped results to restore the texture information of the object to be reconstructed, obtaining the reconstruction result. To better showcase model details and achieve a better visual effect, this method visualizes the mesh data of the triangulation results. Using the point cloud data visualization function built into the PCL open-source library, the triangulation results of the processed point cloud data are displayed. The final 3D reconstruction result is shown below. Figure 12 As shown, the content of indoor and outdoor scenes has been reconstructed, and the core content of the reconstructed model is relatively complete.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for three-dimensional reconstruction based on binocular stereo matching, characterized in that: The method comprises the following steps: Step 1: the left and right cameras shoot pictures of different angles of the calibration board through a binocular camera acquisition program, the left and right images are segmented, and the stereo calibration of the binocular camera is completed through a calibration toolbox based on Zhang Zhengyou's calibration method in Matlab; Step 2: for the calibrated binocular camera, perform epipolar rectification so that the left and right images can be coplanar and aligned after rectification; Step 3: use the binocular camera after calibration to shoot pictures of the actual scene, and perform different image preprocessing according to the image shooting quality; Step 4: perform stereo rectification on the processed images, then select a semi-global stereo matching algorithm to perform stereo matching on the images to obtain the disparity information of the images, generate the actual disparity values of each point in the object through the disparity maps of the left and right cameras, calculate the three-dimensional space coordinates of the corresponding points according to the disparity values, and obtain the point cloud data of the corresponding scene; the specific process of the semi-global stereo matching algorithm is as follows: Step 4-1: use a multi-path neighborhood minimum error gray mean algorithm to calculate the Census cost, including the following steps: Step 4-1-1: take the gray value of the center point as the starting point, and diverge outward from 8 paths in the selected Census region, with path numbers represented by 0-7; Step 4-1-2: calculate the sum of square errors of the pixel gray values of each path in the region and the center point, sort them from small to large, select the four paths with the smallest sum of square errors, and replace the gray value of the original center point with the gray mean value of each pixel in the corresponding path; Step 4-1-3: obtain the Census value of the center point pixel, and convert the comparison results of the pixel gray values in the neighborhood of the matching pixel and the center pixel into a 01 bit string; if the gray value of the current pixel in the neighborhood is greater than the gray value of the center point pixel, the point is recorded as 1, otherwise the value of the point is recorded as 0; assuming that the selected neighborhood window size of the center point pixel is n*m, the final comparison result will be a bit string of (n*m-1) without the center point, which represents the Census value of the current center point; Step 4-1-4: solve the Hamming distance of the Census values between the corresponding points, and the obtained value represents the Census cost value of the point; Step 4-2: fuse the AD cost and the improved Census algorithm to calculate the matching cost; Step 4-3: aggregate the cost to establish the connection between adjacent pixels; Step 4-4: perform disparity calculation and disparity optimization; Step 4-5: convert the disparity map into point cloud data with the help of a depth map; Step 5: obtain the optimized point cloud data through segmentation and filtering, and then use a triangular subdivision algorithm to perform triangular surface processing on the point cloud information obtained from the disparity map; Step 6: perform texture mapping on the surface processing result to restore the texture information of the object to be reconstructed, and obtain the reconstruction result.

2. The binocular stereo matching based 3D reconstruction method of claim 1, wherein: The image preprocessing in step 3 comprises the following steps: Step 3-1: grayscale; For indoor scene model data, the grayscale method provided by OpenCV is selected to perform image grayscale; for outdoor scene, the average method is selected to perform image grayscale, and the grayscale value of the corresponding pixel point is directly represented by the average of the values of the three channels of RGB; Step 3-2: filtering; A 3*3 convolution kernel is selected as a template for the mean filtering stage of the outdoor scene model to perform filtering operation on the grayscale image; a 5*5 convolution kernel is selected as a template for the median filtering stage of the indoor scene model to perform filtering operation on the grayscale image; Step 3-3: image sharpening; In order to compensate for the influence that the image boundary and contour become blurred after median and mean filtering, image sharpening is needed to make the boundary clear; a Log operator is selected to perform sharpening operation, and a Gaussian filter is first used to perform sharpening processing on the image, and then Laplace operation is performed on the filtering result; the image information obtained through edge processing is applied to the initial image, so that the edge information of the original image is enhanced; Step 3-4: adaptive histogram equalization; Adaptive histogram equalization changes the contrast of the image by calculating the local histogram of the image and redistributing the brightness; for the over-bright or over-dark areas in the actual data set, the local details are enhanced.

3. The binocular stereo matching based 3D reconstruction method of claim 1, wherein: In step 4-2, when the two cost fusion calculations are performed, the two cost results are normalized, and the results after normalization are added, and the cost range obtained is [0, 2]; The calculation formula of the fusion cost is shown as formula (1); where ρ(c, λ) = cλc+λ , C(p, d) denotes the fusion generation value of pixel p when the disparity is d, is the corresponding Census generation value, is the corresponding AD generation value, and λ is a control parameter. When c and λ are both positive, the function value interval is [0, 1].

4. The binocular stereo matching based 3D reconstruction method of claim 1, wherein: The energy function of the cost aggregation part in step 4-3 is shown as formula (2); （2） The first term in the above formula represents the data item, which refers to the sum of matching cost of all pixel points when the disparity map is d; The first term in the above formula represents the data item, which refers to the sum of matching cost of all pixel points when the disparity map is d; The first term in the above formula represents the data item, which refers to the sum of matching cost of all pixel points when the disparity map is d; The first term in the above formula represents the data item, which refers to the sum of matching cost of all pixel points when the disparity map is d; The first term in the above formula represents the data item, which refers to the sum of matching cost of all pixel points when the disparity map is d; T[ ] represents the neighborhood pixel points that meet the conditions in the brackets; The energy value in each path direction is solved by the dynamic programming method, and the solution is shown as formula (3): Wherein, the first term represents the initial matching cost value of the pixel point p when the disparity value is d; the second term and the third term are smoothing terms; the penalty cost of the second term smoothing term includes no penalty for the disparity d on the path, penalty for the disparity d in the continuous region, and penalty for the discontinuous region where the pixel point p is located, and the minimum cost value among the three is taken as the penalty term of the second term; the third term smoothing term is to limit the new path cost value not to exceed a certain upper limit. ​​ the minimum cost value when the disparity value of the previous pixel point is . the minimum cost value when the disparity value of the previous pixel point is the minimum cost value when the disparity value of the previous pixel point the minimum cost value when the disparity value of the previous pixel point is the minimum cost value when the disparity value of the previous pixel point the minimum cost value when the disparity value of the previous pixel point is other value i the minimum cost value when the disparity value of the previous pixel point is other value i The minimum generation value is taken for the previous pixel point with different disparity value k. After the path cost of each single path is obtained, a suitable multi-path is selected for cost accumulation, which is shown as formula (4); （4） The final disparity value of the pixel point p is the disparity value with the least accumulated cost of each path; after the operation of all pixel points in the image, the disparity value of the entire image is obtained.

5. The binocular stereo matching based 3D reconstruction method of claim 1, wherein: In step 4-4, the winner-takes-all algorithm is used in the disparity calculation stage, and for a single pixel, the disparity corresponding to the minimum cost after aggregation is selected as the optimal disparity of the point; In the disparity optimization stage, the error matching elimination and noise suppression method is used for optimization.

6. The binocular stereo matching based 3D reconstruction method of claim 1, wherein: In step 4-5, the disparity image is converted into a depth image according to formula (5), and the depth information in the image is obtained through coordinate operation to obtain corresponding point cloud data; where D is the depth, d is the disparity, B is the length of the baseline, and f is the focal length, , respectively denote the column coordinates of the principal point in the left and right views.

7. The binocular stereo matching based 3D reconstruction method of claim 1, wherein: In step 5, the Lawson algorithm based on the point-by-point insertion method is improved, and the improved algorithm steps based on the Lawson algorithm include: Step 5-1: establish a large triangle containing all data points in the point set; Step 5-2: insert a point into the triangle and connect the point with the three vertices of the triangle to form three new triangles; Step 5-3: perform empty circumscribed circle detection on the newly generated triangles, and perform local optimization by using LOP; Step 5-4: insert the points in the point set one by one, repeat step 5-2 and step 5-3 until all data points are inserted; Step 5-5: delete the large triangle initially constructed; Step 5-6: in the application process of the Lawson algorithm, by limiting the maximum length of the triangular mesh edge of the subdivision network, the illegal and meaningless triangles formed in the triangular mesh are removed, and the subdivision result is further optimized.

Images