Improved binocular stereo matching fusion algorithm

Abstract

The application relates to the technical field of computer vision and image processing, and discloses an improved binocular stereo matching fusion algorithm, which comprises the following steps: S1, input image pre-processing; S2, multi-scale depth feature extraction; S3, based on the fused multi-scale features, an improved matching cost function is designed; S4, the calculated matching cost is aggregated and optimized, and a disparity map is generated according to the optimized matching result; and S5, an adaptive learning mechanism is introduced, the depth learning model and the multi-scale feature fusion parameters are fine-tuned through real-time feedback, and a high-precision disparity map is output. The application has higher matching precision, robustness and real-time adjustment capability, and solves the problems of serious mismatching under a complex environment and insufficient real-time performance in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, and particularly relates to an improved binocular stereo matching fusion algorithm. BACKGROUND

[0002] Binocular stereo vision technology is a key technology for obtaining three-dimensional information by simulating the human visual system, and is widely used in automatic driving, robot vision and three-dimensional reconstruction fields. The core of binocular stereo vision technology is to obtain the depth information of the target object by matching the corresponding pixels in the left and right images to obtain the disparity map. However, in practical applications, an accurate and efficient stereo matching algorithm has always been the focus and difficulty of the field.

[0003] Current stereo matching algorithms can be divided into three categories: global matching, local matching and semi-global matching. Among them, the commonly used algorithms include BM (Block Matching) algorithm, SGBM (Semi-Global Block Matching) algorithm and GC (Graph Cuts) algorithm, etc. These algorithms have their own advantages and disadvantages in practical applications.

[0004] BM algorithm: As a simple block matching algorithm, BM calculates the disparity by finding the optimal matching block in a small area. However, this algorithm has poor robustness to light changes and image noise, which can easily lead to matching errors, especially in complex environments such as power inspection robots.

[0005] SGBM algorithm: SGBM is a semi-global matching algorithm based on blocks, which combines the advantages of local and global information, and can obtain better matching accuracy while maintaining high efficiency. However, this algorithm has high computational complexity and may still produce mismatching in environments with high noise.

[0006] GC algorithm: The global matching algorithm based on graph cuts solves the matching problem by global minimization of the energy function, which has high accuracy. However, its computational cost is very high and is not suitable for real-time application scenarios.

[0007] In addition, Census transformation, as a matching cost calculation method based on local regions, is commonly used in practical applications due to its robustness to light changes and fast running speed. However, this algorithm is highly dependent on the gray value of the center pixel and is sensitive to noise, which can easily produce errors in complex outdoor environments. These shortcomings make existing stereo matching algorithms unable to achieve high-precision and robust depth estimation when deployed on power inspection robot platforms.

[0008] The power inspection robot usually needs to perform tasks in a complex outdoor environment, and the image has much noise and obvious light change, which puts higher requirements on the traditional stereo matching algorithm, in order to solve the problems of low matching accuracy and high computational complexity of the existing algorithm in a noisy environment, an improved binocular stereo matching fusion algorithm is designed to solve the above problems. SUMMARY

[0009] This section is intended to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification of the present application in order to avoid obscuring the purpose of this section, the abstract and the title, and such simplifications or omissions cannot be used to limit the scope of the present application.

[0010] In view of the above problems of the prior art, the present application is proposed.

[0011] Therefore, the purpose of the present application is to provide an improved binocular stereo matching fusion algorithm, which is suitable for solving the problems of poor balance between local features and global information and limited optimization effect of the disparity map.

[0012] To solve the above technical problems, the present application provides the following technical scheme: an improved binocular stereo matching fusion algorithm, comprising the following steps:

[0013] S1: input image pre-processing;

[0014] S2: multi-scale depth feature extraction;

[0015] S3: based on the fused multi-scale features, an improved matching cost function is designed;

[0016] S4: the calculated matching cost is aggregated and optimized, and a disparity map is generated according to the optimized matching result;

[0017] S5: an adaptive learning mechanism is introduced, and the depth learning model and the multi-scale feature fusion parameters are fine-tuned through real-time feedback, and a high-precision disparity map is output;

[0018] A luminance difference matching cost calculation formula is constructed, and the luminance difference matching cost calculation formula is as follows:

[0019]

[0020] Wherein, C1(x,y,d) represents the luminance difference matching cost at pixel (x,y), which measures the luminance difference of the left image and the right image under different disparity hypothesis d; I L (x,y) and I R(x-d,y) represents the pixel values of the left and right images at (x,y) and (x-d,y) respectively, and the disparity d represents the displacement between the left and right images; σ s The matching cost at different scales is normalized to the same range for normalization; p is used to control the influence degree of brightness difference.

[0021] As a preferred scheme of the improved binocular stereo matching fusion algorithm, the brightness difference matching cost threshold is set to τ1 according to the output result of the brightness difference matching cost calculation formula.

[0022] When C1(x,y,d) > τ1, it indicates that the brightness difference is too large, and the pixel point corresponding to the disparity d is not accurately matched.

[0023] When C1(x,y,d) ≤ τ1, it indicates that the brightness difference is small, and the pixel point corresponding to the disparity d is accurately matched.

[0024] As a preferred scheme of the improved binocular stereo matching fusion algorithm, the output result of the brightness difference matching cost calculation formula is filtered according to the set threshold τ1, and the brightness difference matching cost with small brightness difference is selected as the input result to construct the multi-scale feature fusion formula, and the multi-scale feature fusion formula is as follows:

[0025]

[0026] Wherein, F S (x,y,d) represents the multi-scale feature fusion cost at pixel (x,y); And represent the feature values extracted at different scales s of the left and right images; w S is a scale weight coefficient, and the features at different scales have different influence weights; S is the total number of multi-scales.

[0027] As a preferred scheme of the improved binocular stereo matching fusion algorithm, the multi-scale feature fusion cost threshold τ2 is set according to the output result of the multi-scale feature fusion formula.

[0028] When F s (x,y,d) > τ2, it indicates that the feature difference is large, and the matching is not accurate.

[0029] When F s (x,y,d) ≤ τ2, it indicates that the feature difference is small, and the matching is accurate.

[0030] As a preferred scheme of the improved binocular stereo matching fusion algorithm, the output result of the multi-scale feature fusion formula is filtered according to a set multi-scale feature fusion cost threshold τ2, the multi-scale feature fusion cost with small multi-scale feature difference is selected as the input result to construct the smoothing constraint and gradient penalty formula, and the smoothing constraint and gradient penalty formula is as follows:

[0031]

[0032] Wherein, P(x, y, d) represents the matching cost after smoothing constraint and gradient penalty; represents the second-order partial derivative of the feature map in the x and y directions, and controls the smoothness of the disparity map; λ represents a smoothing control parameter; α is a coefficient of the gradient penalty term, and is used for controlling the influence of the gradient of the depth feature on the matching cost; represents the penalty term of the feature gradient, and β is a learning rate.

[0033] As a preferred scheme of the improved binocular stereo matching fusion algorithm, the output result of the smoothing constraint and gradient penalty formula is set with a matching cost threshold τ3 after smoothing constraint and gradient penalty.

[0034] When P(x, y, d) > τ3, it indicates that the matching cost of the point is high and not smooth, and there is noise and mismatch, and the disparity needs to be adjusted;

[0035] When P(x, y, d) ≤ τ3, it indicates that the matching cost of the point is low and smooth, and the disparity does not need to be adjusted.

[0036] As a preferred scheme of the improved binocular stereo matching fusion algorithm, an adaptive learning and final matching cost aggregation formula is constructed according to the output result of the smoothing constraint and gradient penalty formula, and the adaptive learning and final matching cost aggregation formula is as follows:

[0037]

[0038] Wherein, D(x, y) represents the final disparity value at the pixel (x, y); γ is an adaptive learning control parameter, and determines the influence of the historical data on the current matching result; The past matching cost is weighted by the adaptive learning mechanism, wherein T is the time step number, μ t and σ t are the mean and standard deviation of the vertical cost respectively, and t is the time.

[0039] As a preferred scheme of the improved binocular stereo matching fusion algorithm, a final disparity threshold τ4 is set in the output result of the adaptive learning and final matching cost aggregation formula.

[0040] When D(x, y) < τ4, the matching result of the pixel point is accurate;

[0041] When D(x, y) ≥ τ4, the matching result of the pixel point is inaccurate, and the matching cost needs to be adjusted or the disparity needs to be re-evaluated.

[0042] As a preferred scheme of the improved binocular stereo matching fusion algorithm, in the S3, the depth features extracted at different scales are fused, and a weighted average or an adaptive weighting method is used to generate a unified multi-scale feature map.

[0043] The beneficial effects of the present application are:

[0044] The present application can more comprehensively capture key information in the image, avoid matching errors caused by a single scale, improve the matching cost function to effectively reduce noise interference, especially in complex outdoor environments, improve the robustness to light and noise, and the adaptive learning mechanism can dynamically adjust the parameters and feature fusion strategy of the deep learning model through real-time feedback, so as to adaptively match according to environmental changes;

[0045] The aggregation and optimization steps of the matching cost can effectively reduce the false matching, and the generated disparity map has high precision and less noise. Compared with the existing Census algorithm, the present application reduces the over-reliance on a single pixel gray value by introducing multi-scale features and an adaptive learning mechanism, avoids noise and errors introduced by changes in the center pixel. At the same time, the optimized matching cost function enables the algorithm to maintain high precision and robustness in a noisy environment. The present application has higher matching precision, robustness and real-time adjustment capability, and solves the problems of serious false matching and insufficient real-time performance in the prior art in complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0047] Figure 1 The overall step flowchart of the improved binocular stereo matching fusion algorithm proposed by the present application is shown in the figure.

[0048] Figure 2 The performance comparison diagram of the improved binocular stereo matching fusion algorithm proposed by the present application is shown in the figure. DETAILED DESCRIPTION

[0049] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0050] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details presented herein. In other instances, well-known methods have not been described in detail in order to avoid obscuring aspects of the present application.

[0051] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an embodiment that is independent or alternative to other embodiments.

[0052] Thirdly, the present application is described in detail in conjunction with the schematic diagram. In the detailed description of the embodiments of the present application, the cross-sectional view of the device structure is locally enlarged without the general proportion for the convenience of description, and the schematic diagram is only an example, which should not limit the scope of protection of the present application herein. In addition, the three-dimensional spatial dimensions of length, width and depth should be included in actual manufacture.

[0053] Embodiment one

[0054] Reference Figure 1 For one embodiment of the present application, an improved binocular stereo matching fusion algorithm is provided, comprising the following steps:

[0055] S1: input image pre-processing;

[0056] The input left and right stereo images are pre-processed, such as denoising, white balance correction and illumination equalization, to ensure the stability of the image quality. Based on the pre-processed images, an image pyramid is constructed, and the images are scaled to multiple different scales to support subsequent multi-scale feature extraction and fusion;

[0057] S2: multi-scale depth feature extraction, the extracted features include local enhanced information under multiple scales, which can capture key features in various scenes;

[0058] S3: based on the fused multi-scale features, an improved matching cost function is designed;

[0059] The depth features extracted under different scales are fused, and weighted average or adaptive weighting method is used to generate a unified multi-scale feature map;

[0060] S4: the calculated matching cost is aggregated and optimized, and a disparity map is generated according to the optimized matching result;

[0061] S5: Introducing an adaptive learning mechanism to fine-tune the deep learning model and multi-scale feature fusion parameters through real-time feedback, outputting high-precision disparity maps;

[0062] Construct a luminance difference matching cost calculation formula, which is as follows:

[0063]

[0064] Where C1(x,y,d) represents the luminance difference matching cost at pixel (x,y), measuring the luminance difference between the left and right images under different disparity hypotheses d; I L (x,y) and I R (x-d,y) represent the pixel values of the left and right images at (x,y) and (x-d,y), respectively, and the disparity d represents the displacement between the left and right images; σ s is a multi-scale parameter for normalization, and the matching cost at different scales is normalized to the same range; p controls the degree of influence of the luminance difference.

[0065] Set the luminance difference matching cost threshold τ1 according to the output of the luminance difference matching cost calculation formula;

[0066] When C1(x,y,d) > τ1, it indicates that the luminance difference is too large, and the pixel matching corresponding to the disparity d is not accurate;

[0067] When C1(x,y,d) ≤ τ1, it indicates that the luminance difference is small, and the pixel matching corresponding to the disparity d is accurate.

[0068] Filter the output of the luminance difference matching cost calculation formula according to the set threshold τ1, and select the luminance difference matching cost with small luminance difference as the input to construct the multi-scale feature fusion formula, which is as follows:

[0069]

[0070] Where F s (x,y,d) represents the multi-scale feature fusion cost at pixel (x,y); F L s (x,y) and F R s (x-d,y) represent the feature values extracted from the left and right images at different scales s; w s is the scale weight coefficient, and the features at different scales have different influence weights; S is the total number of multi-scales.

[0071] Set the multi-scale feature fusion cost threshold τ2 according to the output of the multi-scale feature fusion formula;

[0072] When F s (x, y, d) > τ2, it means that the feature difference is large, and the matching is inaccurate;

[0073] When F s (x, y, d)≤τ2, it means that the feature difference is small, and the matching is accurate.

[0074] According to the set multi-scale feature fusion cost threshold τ2, the output result of the multi-scale feature fusion formula is filtered, and the multi-scale feature fusion cost with small multi-scale feature difference is selected as the input result to construct the smoothing constraint and gradient penalty formula, and the smoothing constraint and gradient penalty formula is as follows:

[0075]

[0076] Wherein, P(x, y, d) represents the matching cost after smoothing constraint and gradient penalty; The second-order partial derivative of the feature map in x and y direction, control the smoothness of the disparity map; λ represents the smoothing control parameter; α is the coefficient of the gradient penalty term, used to control the influence of the gradient of the depth feature on the matching cost; The penalty term of feature gradient, β is the learning rate.

[0077] In the output result of the smoothing constraint and gradient penalty formula, the matching cost threshold τ3 after smoothing constraint and gradient penalty is set;

[0078] When P(x, y, d) > τ3, it means that the matching cost of this point is high and not smooth, there is noise and mismatch, and the disparity needs to be adjusted;

[0079] When P(x, y, d)≤τ3, it means that the matching cost of this point is low and smooth, and the disparity does not need to be adjusted.

[0080] According to the output result of the smoothing constraint and gradient penalty formula, the adaptive learning and final matching cost aggregation formula is constructed, and the adaptive learning and final matching cost aggregation formula is as follows:

[0081]

[0082] Wherein, D(x, y) represents the final disparity value at pixel (x, y); γ is the adaptive learning control parameter, which determines the influence of historical data on the current matching result; The past matching cost is weighted by the adaptive learning mechanism, wherein T is the time step, μ t And σ t Respectively, the mean and standard deviation of the vertical cost, t is the time.

[0083] In the output result of the adaptive learning and final matching cost aggregation formula, the final disparity threshold τ4 is set.

[0084] When D(x, y) < τ4, the matching result of the pixel point is accurate;

[0085] When D(x, y) ≥ τ4, the matching result of the pixel point is inaccurate, and the matching cost needs to be adjusted or the disparity needs to be re-evaluated.

[0086] Embodiment two

[0087] Reference Figure 2 For the second embodiment of the application, which is different from the first embodiment, in order to verify the effect of the improved binocular stereo matching fusion algorithm proposed by the application, we carried out a comparative test. The goal of the test is to compare the performance of the existing stereo matching algorithm (represented by SGBM algorithm and BM algorithm) with the algorithm of the application, to evaluate the generation accuracy and robustness of the disparity map, and the test selects the public Middlebury stereo matching data set, which contains multiple sets of left and right stereo images with light and texture changes.

[0088] The left and right images in the data set containing noise and uneven light are denoised, white balance corrected and light equalized. The processed images are scaled to three different scales, namely the original size, 0.5 times and 0.25 times, to ensure feature extraction under multiple scales.

[0089] Features are extracted at three scales, including local edges, corner points and texture features. For each scale, we use convolutional neural network (CNN) to enhance the features, and through local normalization method to ensure the consistency of the features under different light conditions. We combine the features extracted under each scale, and through weighted average fusion to get a unified multi-scale feature map. The cost function uses a combination of weighted Hamming distance and absolute difference to calculate the matching cost of each pixel.

[0090] All matching costs are subjected to bilateral filtering and aggregation processing to eliminate matching errors. The aggregated cost map is further optimized to generate a disparity map. Through real-time feedback, the weighted coefficients of multi-scale feature fusion are adjusted, and based on the generated disparity map, the model parameters are dynamically fine-tuned to improve the accuracy of the disparity map.

[0091] Table one

[0092]

[0093] From the table data, it can be seen that the algorithm of the application is significantly better than the existing SGBM and BM algorithms in various performance indicators. The following four aspects are analyzed in detail.

[0094] Visual error rate: the average disparity error rate of the algorithm is only 3.5%, which is significantly lower than 7.2% of SGBM and 9.8% of BM. This shows that in complex environment, the algorithm can provide higher precision of disparity map and reduce error.

[0095] Mismatch rate: the mismatch rate of the algorithm is 2.1%, which is significantly lower than 4.5% of SGBM and 6.7% of BM. This improvement is due to the efficiency of multi-scale feature extraction and the application of improved matching cost function, which enables the algorithm to more accurately identify the corresponding points of left and right images and reduce the mismatch.

[0096] Robustness to illumination changes: the robustness of the algorithm under illumination changes is 85%, which is significantly improved compared with SGBM and BM. This is because the local normalization in multi-scale feature extraction enhances the adaptability of features in illumination change scenes.

[0097] Accuracy in noisy environment: in the scene with more noise, the accuracy of the algorithm is 82%, which is significantly better than 60% of SGBM and 52% of BM. This advantage is due to the multi-scale fusion strategy in the preprocessing and adaptive learning mechanism of the algorithm, which effectively reduces the influence of noise.

[0098] In summary, the present application effectively solves the problems of large error and poor robustness of existing algorithms in complex scenes through innovative design such as multi-scale deep feature extraction and adaptive learning mechanism.

[0099] The present application can more comprehensively capture key information in images, avoid matching errors caused by single scale, improve the matching cost function to effectively reduce noise interference, especially in complex outdoor environment, improve the robustness to light and noise, and the adaptive learning mechanism can dynamically adjust the parameters and feature fusion strategy of the deep learning model through real-time feedback, so as to adaptively match according to environmental changes.

[0100] The aggregation and optimization of matching cost can effectively reduce mismatch, and the generated disparity map has high precision and less noise. Compared with the existing Census algorithm, the present application reduces the over-reliance on single pixel gray value by introducing multi-scale features and adaptive learning mechanism, avoids noise and error introduced by center pixel change, and at the same time, the optimized matching cost function enables the algorithm to maintain high precision and robustness in noisy environment. The present application has higher matching precision, robustness and real-time adjustment ability, solves the problems of serious mismatch and insufficient real-time performance in complex environment in the prior art.

[0101] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An improved binocular stereo matching fusion method, characterized in that: The following steps are involved: S1: input image preprocessing; S2: Multi-scale deep feature extraction and feature enhancement using convolutional neural networks; S3: Based on the fused multi-scale features, an improved matching cost function is designed; Construct a brightness difference matching cost calculation formula, which is as follows: in, represents the brightness difference matching cost at pixel (x, y), which measures the brightness difference between the left and right images under different disparity assumptions d; and represents the pixel values ​​of the left and right images, located at (x, y) and (xd, y) respectively, and the disparity d represents the displacement between the left and right images; A multi-scale parameter used for normalization. The matching costs at different scales are normalized to the same range. p is used to control the impact of brightness differences. According to the output result of the brightness difference matching cost calculation formula, the brightness difference matching cost threshold is set as ; when > When , it means that the brightness difference is too large, and the pixel points corresponding to the disparity d are not accurately matched; when ≤ When , it means that the brightness difference is small, and the pixel points corresponding to the disparity d are accurately matched; According to the set threshold The output result of the brightness difference matching cost calculation formula is filtered, and the brightness difference matching cost with small brightness difference is selected as the input result to construct the multi-scale feature fusion formula, and the multi-scale feature fusion formula is as follows: in, Represents the multi-scale feature fusion cost at pixel (x, y); and represents the feature values ​​extracted from the left and right images at different scales s; is the scale weight coefficient, and features at different scales have different influence weights; S is the total number of multiple scales; Set the multi-scale feature fusion cost threshold according to the output of the multi-scale feature fusion formula ; when > When , it means that the features are very different and the matching is inaccurate; when ≤ When , it means that the feature difference is small and the matching is accurate; S4: Aggregate and optimize the calculated matching costs, and generate a disparity map based on the optimized matching results; According to the set multi-scale feature fusion cost threshold The output results of the multi-scale feature fusion formula are filtered, and the multi-scale feature fusion cost with small multi-scale feature differences is selected as the input result to construct the smoothing constraint and gradient penalty formula. The smoothing constraint and gradient penalty formula are as follows: in, Represents the matching cost after smoothness constraint and gradient penalty; Represents the second-order partial derivatives of the feature map in the x and y directions, controlling the smoothness of the disparity map; represents the smoothing control parameter; is the coefficient of the gradient penalty term, which is used to control the impact of the gradient of the deep feature on the matching cost; represents the penalty term of feature gradient, is the learning rate; S5: Introducing an adaptive learning mechanism to fine-tune the deep learning model and multi-scale feature fusion parameters through real-time feedback to output a high-precision disparity map; The adaptive learning and final matching cost aggregation formula is constructed based on the output results of the smoothness constraint and the gradient penalty formula, and the adaptive learning and final matching cost aggregation formula is as follows: in, Represents the final disparity value at pixel (x, y); To adaptively learn control parameters and determine the impact of historical data on current matching results; The past matching costs are weighted by the adaptive learning mechanism, where T is the number of time steps. and are the mean and standard deviation of historical costs respectively, and t is the time.

2. The improved binocular stereo matching fusion method according to claim 1, characterized in that: Set the matching cost threshold after smooth constraint and gradient penalty in the output result of the smooth constraint and gradient penalty formula ; when > , indicating that the matching cost of this point is high and not smooth, there is noise and mismatch, and the disparity needs to be adjusted; when ≤ , indicating that the matching cost of this point is low and smooth, and no parallax adjustment is required.

3. The improved binocular stereo matching fusion method according to claim 1, characterized in that: The final disparity threshold is set in the output of the adaptive learning and final matching cost aggregation formula as ; when < , then the matching result of the pixel is accurate; when ≥ , then the matching result of this pixel is inaccurate, and the matching cost needs to be adjusted or the disparity needs to be re-evaluated.

4. The improved binocular stereo matching fusion method according to claim 1, characterized in that: In S3, the deep features extracted at different scales are fused, and a weighted average or adaptive weighting method is used to generate a unified multi-scale feature map.

Images