A binocular parallax estimation method and its application

Abstract

The present invention discloses a binocular disparity estimation method, which includes: constructing a disparity space and designating the starting position point as the current node; taking the current node as the center point, designating the pixel points within the N-neighborhood as subsequent nodes to form N paths, using a penalty term to correct the cost value of the subsequent nodes at a single disparity value to obtain a value to be updated and an updated feature value; accumulating the feature value into the path cumulative value, and designating the path corresponding to the minimum path cumulative value as the extended path; obtaining the extended paths corresponding to different disparity values, judging whether to continue updating the path according to the path distance, and iterating the feature value; only saving the disparity corresponding to the minimum feature value among the points where the feature value changes to complete the disparity estimation. The cost disparity estimation result of this method is more accurate and can still maintain effectiveness in images with few texture features and low contrast; at the same time, this method also reduces the number of iterations and the amount of calculation data, and has the characteristics of fast calculation speed and high robustness.

Description

Technical Field

[0001] The present invention relates to the field of disparity estimation, and in particular to a binocular disparity estimation method and application thereof. Background Art

[0002] Binocular vision systems are currently widely used in the fields of robot-guided grasping and autonomous driving. They are used to obtain the depth information and 3D point cloud of the measured target with high precision. Among them, when there are no measured features such as coded points and structured light, it is an effective way to calculate the depth information using the disparity estimation method. The disparity estimation method can match the corresponding points of the binocular cameras with the help of the position deviation of the left and right camera imaging pixels, supplemented by the camera stereo imaging model. Traditional binocular disparity estimation methods usually use four steps: feature extraction, cost calculation, cost aggregation, and disparity optimization. For example, the patent document CN107316326A proposes an edge-based disparity map calculation method and device for binocular stereo vision; it solves the minimum cost value and the second minimum cost value through the above four steps, and then uses the two to evaluate whether the pixel point is in the high-level texture area, and then solves the correct disparity based on the high-level texture area.

[0003] Another disparity estimation method uses a network model for cost aggregation, such as a binocular disparity calculation method based on a 3D convolutional neural network proposed in patent document CN110060290A. This method extracts features through a CNN network and performs cost aggregation to ultimately estimate the disparity value.

[0004] When the above disparity estimation method is applied to images with prominent feature textures and good contrast, the accuracy of 3D reconstruction is high. However, when there are occlusions, large texture-free areas (such as the sky and walls in outdoor images), high-brightness reflective surface areas and repeated textures in the image, the estimated disparity value has large noise and deviation, which reduces the subsequent 3D solution results. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a binocular disparity estimation method. The cost aggregation process of the method adopts the Bellman-Ford iterative idea and carries multi-directional cost calculation. The accuracy of the disparity estimation result is more reliable, and the effectiveness of the disparity estimation result can still be maintained in images with few texture features and low contrast. At the same time, the method also limits the starting position point and the ending position point, and only needs to iterate the path of the feature area (the foreground area where the object to be measured is located), thereby reducing the number of iterations and the amount of calculated data, and has the characteristics of fast calculation speed and high robustness.

[0006] To this end, the technical solution of the present invention is as follows:

[0007] A binocular disparity estimation method comprises the following steps:

[0008] Step 1: Calculate the cost values of each pixel point at different disparities based on the left image and the right image collected by the binocular system to form a disparity space; in the disparity space, a single pixel point contains the pixel coordinates of this point, as well as multiple disparity values and cost values with a one-to-one correspondence, and the number of disparity values and cost values at each pixel point is the same;

[0009] Step 2: Mark the starting position point of the object to be measured in the left image / right image

[0010] Match a feature value for each cost value in the disparity space; the value of the feature value of the starting position point is consistent with the corresponding cost value; the initial values of the feature values of other pixel points are all set to the value A;

[0011] Record the starting position point as the current node; starting from the starting position point, search for paths backward;

[0012] Step 3: Arbitrarily select a disparity value and denote it as disparity value d A ;

[0013] Taking the current node as the center point, mark all other pixel points within the N-neighborhood as subsequent nodes; combine the current node and each subsequent node respectively to form N paths, and perform the following calculations on each path:

[0014] ① Use the product of the angle change amount and the disparity change amount of the path at the current node as a penalty term, and use the penalty term to correct the cost value C of the subsequent node at the disparity value d A to obtain the value C to be updated of the subsequent node at the disparity value d dA现有 ; A ; dA更新 ;

[0015] ② Judge whether C dA更新 is less than the feature value of the subsequent node at the disparity value d A : if so, replace the feature value with C dA更新 and denote it as the new feature value, if not, keep the feature value unchanged and delete C dA更新 ;

[0016] ③ Accumulate the feature value obtained in step ② into the existing path cumulative value to obtain a new path cumulative value, and the existing path cumulative value is: the sum of the feature values of all nodes between the starting position point and the current node in the current path at the disparity value d A ;

[0017] Statistically calculate the new path cumulative values corresponding to each path at the disparity value d A , and denote the path corresponding to the minimum path cumulative value as the extended path;

[0018] ④ Set the current disparity value dA Label the processed disparity. Arbitrarily select another unlabeled disparity value and denote it as the disparity value d A , and use the new disparity value d A to continue jumping to and executing step ① until all disparity values are labeled as processed disparities, and then execute step four;

[0019] Step four: Make the following judgments for each extended path separately:

[0020] Calculate the distance between the starting position point and the subsequent node in the extended path and denote it as the path distance. Determine whether the path distance is greater than the preset threshold. If so, execute step five. If not, update the current node in the extended path to the previous node, update the subsequent node to the current node, and jump back to step three to find the next node in the path until the calculated path distance is greater than the preset threshold;

[0021] Step five: If the feature value of a single pixel point at different disparities is always the initial value assigned in step two, then mark this pixel point as a background point and set its disparity value to zero;

[0022] Mark the points where the feature value changes as foreground points. In the disparity space of each foreground point, only save the disparity corresponding to the minimum feature value and delete other disparities to complete disparity estimation.

[0023] Furthermore, in step ①, the value C to be updated at the subsequent node at the disparity value d A is calculated as follows: dA更新

[0024] C dA更新 = C dA现有 + S (dA) - L dA(当前节点)

[0025] where C dA现有 represents the cost value at the subsequent node at the disparity value d A , L dA(当前节点) represents the cost value at the current node at the disparity value d A ;

[0026] S (dA) = MIN(L di(当前节点) + λ(d i - d A ) × e θ × P0)

[0027] where d i represents the i-th disparity value, i = 1, 2... M, and M represents the total number of disparity values in the preset disparity search range. L di(当前节点) represents the cost value at the current node at the disparity d i ​The cost value corresponding to the position, λ represents a preset penalty coefficient, θ represents the angle between the line segment formed by the previous node and the current node and the line segment formed by the current node and the subsequent node. If the current node is the starting position point, then θ = 1; P0 represents a penalty factor, which is a preset value and is an integer greater than 100.

[0028] Traverse all the disparity values d i and assign the minimum value of this formula to S (dA) .

[0029] In order to make the finally estimated disparity value smoother and without mutations, before step five, the following steps are also carried out:

[0030] Step A: Mark the end position point of the object to be measured in the image to be matched, and record it as the current node; start from the end position point and search for the path forward;

[0031] Match a feature value for each cost value in the disparity space obtained in step one; the value of the feature value of the end position point is the same as its corresponding cost value; the initial values of the feature values of other pixel points are the same and this initial value is a value greater than 5000;

[0032] Step B: Arbitrarily select a disparity value and denote it as the disparity value d A ';

[0033] Taking the current node as the center point, record other pixel points within the N-neighborhood as subsequent nodes; combine the current node and each subsequent node respectively to form N paths, and perform the following calculations for each path:

[0034] a1. Use the product of the angle change amount and the disparity change amount of the path at the current node as the penalty term, and use the penalty term to correct the cost value C A ' of the subsequent node at the disparity value d dA现有 ' to obtain the value C A ' to be updated of the subsequent node at the disparity value d dA更新 ';

[0035] a2. Judge whether C dA更新 ' is less than the feature value of the subsequent node at the disparity value d A ': If so, use C dA更新 ' to replace the feature value and denote it as the new feature value. If not, keep the feature value unchanged and delete C dA更新 ′;

[0036] If the feature value of the pixel point changes during this process and also changes during the execution of steps three to four, then add / take the average of the two changed feature values, and record the result value as the final feature value;

[0037] a3. Add the eigenvalue obtained in step a2 to the existing path cumulative value to obtain a new path cumulative value. The existing path cumulative value is the sum of the eigenvalues of all nodes between the starting position point and the current node in the current path at the disparity d A '.

[0038] Statistically calculate the new path cumulative values corresponding to each path at the disparity d A '. Mark the path corresponding to the minimum path cumulative value as the extended path;

[0039] a4. Mark the current disparity d A ' as the processed disparity. Arbitrarily select another unmarked disparity value and denote it as the disparity d A '. Use the new disparity d A ' to continue to jump and execute step a1 until all disparity values are marked as processed disparities, and then execute step C;

[0040] Step C: Make the following judgments for each extended path respectively:

[0041] Calculate the distance between the starting position point and the subsequent node in the extended path and denote it as the path distance. Determine whether the path distance is greater than the preset threshold. If so, execute step five. If not, update the current node in the extended path to the previous node, update the subsequent node to the current node, and jump back to step B to find the next node in this path until the calculated path distance is greater than the preset threshold.

[0042] Furthermore, in step one, calculate the cost values of each pixel point at different disparities based on the left image and the right image collected by the binocular system to form a disparity space. The method includes the following two types:

[0043] Method one:

[0044] 1) Denote the left image or the right image collected by the binocular system as the image to be matched, and denote the other image as the target image; Denote the pixel points in the image to be matched as the points to be matched, and denote the pixel points in the target image as the target points;

[0045] According to the disparity search range corresponding to each point to be matched, match a group of target points that meet the disparity search range for each point to be matched respectively. Calculate the cost values of the points to be matched at different disparities according to each target point, and store the pixel coordinates, multiple disparity values, and multiple cost values of the points to be matched correspondingly; Traverse each point to be matched in the image to be matched, and match the cost values of each point to be matched at different disparity values respectively;

[0046] 2) From the disparity search ranges corresponding to each point to be matched, find the maximum value d max of the disparity value. If the maximum disparity value dj-max Less than d max , then set the cost value corresponding to the parallax between d j-max +1 and d max to a numerical value A, where A is a preset value greater than 5000; so that each point to be matched has the same number of parallax values and cost values;

[0047] Summarize the parallax values and cost values of each point to be matched to obtain a parallax space (cost cube).

[0048] Among them, the parallax search range corresponding to each point to be matched is determined according to the following method:

[0049] Extract multiple feature points on the object to be measured from the left image and the right image respectively, perform stereo matching and binocular three-dimensional reconstruction to obtain the three-dimensional coordinates of the feature points;

[0050] Find the three-dimensional coordinates of each feature point in the three-dimensional coordinate system of the digital model workpiece of the object to be measured;

[0051] Based on the three-dimensional coordinates reconstructed by three-dimensional reconstruction and the three-dimensional coordinates in the three-dimensional coordinate system of the digital model workpiece, solve the conversion relationship RT between the three-dimensional space coordinate system of the binocular system and the three-dimensional coordinate system of the digital model workpiece based on rigid body transformation;

[0052] In the digital model of the object to be measured, sample a digital model point, and use the conversion relationship RT, the projection matrix of the left camera, and the distortion parameters of the left camera to convert the sampled digital model point to the left camera image plane to obtain the converted point coordinate p; use the conversion relationship RT, the projection matrix of the right camera, and the distortion parameters of the right camera to convert multiple digital model points to the right camera image plane respectively to obtain the converted point coordinate q;

[0053] Take the absolute value of the difference between the abscissas of the converted point coordinate p and the converted point coordinate q, and add the preset tolerance δ to the rounded absolute value of the difference to obtain the upper limit value d' of the parallax search range;

[0054] Then set the parallax search range of the pixel points in the left image whose distance from the converted point coordinate p is less than the threshold D to [0~d'];

[0055] In the digital model of the object to be measured, sample a digital model point every other threshold D, and process different digital model points in the same way to obtain the converted point coordinate p and the upper limit value d' corresponding to different digital model points;

[0056] According to different converted point coordinates p and upper limit values d', mark the parallax search ranges of each point to be matched in the left image.

[0057] Preferably, the threshold value D is taken as 30 to 200 pixels; δ is taken as 1 to 30 pixels, or δ is taken as (0.1 to 0.2)×H, where H represents the integer value obtained by taking the absolute value of the difference between the abscissas of the conversion point coordinates p and q.

[0058] Method 2:

[0059] Denote the left image or the right image collected by the binocular system as the image to be matched, and the other image as the target image; denote the pixel points in the image to be matched as the points to be matched, and the pixel points in the target image as the target points;

[0060] Assign the same disparity search range to each point to be matched, and then, according to the disparity search range, match a group of target points that meet the disparity search range for each point to be matched respectively. Calculate the cost values of the points to be matched at different disparities according to each target point respectively, and store the pixel coordinates, multiple disparity values, and multiple cost values of the points to be matched correspondingly; traverse each point to be matched in the image to be matched, and match the cost values of each point to be matched at different disparity values respectively to obtain the disparity space (cost cube).

[0061] Furthermore, the method for calculating the cost values of the points to be matched at different disparities according to each target point is the SAD method (sum of absolute differences of pixels), the SSD method (sum of squared differences of pixels), or the STAD method (sum of truncated absolute differences).

[0062] As an application of the disparity estimation value, use the saved disparity values to calculate the corresponding three-dimensional space coordinates (X, Y, Z) of each point to be matched (u, v):

[0063]

[0064] where, cx l and cx r are the pixel column coordinates of the principal points of the left and right images; f is the focal length, B is the baseline length between the left and right cameras, and D p is the saved disparity value at the pixel point (u, v).

[0065] Preferably, in step 3, the N neighborhood is a four-neighborhood, an eight-neighborhood, or a 16-neighborhood;

[0066] In step 4, the preset threshold is obtained in the following manner:

[0067] In the image, use the minimum circumscribed rectangle to select the area where the object to be measured is located, and denote the diagonal length of the minimum circumscribed rectangle as the preset threshold.

[0068] The path distance is obtained in the following manner:

[0069] Select a pixel point on the path in the middle area of the extension path, calculate the distances between this point and the starting position and the subsequent nodes respectively, and add the two distance values. The sum value is recorded as the path distance.

[0070] The present invention has the following beneficial effects:

[0071] 1) The cost aggregation of this method adopts the Bellman - ford path iteration idea, carrying out multi - direction cost calculation, and the disparity estimation result is more reasonable and accurate; especially in the case where the contrast between the measured object and the background is low and the texture features of the object to be measured are few, the disparity estimated by this method is more accurate.

[0072] 2) Based on the mutual conversion relationship between the two - dimensional image and the three - dimensional digital model, this method obtains the disparity search range. Compared with the existing methods that blindly expand the disparity search range in order to improve accuracy, this method reduces the amount of data in the cost calculation process. Compared with the traditional SGM binocular disparity estimation method, it can effectively improve the calculation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a schematic diagram of the images collected by the left and right cameras in the specific implementation manner;

[0074] Figure 2 It is a schematic diagram of the disparity in the specific implementation manner;

[0075] Figure 3 It is a schematic diagram of multiple disparities and cost values at each pixel point in the specific implementation manner. SPECIFIC IMPLEMENTATION MANNER

[0076] The technical solution of the present invention will be described in detail below with reference to the drawings and embodiments.

[0077] A binocular disparity estimation method includes the following steps:

[0078] Step 1: Calculate the cost values of each pixel point at different disparities based on the left image and the right image collected by the binocular system to form a disparity space; in the disparity space, as Figure 3 shown, a single pixel point includes the pixel coordinates of this point, as well as multiple corresponding disparity values and cost values, and the number of disparity values and cost values at each pixel point is the same;

[0079] A single pixel point includes the pixel coordinates of this point, as well as multiple corresponding disparity values and multiple cost values;

[0080] Step 2: As Figure 1 shown, mark the starting position point of the object to be measured in the left image / right image;

[0081] The starting position point is manually marked or, when detecting a batch of parts, according to the previous teaching information, the specific position coordinates of the starting position point in the image are preset, and the starting position point is marked according to these coordinates.

[0082] Match a feature value to each cost value in the disparity space; the value of the feature value of the starting position point is consistent with its corresponding cost value; the initial values of the feature values of other pixel points are all set to the value A;

[0083] Record the starting position point as the current node; starting from the starting position point, search for the path backward;

[0084] Step 3: Arbitrarily select a disparity value and denote it as the disparity value d A ;

[0085] Taking the current node as the center point, denote all other pixel points within the N-neighborhood as subsequent nodes; combine the current node and each subsequent node respectively to form N paths, and perform the following calculations on each path:

[0086] ① Use the product of the angle change amount and the disparity change amount of the path at the current node as the penalty term, and use the penalty term to correct the cost value C of the subsequent node at the disparity value d A at dA现有 , and obtain the value C to be updated of the subsequent node at the disparity value d A ; dA更新 ;

[0087] Specifically, the calculation method is as follows:

[0088] C dA更新 = C dA现有 + S (dA) - L dA(当前节点)

[0089] where C dA现有 represents the cost value of the subsequent node at the disparity value d A , and L dA(当前节点) represents the cost value of the current node at the disparity value d A ;

[0090] S (dA) = MIN(L di(当前节点) + λ(d i - d A ) × e θ × P0)

[0091] where d i represents the i-th disparity value, i = 1, 2... M, and M represents the total number of disparity values in the preset disparity search range, and L di(当前节点) represents the cost value of the current node at the disparity d iThe cost value corresponding to the place, λ represents a preset penalty coefficient, θ represents the angle between the line segment formed by the previous node and the current node and the line segment formed by the current node and the subsequent node. If the current node is the starting position point, then θ = 1 is taken; P0 represents a penalty factor, which is a preset value, and this value is an integer greater than 100.

[0092] Traverse all disparity values d i and assign the minimum value of this formula to S (dA) .

[0093] Among them, λ(d i - d A ) × e θ × P0 is the penalty term;

[0094] ② Judge whether C dA更新 is less than the eigenvalue of the subsequent node at the disparity value d A : If so, use C dA更新 to replace this eigenvalue and denote it as the new eigenvalue. If not, keep this eigenvalue unchanged and delete C dA更新 ;

[0095] ③ Accumulate the eigenvalues obtained in step ② into the existing path cumulative value to obtain a new path cumulative value. The existing path cumulative value is: the sum of the eigenvalues of all nodes between the starting position point and the current node in the current path at the disparity value d A ;

[0096] Statistical new path cumulative values corresponding to each path at the disparity value d A , and denote the path corresponding to the minimum path cumulative value as the extended path;

[0097] ④ Mark the current disparity value d A as the processed disparity. Arbitrarily select another unmarked disparity value and denote it as the disparity value d A , and continue to jump to execute step ① using the new disparity value d A until all disparity values are marked as processed disparities, and then execute step four;

[0098] The extended paths corresponding to different disparity values d A may be the same or different

[0099] Step four: Make the following judgments for each extended path respectively:

[0100] The distance between the starting position point and the subsequent node in the extended path is denoted as the path distance. Determine whether the path distance is greater than a preset threshold. If so, execute Step Five. If not, update the current node in the extended path to the previous node, update the subsequent node to the current node, and jump back to Step Three to find the next node in the path until the calculated path distance is greater than the preset threshold;

[0101] Step Five: If the eigenvalue of a single pixel at different disparities is always the initial value assigned in Step Two, then mark this pixel as a background point and set its disparity value to zero;

[0102] Mark the points where the eigenvalue changes as foreground points. In the disparity space of each foreground point, only save the disparity corresponding to the minimum eigenvalue, and delete other disparities to complete the disparity estimation.

[0103] To make the finally estimated disparity value smoother and without mutations, before Step Five, the following steps are also carried out: Step A: Mark the end position point of the object to be measured in the image to be matched (such as Figure 1 ), and mark it as the current node; this process starts from the end position point and searches for the path forward;

[0104] Match a feature value to each cost value in the disparity space obtained in Step One; the value of the eigenvalue of the end position point is consistent with its corresponding cost value; the initial values of the eigenvalues of other pixel points are the same and this initial value is a value greater than 5000;

[0105] Step B: Arbitrarily select a disparity value and denote it as the disparity value d A ';

[0106] Taking the current node as the center point, mark all other pixel points within the N-neighborhood as subsequent nodes; combine the current node and each subsequent node respectively to form N paths, and perform the following calculations for each path:

[0107] a1. Use the product of the angle change amount and the disparity change amount of the path at the current node as the penalty term, and use this penalty term to correct the cost value C A ' of the subsequent node at the disparity value d dA现有 ' to obtain the value C A ' to be updated of the subsequent node at the disparity value d dA更新 ';

[0108] a2. Determine whether C dA更新 ' is less than the eigenvalue of the subsequent node at the disparity value d A ': If so, use C dA更新 ' to replace this eigenvalue and denote it as the new eigenvalue. If not, keep this eigenvalue unchanged and delete C dA更新 ′;

[0109] If the eigenvalue of the pixel point changes during this process and also changes during the execution of Step 3 to Step 4, then add or take the average of the eigenvalues of the two changes, and record the result value as the final eigenvalue;

[0110] a3. Accumulate the eigenvalue obtained in step a2 into the existing path cumulative value to obtain a new path cumulative value. The existing path cumulative value is: the sum of the eigenvalues of all nodes between the starting position point and the current node in the current path at the disparity d A '.

[0111] Statistically calculate the new path cumulative values corresponding to each path at the disparity d A ', and record the path corresponding to the minimum path cumulative value as the extended path;

[0112] a4. Mark the current disparity d A ' as the processed disparity, randomly select another unmarked disparity value, denoted as the disparity d A ', and use the new disparity d A ' to continue to jump and execute step a1 until all disparity values are marked as processed disparities, and then execute step C;

[0113] Step C: Make the following judgments for each extended path respectively:

[0114] Calculate the distance between the starting position point and the subsequent node in the extended path, denoted as the path distance, and determine whether the path distance is greater than the preset threshold. If so, execute step five. If not, update the current node in the extended path to the previous node, update the subsequent node to the current node, and jump back to step B to find the next node in this path until the calculated path distance is greater than the preset threshold.

[0115] Specifically, in step one, based on the left image and the right image collected by the binocular system, calculate the cost values of each pixel point at different disparities to form a disparity space. The method includes the following two types:

[0116] Method 1:

[0117] 1) Denote the left image or the right image collected by the binocular system as the image to be matched, and denote the other image as the target image; denote the pixel points in the image to be matched as the points to be matched, and denote the pixel points in the target image as the target points;

[0118] According to the disparity search range corresponding to each to-be-matched point, a group of target points that meet the disparity search range are matched for each to-be-matched point, and the cost value of the to-be-matched point under different disparities is calculated according to each target point, and the pixel coordinates of the to-be-matched point, multiple disparity values, and multiple cost values are correspondingly stored; each to-be-matched point in the to-be-matched image is traversed, and the cost value under different disparity values is matched for each to-be-matched point;

[0119] 2) Find the maximum disparity value d from the disparity search range corresponding to each point to be matched max , if the maximum disparity d of a certain point to be matched j-max Less than d max , then d j-max +1 with d max The cost value corresponding to the disparity between is set to value A, where value A is a preset value greater than 5000, so that each point to be matched has the same number of disparity values and cost values;

[0120] The disparity values and cost values of each point to be matched are summed up to obtain the disparity space (cost cube).

[0121] That is, the original number of cost values at individual pixel points is insufficient, and the value A needs to be used to make up for it so that the number of its cost values is consistent with the number of cost values of other pixel points.

[0122] In order to reduce the amount of cost value calculation, as a preferred implementation, different disparity values are matched for each pixel point based on digital-analog point back projection, and the specific method is as follows:

[0123] Extract multiple feature points on the object to be tested from the left image and the right image respectively, perform stereo matching and binocular 3D reconstruction, and obtain the 3D coordinates of the feature points;

[0124] More specifically, a SIFT, SURF feature point extraction method or a template matching method based on NCC grayscale features is used to extract edge points of the object to be measured as feature points; a left image feature point set and a right image feature point set are formed, and stereo matching and three-dimensional reconstruction are performed based on the two feature point sets, left camera intrinsic parameters, right camera intrinsic parameters, distortion parameters, and external parameters between the left and right cameras to obtain the three-dimensional coordinates of the feature points;

[0125] Find the three-dimensional coordinates of each feature point in the digital model workpiece coordinate system of the object to be measured;

[0126] Based on the three-dimensional coordinates reconstructed by three-dimensional reconstruction and the three-dimensional coordinates in the digital model workpiece coordinate system, the conversion relationship RT between the three-dimensional space coordinate system of the binocular system and the digital model workpiece coordinate system is solved based on the rigid body transformation;

[0127] In the digital model of the object to be measured, sample a digital model point. Using the transformation relationship RT, the projection matrix of the left camera, and the distortion parameters of the left camera, transform the sampled digital model point onto the image plane of the left camera to obtain the transformed point coordinates p. Using the transformation relationship RT, the projection matrix of the right camera, and the distortion parameters of the right camera, transform multiple digital model points onto the image plane of the right camera respectively to obtain the transformed point coordinates q.

[0128] Take the absolute value of the difference between the abscissa of the transformed point coordinates p and the abscissa of the transformed point coordinates q, round the absolute value of the difference, and add the preset tolerance δ to obtain the upper limit value d' of the disparity search range.

[0129] Then set the disparity search range of the pixel points in the left image whose distance from the transformed point coordinates p is less than the threshold D to [0~d'].

[0130] In the digital model of the object to be measured, sample a digital model point every interval of the threshold D, process different digital model points in the same way, and obtain the transformed point coordinates p and the upper limit value d' corresponding to different digital model points.

[0131] According to different transformed point coordinates p and the upper limit value d', mark the disparity search ranges of each point to be matched in the left image.

[0132] For example: sample a digital model point every 100 pixels (threshold D), project it onto the image plane of the left camera to obtain the transformed point coordinates p, and set the disparity search range of the pixel points in the left image whose distance from the transformed point coordinates p is less than the threshold D (100 pixels) to [0~d'].

[0133] Specifically, the value of the threshold D is 30~200 pixels; the value of δ is 1~30 pixels, or the value of δ is (0.1~0.2)×H, where H represents the integer part of the absolute value of the difference between the abscissa of the transformed point coordinates p and the abscissa of the transformed point coordinates q.

[0134] Method 2:

[0135] Record the left image or the right image collected by the binocular system as the image to be matched, and record the other image as the target image; record the pixel points in the image to be matched as the points to be matched, and record the pixel points in the target image as the target points.

[0136] This method is a common method. Assign the same disparity search range to each point to be matched, and then according to the disparity search range, match a group of target points that meet the disparity search range for each point to be matched respectively. Calculate the cost values of the points to be matched at different disparities according to each target point, and store the pixel coordinates, multiple disparity values, and multiple cost values of the points to be matched correspondingly. Traverse each point to be matched in the image to be matched, and match the cost values at different disparity values for each point to be matched respectively to obtain the disparity space (cost cube).

[0137] Among them, the method for calculating the cost values of the points to be matched at different disparities according to each target point is the SAD method (Sum of Absolute Differences of pixels), the SSD method (Sum of Squared Differences of pixels), or the STAD method (Truncated Sum of Absolute Differences).

[0138] In specific implementation, in step three, the N neighborhood is a four-neighborhood, an eight-neighborhood, or a 16-neighborhood; when the N neighborhood is a four-neighborhood, four paths are iterated simultaneously; when the N neighborhood is an eight-neighborhood, eight paths are iterated simultaneously; when the N neighborhood is a 16-neighborhood, 16 paths are iterated simultaneously;

[0139] In step four, the preset threshold is obtained in the following manner:

[0140] In the image, the region where the object to be measured is located is selected by using the minimum circumscribed rectangle, and the diagonal length of the minimum circumscribed rectangle is recorded as the preset threshold.

[0141] The path distance is obtained in the following manner:

[0142] In the middle region of the extended path, a pixel point on the path is selected, and the distances between this point and the starting position and the subsequent node are calculated respectively, and the two distance values are added, and the sum value is recorded as the path distance.

[0143] Alternatively, the number of pixel points in the extended path can be directly recorded as the path distance.

[0144] As a subsequent application of the disparity estimation value, the three-dimensional space coordinates (X, Y, Z) corresponding to each point to be matched (u, v) are calculated by using the saved disparity values:

[0145]

[0146] wherein, cx l and cx r are the pixel column coordinates of the principal points of the left and right images; f is the focal length, B is the baseline length between the left and right cameras, and D p is the saved disparity value at the pixel point (u, v).

[0147] Next, taking the binocular reconstruction of a pantograph as an example, an exemplary elaboration is carried out:

[0148] Step 1: Based on the left image and the right image collected by the binocular system (such as Figure 1 ), a disparity space is constructed. In the disparity space (cost cube), a single pixel point contains the pixel coordinates of this point, as well as multiple disparity values and cost values with a one-to-one correspondence, and the number of disparity values and cost values at each pixel point is the same;

[0149] For example, each pixel point includes the pixel coordinates of this point, as well as 3 disparity values (d = 0, d = 1, d = 2) and 3 cost values with a one-to-one correspondence;

[0150] Step 2: As shown in Figure 1 , mark the starting position point (the circled area on the left) of the object to be measured (pantograph) in the left image / right image;

[0151] Starting from the starting position point, perform path iteration backward;

[0152] Match a feature value to each cost value in the disparity space; the value of the feature value at the starting position point is the same as its corresponding cost value; assign an initial value of 10000 (numerical value A) to the feature values of other pixel points;

[0153] Step 3: Arbitrarily select a disparity value from the preset disparity search range [0 - 2] and denote it as the disparity value d A ; if d = 0 is selected, denote it as the disparity value d A ;

[0154] Taking the current node as the center point, mark all other pixel points within the eight-neighborhood as subsequent nodes; combine the current node and each subsequent node respectively to form eight paths, and perform the following calculations for each path:

[0155] ① Calculate the value C to be updated at the subsequent node at the disparity value d A (d = 0) dA更新 = C dA现有 + S (dA) - L dA(当前节点) ; where C dA现有 represents the cost value at the subsequent node at the disparity value d A (d = 0), and L dA(当前节点) represents the cost value at the current node at the disparity value d A ;

[0156] S (dA) = MIN(L di(当前节点) + λ(d i - d A ) × e θ × P0)

[0157] where d i represents the i-th disparity value, i = 1, 2... M, and M represents the total number of disparity values in the preset disparity search range, L di(当前节点) represents the cost value corresponding to the current node at the disparity d i , λ represents the preset penalty coefficient, θ represents the angle between the line segment formed by the previous node and the current node and the line segment formed by the current node and the subsequent node. If the current node is the starting position point, then take θ = 1; P0 represents the penalty factor, which is a preset value and is a multiple of 10;

[0158] Traverse all disparity values d i, and assign the minimum value of this formula to S (dA) ;

[0159] That is: calculate respectively: L d＝0(当前节点) +λ(0 - 0)×e θ ×P0, L d＝1(当前节点) +λ(1 - 0)×e θ ×P0, L d＝2(当前节点) +λ(2 - 0)×e θ ×P0. Assign the minimum value among the calculation results of these three formulas to S (dA) .

[0160] ② Judge whether C dA更新 is less than the eigenvalue of the subsequent node at the disparity value d A : If so, use C dA to update and replace this eigenvalue and record it as the new eigenvalue (shrink and update the eigenvalue from the initial value of 10000 to C dA update), if not, keep this eigenvalue unchanged and delete C dA update;

[0161] ③ Add the eigenvalue obtained in step ② to the existing path cumulative value to get the new path cumulative value. The existing path cumulative value is: the sum of the eigenvalues of all nodes between the starting position point and the current node in the current path at the disparity value d A ;

[0162] Statistically calculate the new path cumulative values corresponding to each path at the disparity value d A , and record the path corresponding to the minimum path cumulative value as the extended path; if the path cumulative value of path 2 is the smallest, then record it as the extended path;

[0163] ④ Mark the current disparity value d A (d = 0) as the processed disparity, and randomly select an unmarked disparity value in the preset disparity search range, such as d = 1, and record it as the disparity value d A , and continue to jump and execute step ① using the new disparity value d A ;

[0164] ① Calculate the value C A to be updated at the subsequent node at the disparity value d dA更新 (d = 1): C dA现有 = C (dA) + S dA(当前节点) - L dA现有 ; where, C A represents the cost value of the subsequent node at the disparity value d dA(当前节点) (d = 1), and L A represents the cost value of the current node at the disparity value d

[0165] S(dA) = MIN(L di(当前节点) + λ(d i - d A ) × e θ ) × P0)

[0166] Traverse all disparity values d i , and assign the minimum value of this formula to S (dA) ;

[0167] That is, calculate respectively: L d＝0(当前节点) + λ(0 - 1) × e θ × P0, L d＝1(当前节点) + λ(1 - 1) × e θ × P0, L d＝2(当前节点) + λ(2 - 1) × e θ × P0. Assign the minimum value among the calculation results of these three formulas to S (dA) .

[0168] ② Judge whether C dA更新 is less than the eigenvalue of the subsequent node at the disparity value d A (d = 1): If so, use C dA更新 to replace this eigenvalue and record it as the new eigenvalue (reduce the eigenvalue from the initial value 10000 and update it to C dA更新 ), if not, keep this eigenvalue unchanged and delete C dA更新 ;

[0169] ③ Accumulate the eigenvalue obtained in step ② to the existing path cumulative value to obtain a new path cumulative value. The existing path cumulative value is: the sum of the eigenvalues of all nodes between the starting position point and the current node in the current path at the disparity value d A ((d = 1));

[0170] Statistically calculate the new path cumulative values corresponding to each path at the disparity value d A , and record the path corresponding to the minimum path cumulative value as the extended path; if the path cumulative value of path 5 is the smallest, then record it as the extended path;

[0171] ④ Mark the current disparity value d A as the processed disparity, and randomly select an unmarked disparity value in the preset disparity search range. For example, d = 2 is recorded as the disparity value d A , and continue to jump to execute step ① using the new disparity value d A ;

[0172] ① Calculate the value C A to be updated at the disparity value d dA更新 = C dA现有 + S (dA) - LdA(当前节点) ; where C dA现有 represents the cost value of the subsequent node at the disparity value d A (d = 2), and L dA(当前节点) represents the cost value of the current node at the disparity value d A ;

[0173] S (dA) = MIN(L di(当前节点) + λ(d i - d A ) × e θ × P0)

[0174] Traverse all the disparity values d i , and assign the minimum value of this formula to S (dA) ;

[0175] That is, calculate respectively: L d＝0(当前节点) + λ(0 - 2) × e θ × P0, L d＝1(当前节点) + λ(1 - 2) × e θ × P0, L d＝2(当前节点) + λ(2 - 2) × e θ × P0, and assign the minimum value among the calculation results of these three formulas to S (dA) .

[0176] ② Judge whether C dA更新 is less than the feature value of the subsequent node at the disparity value d A (d = 2): If so, use C dA更新 to replace this feature value and record it as the new feature value (reduce the feature value from the initial value 10000 and update it to C dA更新 ), if not, keep this feature value unchanged and delete C dA更新 ;

[0177] ③ Add the feature value obtained in step ② to the existing path cumulative value to obtain the new path cumulative value. The existing path cumulative value is: the sum of the feature values of all nodes between the starting position point and the current node in the current path at the disparity value d A (d = 2);

[0178] Statistically calculate the new path cumulative values corresponding to each path at the disparity value d A , and record the path corresponding to the minimum path cumulative value as the extended path; if the path cumulative value of path 1 is the smallest, record it as the extended path;

[0179] ④ Mark the current disparity value d A (d = 2) as the processed disparity, and execute step four;

[0180] Step 4. Make the following judgments for each extension path (Path 2, Path 5, Path 1) respectively:

[0181] Calculate the distance between the starting position point and the subsequent node in the extension path and denote it as the path distance. Determine whether the path distance is greater than the preset threshold. If so, execute Step 5. If not, update the current node in the extension path to the previous node, update the subsequent node to the current node, jump back to Step 3, and search for the next node in this path until the calculated path distance is greater than the preset threshold;

[0182] Step 5. If the eigenvalue of a single pixel at different disparities is always the initial value assigned in Step 2, then mark this pixel as a background point and set its disparity value to zero;

[0183] Mark the point where the eigenvalue changes as a foreground point. As Figure 2 shown, only the disparity corresponding to the minimum eigenvalue is saved in the disparity space of each foreground point, and other disparities are deleted to complete the disparity estimation.

[0184] The cost aggregation of this method adopts the path iteration idea of Bellman - ford, carrying multi - direction cost calculation, and the disparity estimation result is more reasonable and accurate; especially in the case where the contrast between the measured object and the background is low and the texture features of the object to be measured are few (such as pantograph detection), the disparity estimated by this method is more accurate.

[0185] The previous description of the specific exemplary embodiments of the present invention has been presented for purposes of illustration and description. The foregoing description is not intended to be exhaustive nor to limit the invention to the precise form disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application so that others skilled in the art may realize and utilize the various exemplary embodiments of the invention and their various alternative forms and modifications. The scope of the invention is intended to be defined by the appended claims and their equivalents.

Claims

1. A binocular parallax estimation method, characterized in that, It includes the following steps: Step 1: Calculate the cost values of each pixel at different disparities based on the left image and the right image collected by the binocular system, and form a disparity space. In the disparity space, a single pixel contains the pixel coordinates of this point, as well as multiple disparity values and cost values with a one-to-one correspondence, and the number of disparity values and cost values at each pixel is the same. Step 2: Mark the starting position point of the object to be measured in the left image / right image. Match a feature value to each cost value in the disparity space; the value of the feature value of the starting position point is consistent with its corresponding cost value; the initial values of the feature values of other pixel points are all set to the value A. Record the starting position point as the current node. Step 3: Arbitrarily select a parallax value and denote it as parallax value d A ; Taking the current node as the center point, record all other pixel points within the N-neighborhood as subsequent nodes; combine the current node and each subsequent node respectively to form N paths, and perform the following calculations for each path: ① Use the product of the angular change amount and the parallax change amount of the path at the current node as a penalty term, and use the penalty term to correct the cost value C of the subsequent node at the parallax value d A to obtain the value C to be updated of the subsequent node at the parallax value d dA现有 ; A dA更新 ;​ ② Determine C dA更新 whether it is less than the eigenvalue at the disparity value d A of the subsequent node: If so, use C dA更新 to replace this eigenvalue and denote it as the new eigenvalue; if not, keep this eigenvalue unchanged and delete C dA更新 ; ③ Add the eigenvalue obtained in step ② to the existing path cumulative value to obtain a new path cumulative value. The existing path cumulative value is the sum of the eigenvalues of all nodes between the starting position point and the current node in the current path at the disparity value d A ; Statistically determine the new cumulative path values corresponding to each path at the disparity value d A and record the path corresponding to the minimum cumulative path value as the extended path; ④ Label the current parallax value d A as the processed parallax, and randomly select another unlabeled parallax value, denoted as parallax value d A . Use the new parallax value d A to continue jumping to execute step ① until all parallax values are marked as processed parallaxes, and then execute step four; Step 4: Make the following judgments for each extended path respectively: Calculate the distance between the starting position point and the subsequent node in the extended path and record it as the path distance, and judge whether the path distance is greater than the preset threshold. If so, execute Step 5. If not, update the current node in the extended path to the previous node, update the subsequent node to the current node, and jump back to Step 3 to find the next node in this path until the calculated path distance is greater than the preset threshold. Step 5: If the feature value of a single pixel at different disparities is always the initial value assigned in Step 2, mark this pixel as a background point and set its disparity value to zero. Mark the points where the feature value changes as foreground points, and only save the disparity corresponding to the minimum feature value in the disparity space of each foreground point, and delete other disparities to complete the disparity estimation.

2. The method according to claim 1, wherein: In step ①, the value C to be updated at the subsequent node for the disparity value d A is calculated as follows: dA更新 ​ C dA更新 = C dA现有 + S (dA) - L dA(当前节点) Among them, C dA现有 represents the cost value of the subsequent node at the disparity value d A , and L dA(当前节点) represents the cost value of the current node at the disparity value d A ; S (dA) = MIN(L di(当前节点) + λ(d i - d A ) × e θ × P0) Among them, d i represents the i-th disparity value, where i = 1, 2,..., M, and M represents the total number of disparity values in the preset disparity search range. L di(当前节点) represents the cost value corresponding to the current node at the disparity d i , λ represents the preset penalty coefficient, θ represents the angle between the line segment formed by the previous node and the current node and the line segment formed by the current node and the subsequent node. If the current node is the starting position point, then θ = 1; P0 represents the penalty factor, which is a preset value and is an integer greater than 100; Traverse all the disparity values d i , and assign the minimum value of this expression to S (dA) .

3. The method according to claim 1, characterized in that: Before Step 5, the following steps are also performed: Step A: Mark the ending position point of the object to be measured in the image to be matched, and record it as the current node. Match a feature value to each cost value in the disparity space obtained in Step 1; the value of the feature value of the ending position point is consistent with its corresponding cost value; the initial values of the feature values of other pixel points are the same and this initial value is a value greater than 5000. Step B: Arbitrarily select a parallax value and denote it as parallax value d A '; Taking the current node as the center point, record all other pixel points within the N-neighborhood as subsequent nodes; combine the current node and each subsequent node respectively to form N paths, and perform the following calculations for each path: a1. Use the product of the angular change amount and the parallax change amount of the path at the current node as a penalty term, and use the penalty term to correct the cost value C of the subsequent node at the parallax value d A ', to obtain the value C to be updated of the subsequent node at the parallax value d dA现有 '; A ' dA更新 '; a2. Determine C dA更新 ' whether it is less than the feature value at the disparity value d A ' of the subsequent node: if so, use C dA更新 ' to replace the feature value and record it as the new feature value; if not, keep the feature value unchanged and delete C dA更新 '; If the feature value of the pixel changes during this process and also changes during the execution of Steps 3 to 4, add / take the average of the two changed feature values, and record the result value as the final feature value. a3. Add the eigenvalue obtained in step a2 to the existing path cumulative value to obtain a new path cumulative value. The existing path cumulative value is the sum of the eigenvalues of all nodes between the starting position point and the current node in the current path at the disparity d A '. Statistically calculate the new cumulative path values corresponding to each path at the parallax value d A ', and record the path corresponding to the minimum cumulative path value as the extended path; a4. Label the current parallax value d A ' as the processed parallax. Arbitrarily select another unlabeled parallax value, denoted as parallax value d A '. Use the new parallax value d A ' to continue jumping to step a1 until all parallax values are marked as processed parallax, and then execute step C; Step C: Make the following judgments for each extended path respectively: Calculate the distance between the starting position point and the subsequent node in the extended path and record it as the path distance, and judge whether the path distance is greater than the preset threshold. If so, execute Step 5. If not, update the current node in the extended path to the previous node, update the subsequent node to the current node, and jump back to Step B to find the next node in this path until the calculated path distance is greater than the preset threshold.

4. The method according to claim 1, characterized in that: In step 1, the cost value of each pixel under different parallaxes is calculated based on the left image and the right image collected by the binocular system to form a parallax space; the method is as follows: 1) The left image or the right image collected by the binocular system is recorded as the image to be matched, and the other image is recorded as the target image; the pixel points in the image to be matched are recorded as the points to be matched, and the pixel points in the target image are recorded as the target points; According to the disparity search range corresponding to each to-be-matched point, a group of target points that meet the disparity search range are matched for each to-be-matched point, and the cost value of the to-be-matched point under different disparities is calculated according to each target point, and the pixel coordinates of the to-be-matched point, multiple disparity values, and multiple cost values are correspondingly stored; each to-be-matched point in the to-be-matched image is traversed, and the cost value under different disparity values is matched for each to-be-matched point; 2) Search for the maximum disparity value d within the disparity search range corresponding to each point to be matched max , if the maximum disparity value d of a certain point to be matched j-max is less than d max , then set the cost value corresponding to the disparity between d j-max +1 and d max to the numerical value A. The numerical value A is a preset value, which is greater than 5000; so that each point to be matched has the same number of disparity values and cost values; The disparity values and cost values of each point to be matched are summed up to obtain the disparity space.

5. The method according to claim 4, wherein: The disparity search range corresponding to each point to be matched is determined according to the following method: Extract multiple feature points on the object to be tested from the left image and the right image respectively, perform stereo matching and binocular 3D reconstruction, and obtain the 3D coordinates of the feature points; Find the three-dimensional coordinates of each feature point in the digital model workpiece coordinate system of the object to be measured; Based on the three-dimensional coordinates reconstructed by three-dimensional reconstruction and the three-dimensional coordinates in the digital model workpiece coordinate system, the conversion relationship RT between the three-dimensional space coordinate system of the binocular system and the digital model workpiece coordinate system is solved based on the rigid body transformation; In the digital model of the object to be measured, a digital model point is sampled, and the sampled digital model point is transformed to the image plane of the left camera using the transformation relationship RT, the projection matrix of the left camera and the distortion parameter of the left camera to obtain the transformation point coordinate p; Using the transformation relationship RT, the projection matrix of the right camera and the distortion parameter of the right camera, multiple digital-analog points are transformed to the image plane of the right camera to obtain the transformation point coordinates q; The absolute value of the difference between the horizontal coordinates of the conversion point coordinate p and the conversion point coordinate q is taken, and the absolute value of the difference is rounded and added to the preset tolerance δ to obtain the upper limit d' of the parallax search range; Then the disparity search range of the pixel points in the left image whose distance from the conversion point coordinate p is less than the threshold D is set to [0~d']; In the digital-analog of the object to be tested, a digital-analog point is sampled at each interval threshold D, and different digital-analog points are processed in the same way to obtain the conversion point coordinates p and upper limit values d' corresponding to different digital-analog points; According to different conversion point coordinates p and upper limit values d', the disparity search range of each to-be-matched point is marked in the left image.

6. The method according to claim 5, wherein: The threshold D is 30 to 200 pixels; δ is 1 to 30 pixels, or δ is (0.1 to 0.2)×H, where H represents the integer of the absolute value of the difference between the horizontal coordinates of the conversion point coordinates p and q.

7. The method according to claim 1, characterized in that: In step 1, the cost value of each pixel under different parallaxes is calculated based on the left image and the right image collected by the binocular system to form a parallax space; the method is as follows: The left image or the right image collected by the binocular system is recorded as the image to be matched, and the other image is recorded as the target image; the pixel points in the image to be matched are recorded as the points to be matched, and the pixel points in the target image are recorded as the target points; The same disparity search range is assigned to each point to be matched. Then, according to the disparity search range, a set of target points that meet the disparity search range are respectively matched for each point to be matched. The cost values of the point to be matched at different disparities are respectively calculated based on each target point, and the pixel coordinates, multiple disparity values, and multiple cost values of the point to be matched are stored correspondingly. Each point to be matched in the image to be matched is traversed, and the cost values at different disparity values are respectively matched for each point to be matched to obtain a disparity space.

8. The method according to claim 4 or 7, characterized in that: The method for respectively calculating the cost values of the point to be matched at different disparities based on each target point is the SAD method, the SSD method, or the STAD method.

9. The method according to any one of claims 1 to 7, characterized in that: The three-dimensional space coordinates (X, Y, Z) corresponding to each point to be matched (u, v) are calculated by using the saved disparity values: where, cx l and cx r are the pixel column coordinates of the principal points of the left and right images; f is the focal length, B is the baseline length between the left and right cameras, and D p is the disparity value saved at the pixel point (u, v).

10. The method according to claim 1, characterized in that: In step three, the N neighborhood is a four-neighborhood, an eight-neighborhood, or a 16-neighborhood; In step four, the preset threshold is obtained in the following way: The area where the object to be measured is located is selected in the image by using the minimum bounding rectangle, and the diagonal length of the minimum bounding rectangle is recorded as the preset threshold; The path distance is obtained in the following way: A pixel point on the path is selected in the middle area of the extended path, and the distances between this point and the starting position and the subsequent node are respectively calculated, and the two distance values are added, and the sum value is recorded as the path distance.

Images