Wave image matching method based on fusion matching cost

Abstract

The invention discloses a wave image matching method based on fusion matching cost, and relates to the field of stereo matching, and the method comprises the following steps: S1, collecting left and right eye images of waves through a binocular camera, and determining a parallax range; s2, preprocessing the left and right eye images of the original waves collected in the step S1; s3, obtaining a matching cost value of the wave image by using an improved Census algorithm; s4, fusing the improved Census cost, AD cost and gradient cost to obtain a cost space; s5, carrying out cost aggregation by adopting a cross domain; s6, calculating the parallax of the wave image by using a WTA strategy; and S7, performing left and right consistency check on the disparity map in the step S6, and performing hole filling and sub-pixel optimization processing. By the adoption of the method, the problem that a disparity map obtained from a wave image is prone to having a large number of invalid points and mismatching points is solved, the accuracy of wave stereo matching is improved, and the matching precision of a depth discontinuous area is guaranteed.

Description

Technical Field

[0001] The invention relates to the technical field of stereo matching, and in particular to a wave image matching method based on fusion matching cost. Background Art

[0002] At present, a variety of technologies have been developed for monitoring sea conditions, especially for measuring waves, including buoy method, satellite remote sensing technology, pressure sensor wave height measurement method, etc. However, traditional wave measurement methods are not universal for different measurement scenarios. For example, the buoy wave measurement method, as a contact measurement method, will be disturbed by surges, and satellite remote sensing technology is only suitable for large-scale real sea wave measurement, and has the disadvantages of difficult installation and high cost of measurement equipment. In the existing technology, there is a problem that the texture features of semi-transparent wave images are weak and easily affected by light, resulting in difficulties in stereo matching and high mismatching rate. Therefore, there is an urgent need for a method that is suitable for both laboratory wave measurement and real sea wave measurement.

[0003] Therefore, it is necessary to provide a wave image matching method based on fusion matching cost to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide a wave image matching method based on fusion matching cost, which improves the accuracy of wave stereo matching and ensures the matching accuracy of depth discontinuous areas.

[0005] To achieve the above object, the present invention provides a wave image matching method based on fusion matching cost, comprising the following steps:

[0006] S1: Use a binocular camera to collect left and right images of the waves and determine the parallax range;

[0007] S2: preprocessing the left and right eye images of the original wave collected in step S1;

[0008] S3: Use the improved Census algorithm to obtain the matching cost value of the wave image;

[0009] S4: Fusion of improved Census cost, AD cost and gradient cost to obtain the cost space;

[0010] S5: Use cross-domain for cost aggregation;

[0011] S6: Calculate the parallax of the wave image using the winner-takes-all WTA strategy;

[0012] S7: Check the left and right consistency of the disparity map in step S6, and perform hole filling and sub-pixel optimization processing.

[0013] Preferably, in step S1, the following steps are specifically included:

[0014] S11: determining the photographed wave area according to the baseline distance of the binocular camera, and the photographed width of the wave area is not greater than 10 times the baseline distance of the binocular camera;

[0015] S12: Determine the parallax range according to the baseline distance of the binocular camera, the focal length of the lens and the distance between the camera and the target. The relationship between the parameters is:

[0016]

[0017] Where D is the distance between the wave and the camera, d is the parallax, B is the camera baseline distance, f is the camera focal length, and x ol and x or are the x coordinates of the principal point in the left and right views respectively.

[0018] Preferably, in step S2, image preprocessing includes distortion correction and stereo correction, and the specific steps are as follows:

[0019] S21: Distortion includes radial distortion and tangential distortion. The distortion correction model is expressed as follows:

[0020]

[0021] in, is the normalized image coordinate after distortion, (x, y) is the normalized image coordinate without distortion, r is the distance from the image pixel to the center of the image, r 2 =x 2 +y 2 , k 1 , k 2 , k 3 are radial distortion coefficients, p 1 、p 2 are tangential distortion coefficients;

[0022] S22: Stereo correction uses the principle of epipolar geometry to correct the left and right eye images to the same horizontal epipolar line.

[0023] Preferably, step S3 specifically includes the following steps:

[0024] S31: Select a rectangular window with a fixed size of 5×5 and traverse the corrected image;

[0025] S32: Compare the grayscale values ​​of the neighborhood pixels within the window with those of the central pixel:

[0026] If the neighborhood pixel q is not less than the central pixel p, it is marked as 1;

[0027] If the neighborhood pixel q is smaller than the central pixel p, it is marked as 0;

[0028] After all comparisons are completed, a binary sequence of length 25 consisting of 0s and 1s is obtained, which is expressed as:

[0029]

[0030] Among them, B s (p) represents the bit string generated by window transformation of the center pixel p, N p represents the neighborhood of pixel p, I(p) and I(q) are the pixel values ​​of p and q respectively, and ζ is an operator defined to compare the neighborhood pixels with the central pixel;

[0031] S33: transform the central pixel window of the left and right images into value B s (p) Perform bitwise XOR operation to obtain a new bit string C with a length of 25. B (p 1 ,p 2 ), p 1 is the center pixel of the left image, p 2 is the center pixel of the right eye image;

[0032] S34: Setting threshold T a Equal to 8, threshold T b =12, and compare the absolute values ​​of I(p) and I(q) with the threshold T a and threshold T b Compare and determine the bit string C according to the comparison result B (p 1 ,p 2 ) The weight coefficient of each digit in the comparison formula is as follows:

[0033]

[0034] Where ω is the weight coefficient, |I(p)-I(q)| is the absolute value of I(p)-I(q);

[0035] S35: Convert the bit string C B (p 1 ,p 2 ) is multiplied and summed with the weight coefficient in bitwise weighted manner to obtain the matching cost C of the improved Census Census (p,d).

[0036] Preferably, step S4 specifically includes the following steps:

[0037] S41: Normalize the range of AD cost, improved Census cost and gradient cost to [0, 1]. The specific formula is as follows:

[0038]

[0039] Among them, c is the cost value, λ is the weight ratio;

[0040] S42: The normalized AD cost, improved Census cost and gradient cost are fused. The fusion formula is:

[0041] C(p,d)=ρ(C grad (p,d),λ grad )

[0042] +ρ(C AD (p,d),λ AD )

[0043] +ρ(C Census (p,d),λ Census )

[0044] Where C(p,d) is the final cost after fusion; ρ(C grad (p,d),λ grad ) is the gradient cost; ρ(C AD (p,d),λ AD ) is the AD cost; ρ(C Census (p,d),λ Census ) is the improved Census cost;

[0045] S43: Repeat steps S3 and S4 pixel by pixel and disparity by disparity to obtain a cost space.

[0046] Preferably, in step S5, the specific steps of cost aggregation are as follows:

[0047] S51: setting a color threshold and a distance threshold to determine a horizontal arm length and a vertical arm length of a single pixel;

[0048] S52: for all pixels, add the pixel cost values ​​on their horizontal arms and store them as temporary values;

[0049] S53: For all pixels, add the temporary values ​​stored in the previous step along their vertical arms to obtain the final aggregate cost value C of a single pixel s (p,d).

[0050] Preferably, in step S6, a winner-takes-all WTA strategy is adopted to select the final aggregate cost C of a single pixel. s The disparity d corresponding to the minimum of (p, d) is taken as the initial disparity, and the initial disparity map of the left eye image is constructed.

[0051] Preferably, step S7 specifically includes the following steps:

[0052] S71: Using left-right consistency detection to process the incorrect matching points and occluded points in the disparity map, specifically including the following steps:

[0053] S711: Calculate the right image disparity map according to the method of step S6;

[0054] S712: Compare the disparity values ​​of corresponding points of the left and right disparity images:

[0055] If the difference is greater than the set threshold, the parallax of the point is considered abnormal and is removed;

[0056] S72: Hole filling specifically includes calculating the average of the disparity values ​​near the hole area in the disparity map, and using the average of the disparity values ​​as the new disparity value of the hole area;

[0057] S73: Perform sub-pixel optimization to obtain a final disparity map of wave stereo matching.

[0058] Therefore, the present invention adopts the above-mentioned wave image matching method based on fusion matching cost, which has the following beneficial effects:

[0059] (1) According to the properties of the matching target, the present invention designs different matching methods to improve the matching quality. The matching method that combines Census cost with AD and gradient cost can effectively deal with the translucency and weak texture properties of waves and increase the accuracy of stereo matching of wave images.

[0060] (2) The present invention increases the cost value when the pixel value difference in the window is too large, reduces the mismatch rate, and integrates multiple costs to improve the matching quality of the wave image.

[0061] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flow chart of a wave image matching method based on fusion matching cost of the present invention;

[0063] Figure 2 is a flow chart of the improved Census algorithm of the present invention;

[0064] Figure 3 This is an example diagram of improving Census cost calculation in the present invention. DETAILED DESCRIPTION

[0065] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0066] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0067] The words "include" or "comprises" and the like used in the present invention mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of also including other elements. The orientation or position relationship indicated by the terms "inside", "outside", "upper", "lower", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation of the present invention. When the absolute position of the described object changes, the relative position relationship may also change accordingly. In the present invention, unless otherwise clearly specified and limited, the terms "attachment" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral body; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0068] like Figure 1 As shown, the present invention provides a wave image matching method based on fusion matching cost, comprising the following steps:

[0069] S1: Use a binocular camera to collect left and right eye images of waves and determine the parallax range; in step S1, the following steps are specifically included:

[0070] S11: determining the photographed wave area according to the baseline distance of the binocular camera, and the photographed width of the wave area is not greater than 10 times the baseline distance of the binocular camera;

[0071] S12: Determine the parallax range according to the baseline distance of the binocular camera, the focal length of the lens and the distance between the camera and the target. The relationship between the parameters is:

[0072]

[0073] Where D is the distance between the wave and the camera, d is the parallax, B is the camera baseline distance, f is the camera focal length, and x ol and x or are the x coordinates of the principal point in the left and right views respectively.

[0074] S2: Preprocess the left and right eye images of the original wave collected in step S1; in step S2, the image preprocessing includes distortion correction and stereo correction, and the specific steps are as follows:

[0075] S21: Distortion includes radial distortion and tangential distortion. The distortion correction model is expressed as follows:

[0076]

[0077] in, is the normalized image coordinate after distortion, (x, y) is the normalized image coordinate without distortion, r is the distance from the image pixel to the center of the image, r 2 =x 2 +y 2 , k 1 , k 2 , k 3 are radial distortion coefficients, p 1 、p 2 are tangential distortion coefficients;

[0078] S22: Stereo correction uses the principle of epipolar geometry to correct the same point names of the left and right eye images to the same horizontal epipolar line. The same point name is the noun of the stereo matching direction.

[0079] S3: using the improved Census algorithm to obtain the matching cost value of the wave image; step S3 specifically includes the following steps:

[0080] S31: Select a rectangular window with a fixed size of 5×5 and traverse the corrected image;

[0081] S32: Compare the grayscale values ​​of the neighborhood pixels within the window with those of the central pixel:

[0082] If the neighborhood pixel q is not less than the central pixel p, it is marked as 1;

[0083] If the neighborhood pixel q is smaller than the central pixel p, it is marked as 0;

[0084] After all comparisons are completed, a binary sequence of length 25 consisting of 0s and 1s is obtained, which is expressed as:

[0085]

[0086] Among them, B s (p) represents the bit string generated by window transformation of the center pixel p, N p represents the neighborhood of pixel p, I(p) and I(q) are the pixel values ​​of p and q respectively, and ξ is an operator defined to compare the neighborhood pixels with the central pixel;

[0087] S33: transform the central pixel window of the left and right images into value B s (p) Perform bitwise XOR operation to obtain a new bit string C with a length of 25. B (p 1 ,p 2 ), p 1 is the center pixel of the left image, p 2 is the center pixel of the right eye image;

[0088] S34: Setting threshold T a Equal to 8, threshold T b =12, and compare the absolute values ​​of I(p) and I(q) with the threshold T a and threshold T b Compare and determine the bit string C according to the comparison result B (p 1 ,p 2 ) The weight coefficient of each digit in the comparison formula is as follows:

[0089]

[0090] Among them, ω is the weight coefficient, |I(p)-I(q)| is the absolute value of I(p)-I(q);

[0091] S35: Convert the bit string C B (p 1 ,p 2 ) is multiplied and summed with the weight coefficient in bitwise weighted manner to obtain the matching cost C of the improved Census Census (p,d).

[0092] S4: Fusing the improved Census cost, AD cost and gradient cost to obtain the cost space; Step S4 specifically includes the following steps: Step S4 specifically includes the following steps:

[0093] S41: Normalize the range of AD cost, improved Census cost and gradient cost to [0, 1]. The specific formula is as follows:

[0094]

[0095] Among them, c is the cost value, λ is the weight ratio;

[0096] S42: The normalized AD cost, improved Census cost and gradient cost are fused. The fusion formula is:

[0097] C(p,d)=ρ(C grad (p,d),λ grad )

[0098] +ρ(C AD (p,d),λ AD )

[0099] +ρ(C Census (p,d),λ Census )

[0100] Where C(p,d) is the final cost after fusion; ρ(C grad (p,d),λ grad) is the gradient cost; ρ(C AD (p,d),λ AD ) is the AD cost; ρ(C Census (p,d),λ Census ) is the improved Census cost;

[0101] S43: Repeat steps S3 and S4 pixel by pixel and disparity by disparity to obtain a cost space.

[0102] S5: Cost aggregation is performed using a cross domain. In step S5, the specific steps of cost aggregation are as follows:

[0103] S51: setting a color threshold and a distance threshold to determine a horizontal arm length and a vertical arm length of a single pixel;

[0104] S52: for all pixels, add the pixel cost values ​​on their horizontal arms and store them as temporary values;

[0105] S53: For all pixels, add the temporary values ​​stored in the previous step along their vertical arms to obtain the final aggregate cost value C of a single pixel s (p,d).

[0106] S6: Calculate the parallax of the wave image using the winner-takes-all WTA strategy; In step S6, the winner-takes-all WTA strategy is used to select the final aggregate cost value C of a single pixel s The disparity d corresponding to the minimum of (p, d) is taken as the initial disparity, and the initial disparity map of the left eye image is constructed.

[0107] S7: Check the left and right consistency of the disparity map in step S6, and perform hole filling and sub-pixel optimization. Step S7 specifically includes the following steps:

[0108] S71: Using left-right consistency detection to process the incorrect matching points and occluded points in the disparity map, specifically including the following steps:

[0109] S711: Calculate the right image disparity map according to the method of step S6;

[0110] S712: Compare the disparity values ​​of corresponding points of the left and right disparity images:

[0111] If the difference is greater than the set threshold, the parallax of the point is considered abnormal and is removed;

[0112] S72: Hole filling specifically includes calculating the average of the disparity values ​​near the hole area in the disparity map, and using the average of the disparity values ​​as the new disparity value of the hole area;

[0113] S73: Perform sub-pixel optimization to obtain a final disparity map of wave stereo matching.

[0114] Example

[0115] The implementation process of this program is divided into three parts

[0116] Part I: Determination of parallax and acquisition of distortion coefficients.

[0117] Determine the various parameters of the binocular camera that captures the left and right wave images, including the baseline distance B, the lens focal length f, the area of ​​the wave region of interest, and the distance D between it and the camera. The relationship between the various parameters is:

[0118]

[0119] Where d is the parallax, x ol and x or are the x coordinates of the principal point in the left and right views respectively. According to the shortest and farthest distances of the area of ​​interest from the camera, the maximum and minimum parallax that need to be input can be determined.

[0120] Using the stereoCameraCalibrator toolbox of MATLAB, input 20 calibration images, the grid size of the calibration board, and the number of corner points to obtain the calibration result. When the calibration error is less than 0.1 pixel, the calibration result is considered correct. The calibration result contains all the data required for distortion correction and stereo correction.

[0121] Image preprocessing needs to be carried out based on the calibration results, and the correction function in Opencv is used to perform distortion correction and stereo correction on the left and right images.

[0122] Part 2: Improvement of Census matching cost.

[0123] The cost of the corrected image is calculated by first calculating the improved Census cost and then fusing it with the AD cost and gradient cost.

[0124] like Figure 2-Figure 3 As shown in the figure, unlike the traditional Census cost calculation method that directly uses the Hamming distance as the cost value, the improved Census method introduces a weight matrix of the window neighborhood with a matrix size of 5×5, and multiplies the XOR result by the weight coefficient bit by bit as the cost value.

[0125] If the absolute value of the difference between a certain neighborhood pixel and the central pixel in the window is between the threshold T a and T b If the value is between |I(p)-I(q)| and |I(p)-I(q)|, it means that the point is most likely not a true disparity discontinuity point, and the selected rectangular window is not suitable for the point. In this way, the cost is increased by increasing its weight coefficient to |I(p)-I(q)| to reduce the probability of mismatching.

[0126] In the window, except for the absolute value of the difference between the neighborhood pixel and the central pixel between the threshold T a and T b Except for the points between , the weight coefficients of the remaining points are set to 1 to ensure that the real discontinuous points of parallax can still be matched normally.

[0127] like Figure 3 As shown in FIG. 1 , although the window contains an outlier, the XOR result of the outlier is 0. At this time, the cost calculated by the improved Census method is equal to the cost calculated by the traditional method.

[0128] Part III: Integration of three costs.

[0129] For different wave images, the fusion ratio of the improved Census cost, AD cost and gradient cost is dynamically adjusted according to the strength of the wave texture and the size of the parallax discontinuity area, so that the present invention can be applied to the matching of wave images in different scenes.

[0130] Therefore, the present invention adopts the above-mentioned wave image matching method based on fusion matching cost, which solves the problem that the wave image disparity map is prone to have a large number of invalid points and mismatched points, improves the accuracy of wave stereo matching, and ensures the matching accuracy of depth discontinuous areas.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A wave image matching method based on fusion matching cost, characterized by: The following steps are involved: S1: Use a binocular camera to collect left and right images of the waves and determine the parallax range; S2: preprocessing the left and right eye images of the original wave collected in step S1; S3: Use the improved Census algorithm to obtain the matching cost value of the wave image; S4: Fusion of improved Census cost, AD cost and gradient cost to obtain the cost space; S5: Use cross-domain for cost aggregation; S6: Calculate the parallax of the wave image using the winner-takes-all WTA strategy; S7: Check the left and right consistency of the disparity map in step S6, and perform hole filling and sub-pixel optimization processing.

2. The wave image matching method based on fusion matching cost according to claim 1, characterized in that: In step S1, the following steps are specifically included: S11: determining the photographed wave area according to the baseline distance of the binocular camera, and the photographed width of the wave area is not greater than 10 times the baseline distance of the binocular camera; S12: Determine the parallax range according to the baseline distance of the binocular camera, the focal length of the lens and the distance between the camera and the target. The relationship between the parameters is: Where D is the distance between the wave and the camera, d is the parallax, B is the camera baseline distance, f is the camera focal length, and x ol and x or are the x coordinates of the principal point in the left and right views respectively.

3. The wave image matching method based on fusion matching cost according to claim 1 is characterized in that: In step S2, image preprocessing includes distortion correction and stereo correction. The specific steps are as follows: S21: Distortion includes radial distortion and tangential distortion. The distortion correction model is expressed as follows: in, is the normalized image coordinate after distortion, (x, y) is the normalized image coordinate without distortion, r is the distance from the image pixel to the center of the image, r 2 =x 2 +y 2 , k1, k2, k3 are radial distortion coefficients, p1, p2 are tangential distortion coefficients; S22: Stereo correction uses the principle of epipolar geometry to correct the left and right eye images to the same horizontal epipolar line.

4. The wave image matching method based on fusion matching cost according to claim 3 is characterized in that: Step S3 specifically includes the following steps: S31: Select a rectangular window with a fixed size of 5×5 and traverse the corrected image; S32: Compare the grayscale values ​​of the neighborhood pixels within the window with those of the central pixel: If the neighborhood pixel q is not less than the central pixel p, it is marked as 1; If the neighborhood pixel q is smaller than the central pixel p, it is marked as 0; After all comparisons are completed, a binary sequence of length 25 consisting of 0s and 1s is obtained, which is expressed as: Among them, B s (p) represents the bit string generated by window transformation of the center pixel p, N p represents the neighborhood of pixel p, I(p) and I(q) are the pixel values ​​of p and q respectively, and ζ is an operator defined to compare the neighborhood pixels with the central pixel; S33: transform the central pixel window of the left and right images into value B s (p) Perform bitwise XOR operation to obtain a new bit string C with a length of 25. B (p1, p2), p1 is the center pixel of the left image, p2 is the center pixel of the right image; S34: Setting threshold T a Equal to 8, threshold T b =12, and compare the absolute values ​​of I(p) and I(q) with the threshold T a and threshold T b Compare and determine the bit string C according to the comparison result B The weight coefficient of each bit in (p1, p2) is compared as follows: Where ω is the weight coefficient, |I(p)-I(q)| is the absolute value of I(p)-I(q); S35: Convert the bit string C B (p1, p2) is multiplied and summed with the weight coefficient in bitwise weighted manner to obtain the matching cost C of the improved Census Census (p,d).

5. The wave image matching method based on fusion matching cost according to claim 4 is characterized in that: Step S4 specifically includes the following steps: S41: Normalize the range of AD cost, improved Census cost and gradient cost to [0, 1]. The specific formula is as follows: Among them, c is the cost value, λ is the weight ratio; S42: The normalized AD cost, improved Census cost and gradient cost are fused. The fusion formula is: C(p,d)=ρ(C grad (p,d),λ grad ) +ρ(C AD (p,d),λ AD ) +ρ(C Census (p,d),λ Census ) Where C(p,d) is the final cost after fusion; ρ(C grad (p,d),λ grad ) is the gradient cost; ρ(C AD (p,d),λ AD ) is the AD cost; ρ(C Census (p,d),λ Census ) is the improved Census cost; S43: Repeat steps S3 and S4 pixel by pixel and disparity by disparity to obtain a cost space.

6. The wave image matching method based on fusion matching cost according to claim 1, characterized in that: In step S5, the specific steps of cost aggregation are as follows: S51: setting a color threshold and a distance threshold to determine a horizontal arm length and a vertical arm length of a single pixel; S52: for all pixels, add the pixel cost values ​​on their horizontal arms and store them as temporary values; S53: For all pixels, add the temporary values ​​stored in the previous step along their vertical arms to obtain the final aggregate cost value C of a single pixel s (p,d).

7. The wave image matching method based on fusion matching cost according to claim 6 is characterized in that: In step S6, the winner-takes-all WTA strategy is adopted to select the final aggregation cost C of a single pixel s The disparity d corresponding to the minimum of (p, d) is taken as the initial disparity, and the initial disparity map of the left eye image is constructed.

8. The wave image matching method based on fusion matching cost according to claim 6 is characterized by: Step S7 specifically includes the following steps: S71: Using left-right consistency detection to process the incorrect matching points and occluded points in the disparity map, specifically including the following steps: S711: Calculate the right image disparity map according to the method of step S6; S712: Compare the disparity values ​​of corresponding points of the left and right disparity images: If the difference is greater than the set threshold, the parallax of the point is considered abnormal and is removed; S72: Hole filling specifically includes calculating the average of the disparity values ​​near the hole area in the disparity map, and using the average of the disparity values ​​as the new disparity value of the hole area; S73: Perform sub-pixel optimization to obtain a final disparity map of wave stereo matching.

Images