A wave image matching method based on fusion matching cost

Abstract

The application discloses a wave image matching method based on fusion matching cost, and relates to the field of stereo matching, and comprises the following steps: S1, left and right eye images of waves are collected by using binocular cameras to determine a disparity range; S2, the left and right eye images of original waves collected in the step S1 are preprocessed; S3, a matching cost value of the wave images is obtained by using an improved Census algorithm; S4, improved Census cost, AD cost and gradient cost are fused to obtain a cost space; S5, a cross-domain is used for cost aggregation; S6, a winner-takes-all (WTA) strategy is used to calculate the disparity of the wave images; S7, the disparity map in the step S6 is subjected to left-right consistency checking, and is subjected to hole filling and sub-pixel optimization processing. The application adopts the above method, solves the problem that the disparity map obtained from the wave images is prone to a large number of invalid points and mismatching points, improves the correctness of wave stereo matching, and guarantees the matching precision of the depth discontinuous region.

Description

Technical Field

[0001] This invention relates to the field of stereo matching technology, and in particular to a wave image matching method based on fusion matching cost. Background Technology

[0002] Currently, various technologies have been developed for monitoring sea conditions, especially for wave measurement, including buoy methods, satellite remote sensing, and pressure sensor wave height measurement. However, traditional wave measurement methods lack universality for different measurement scenarios. For example, buoy wave measurement, as a contact measurement method, is affected by wave surge interference. Satellite remote sensing technology is only suitable for large-area real-sea wave measurement and suffers from drawbacks such as difficult installation and high cost of measurement equipment. Existing technologies also suffer from weak texture features in semi-transparent wave images, which are easily affected by lighting, leading to difficulties in stereo matching and a high mismatch rate. Therefore, there is an urgent need for a method suitable for both laboratory and real-sea wave measurement.

[0003] Therefore, it is necessary to provide a wave image matching method based on fusion matching cost to solve the above problems. Summary of the Invention

[0004] The purpose of this invention is to provide a wave image matching method based on fusion matching cost, which improves the accuracy of three-dimensional wave matching and ensures the matching accuracy in areas of depth discontinuity.

[0005] To achieve the above objectives, this invention provides a wave image matching method based on fusion matching cost, comprising the following steps:

[0006] S1: Use a binocular camera to acquire left and right eye images of the waves to determine the parallax range;

[0007] S2: Preprocess the left and right eye images of the original waves acquired in step S1;

[0008] S3: Use the improved Census algorithm to obtain the matching cost of the wave image;

[0009] S4: Integrate the improved Census cost, AD cost, and gradient cost to obtain the cost space;

[0010] S5: Cost aggregation is performed using a cross-domain approach;

[0011] S6: Calculate the parallax of the wave image using the winner-take-all WTA strategy;

[0012] S7: Perform left-right consistency checks on the disparity map in step S6, and perform hole filling and sub-pixel optimization processing.

[0013] Preferably, step S1 specifically includes the following steps:

[0014] S11: Determine the area of ​​the wave to be photographed based on the baseline distance of the binocular camera. The width of the wave area to be photographed shall not exceed 10 times the baseline distance of the binocular camera.

[0015] S12: Determine the parallax range based on the baseline distance of the binocular camera, the focal length of the lens, and the distance between the camera and the target. The relationship between the parameters is as follows:

[0016]

[0017] Where D is the distance between the wave and the camera, d is the parallax, B is the distance from the camera baseline, f is the camera focal length, and x is the distance from the camera baseline. ol and x or These are the x-coordinates of the principal point in the left and right views, respectively.

[0018] Preferably, in step S2, image preprocessing includes distortion correction and stereo correction, and the specific steps are as follows:

[0019] S21: Distortion includes radial distortion and tangential distortion. The distortion correction model is as follows:

[0020]

[0021] in, Let (x, y) be the normalized image coordinates after distortion, (x, y) be the normalized image coordinates without distortion, and r be the distance from a pixel in the image to the center point of the image. 2 =x 2 +y 2 k1, k2, and k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients.

[0022] S22: Stereo correction, based on the principle of epipolar geometry, corrects the left and right eye images to the same horizontal epipolar line.

[0023] Preferably, step S3 specifically includes the following steps:

[0024] S31: Select a rectangular window with a fixed size of 5×5 and iterate through the corrected image;

[0025] S32: Compare the grayscale values ​​of neighboring pixels within the window with those of the center pixel.

[0026] If the neighboring pixel q is not less than the center pixel p, mark it as 1;

[0027] If a neighboring pixel q is smaller than the center pixel p, then it is marked as 0;

[0028] After all comparisons are completed, a binary sequence of length 25 consisting of 0s and 1s is obtained, represented as:

[0029]

[0030] Among them, B s (p) represents the bit string generated by the window transformation of the center pixel p, N p Let I(p) and I(q) represent the neighborhood of pixel p, respectively, and let ζ be an operator that compares the neighboring pixels with the center pixel.

[0031] S33: Transform the center pixel window value B of the left and right eye images s (p) Perform a bitwise XOR operation to obtain a new bit string C of length 25. B (p1, p2), where p1 is the center pixel of the left eye image and p2 is the center pixel of the right eye image;

[0032] S34: Set threshold T a Equals 8, threshold T b The absolute values ​​of I(p) and I(q) are equal to 12. The values ​​of these absolute values ​​are then compared with the threshold T. a and threshold T b Compare the results and determine the bit string C based on the comparison. B The weight coefficients of each element in (p1, p2) are compared using the following formula:

[0033]

[0034] Where ω is the weighting coefficient, and |I(p)-I(q)| is the absolute value of I(p)-I(q);

[0035] S35: Transfer bit string C B The matching cost C of the improved Census is obtained by multiplying (p1, p2) by the weight coefficients in a position-weighted manner and summing the results. Census (p,d).

[0036] Preferably, step S4 specifically includes the following steps:

[0037] S41: Normalize the ranges of AD cost, improved Census cost, and gradient cost to [0, 1], as shown in the following formula:

[0038]

[0039] Where c is the cost value and λ is the weight ratio;

[0040] S42: The normalized AD cost, the improved Census cost, and the gradient cost are fused using the following formula:

[0041] C(p,d)=ρ(C grad (p,d),λgrad )

[0042] +ρ(C AD (p,d),λ AD )

[0043] +ρ(C Census (p,d),λ Census )

[0044] Where C(p,d) is the final cost after fusion; ρ(C grad (p,d),λ grad ) represents the gradient cost; ρ(C) AD (p,d),λ AD ) represents the cost of AD; ρ(C) Census (p,d),λ Census The cost of improved Census;

[0045] S43: Repeat steps S3 and S4 pixel by pixel and disparity to obtain the cost space.

[0046] Preferably, in step S5, the specific steps of cost aggregation are as follows:

[0047] S51: Set the color threshold and distance threshold to determine the horizontal and vertical arm lengths of a single pixel;

[0048] S52: For all pixels, sum the pixel values ​​on their horizontal arms and store the sum as a temporary value;

[0049] S53: For all pixels, sum the temporary values ​​stored in the previous step along their vertical arms to obtain the final aggregated cost C for a single pixel. s (p,d).

[0050] Preferably, in step S6, a winner-take-all (WTA) strategy is adopted to select the final aggregated cost C of a single pixel. s The initial disparity map of the left eye image is constructed using the disparity d corresponding to the minimum (p,d) value.

[0051] Preferably, step S7 specifically includes the following steps:

[0052] S71: Left-right consistency detection is used to process mismatched and occluded points in the disparity map, specifically including the following steps:

[0053] S711: Calculate the disparity map on the right according to the method in step S6;

[0054] S712: Compare the disparity values ​​of corresponding points in the left and right disparity maps:

[0055] If the difference is greater than the set threshold, the disparity of that point is considered abnormal and it is removed.

[0056] S72: Hole filling specifically includes calculating the mean value of the disparity values ​​near the hole area in the disparity map and using it as the new disparity value for the hole area;

[0057] S73: Perform sub-pixel optimization to obtain the final disparity map of wave stereo matching.

[0058] Therefore, the wave image matching method based on fusion matching cost described above, as used in this invention, has the following beneficial effects:

[0059] (1) Based on the properties of the matching target, the present invention designs different matching methods to improve the matching quality. The matching method that uses the fusion of Census cost, AD and gradient cost can effectively deal with the semi-transparency and weak texture properties of waves and increase the accuracy of three-dimensional matching of wave images.

[0060] (2) The present invention increases the cost value when the pixel value difference within the window is too large, reduces the false matching rate, and integrates multiple costs to improve the matching quality of wave images.

[0061] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0062] Figure 1 This is a flowchart of a wave image matching method based on fusion matching cost according to the present invention;

[0063] Figure 2 This is a flowchart of the improved Census algorithm of this invention;

[0064] Figure 3 This is an example diagram of the improved Census cost calculation of the present invention. Detailed Implementation

[0065] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0066] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0067] The terms "comprising" or "including" as used in this invention mean that the element preceding the term encompasses the element listed after the term, and do not exclude the possibility of encompassing other elements. Terms such as "inner," "outer," "upper," and "lower" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In this invention, unless otherwise explicitly specified and limited, the term "attached" and similar terms should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can refer to a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication of two elements or the interaction relationship between two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0068] like Figure 1 As shown, this invention provides a wave image matching method based on fusion matching cost, comprising the following steps:

[0069] S1: Acquire left and right eye images of the waves using a binocular camera to determine the parallax range; Step S1 specifically includes the following steps:

[0070] S11: Determine the area of ​​the wave to be photographed based on the baseline distance of the binocular camera. The width of the wave area to be photographed shall not exceed 10 times the baseline distance of the binocular camera.

[0071] S12: Determine the parallax range based on the baseline distance of the binocular camera, the focal length of the lens, and the distance between the camera and the target. The relationship between the parameters is as follows:

[0072]

[0073] Where D is the distance between the wave and the camera, d is the parallax, B is the distance to the camera baseline, f is the camera focal length, and x is the distance to the camera baseline. ol and x or These are the x-coordinates of the principal point in the left and right views, respectively.

[0074] S2: Preprocess the left and right eye images of the original waves acquired in step S1; in step S2, image preprocessing includes distortion correction and stereo correction, and the specific steps are as follows:

[0075] S21: Distortion includes radial distortion and tangential distortion. The distortion correction model is as follows:

[0076]

[0077] in, Let (x, y) be the normalized image coordinates after distortion, (x, y) be the normalized image coordinates without distortion, and r be the distance from a pixel in the image to the center point of the image. 2 =x 2 +y 2 k1, k2, and k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients.

[0078] S22: Stereo correction, based on the principle of epipolar geometry, corrects the same point names in the left and right eye images to the same horizontal epipolar line. The same point name is the term for the stereo matching direction.

[0079] S3: Obtain the matching cost of the wave image using the improved Census algorithm; Step S3 specifically includes the following steps:

[0080] S31: Select a rectangular window with a fixed size of 5×5 and iterate through the corrected image;

[0081] S32: Compare the grayscale values ​​of neighboring pixels within the window with those of the center pixel.

[0082] If the neighboring pixel q is not less than the center pixel p, mark it as 1;

[0083] If a neighboring pixel q is smaller than the center pixel p, then it is marked as 0;

[0084] After all comparisons are completed, a binary sequence of length 25 consisting of 0s and 1s is obtained, represented as:

[0085]

[0086] Among them, B s (p) represents the bit string generated by the window transformation of the center pixel p, N p Let I(p) and I(q) represent the neighborhood of pixel p, respectively, and let ξ be the operator that compares the neighboring pixels with the center pixel.

[0087] S33: Transform the center pixel window value B of the left and right eye images s (p) Perform a bitwise XOR operation to obtain a new bit string C of length 25. B (p1, p2), where p1 is the center pixel of the left eye image and p2 is the center pixel of the right eye image;

[0088] S34: Set threshold T a Equals 8, threshold T b The absolute values ​​of I(p) and I(q) are equal to 12. The values ​​of these absolute values ​​are then compared with the threshold T. a and threshold T b Compare the results and determine the bit string C based on the comparison.B The weight coefficients of each element in (p1, p2) are compared using the following formula:

[0089]

[0090] Where ω is the weighting coefficient, and |I(p)-I(q)| is the absolute value of I(p)-I(q);

[0091] S35: Transfer bit string C B The matching cost C of the improved Census is obtained by multiplying (p1, p2) by the weight coefficients in a position-weighted manner and summing the results. Census (p,d).

[0092] S4: Integrate the improved Census cost, AD cost, and gradient cost to obtain the cost space; Step S4 specifically includes the following steps: Step S4 specifically includes the following steps:

[0093] S41: Normalize the ranges of AD cost, improved Census cost, and gradient cost to [0, 1], as shown in the following formula:

[0094]

[0095] Where c is the cost value and λ is the weight ratio;

[0096] S42: The normalized AD cost, the improved Census cost, and the gradient cost are fused using the following formula:

[0097] C(p,d)=ρ(C grad (p,d),λ grad )

[0098] +ρ(C AD (p,d),λ AD )

[0099] +ρ(C Census (p,d),λ Census )

[0100] Where C(p,d) is the final cost after fusion; ρ(C grad (p,d),λ grad ) represents the gradient cost; ρ(C) AD (p,d),λ AD ) represents the cost of AD; ρ(C) Census (p,d),λ Census The cost of improved Census;

[0101] S43: Repeat steps S3 and S4 pixel by pixel and disparity to obtain the cost space.

[0102] S5: Cost aggregation is performed using a cross-domain approach; the specific steps of cost aggregation in step S5 are as follows:

[0103] S51: Set the color threshold and distance threshold to determine the horizontal and vertical arm lengths of a single pixel;

[0104] S52: For all pixels, sum the pixel values ​​on their horizontal arms and store the sum as a temporary value;

[0105] S53: For all pixels, sum the temporary values ​​stored in the previous step along their vertical arms to obtain the final aggregated cost C for a single pixel. s (p,d).

[0106] S6: Calculate the parallax of the wave image using the Winner-Take-All (WTA) strategy; in step S6, the Winner-Take-All (WTA) strategy is used to select the final aggregated cost C of a single pixel. s The initial disparity map of the left eye image is constructed using the disparity d corresponding to the minimum (p,d) value.

[0107] S7: Perform a left-right consistency check on the disparity map from step S6, and then perform hole filling and sub-pixel optimization processing. Step S7 specifically includes the following steps:

[0108] S71: Left-right consistency detection is used to process mismatched and occluded points in the disparity map, specifically including the following steps:

[0109] S711: Calculate the disparity map on the right according to the method in step S6;

[0110] S712: Compare the disparity values ​​of corresponding points in the left and right disparity maps:

[0111] If the difference is greater than the set threshold, the disparity of that point is considered abnormal and it is removed.

[0112] S72: Hole filling specifically includes calculating the mean value of the disparity values ​​near the hole area in the disparity map and using it as the new disparity value for the hole area;

[0113] S73: Perform sub-pixel optimization to obtain the final disparity map of wave stereo matching.

[0114] Example

[0115] The implementation process of this plan is divided into three parts.

[0116] Part 1: Determination of parallax and acquisition of distortion coefficients.

[0117] Determine the parameters of the stereo camera used to capture left and right wave images, including the baseline distance B, lens focal length f, area of ​​the wave region of interest, and its distance D from the camera. The relationships between these parameters are as follows:

[0118]

[0119] Where d is the disparity, x ol and x or These are the x-coordinates of the principal point in the left and right views, respectively. Based on the nearest and farthest distances of the region of interest from the camera, the maximum and minimum parallax that need to be input can be determined.

[0120] Using the stereoCameraCalibrator toolbox in MATLAB, input approximately 20 calibration images, the checkerboard grid size of the calibration board, and the number of corner points to obtain the calibration results. A calibration result is considered correct when the calibration error is less than 0.1 pixels. The calibration results contain all the data required for distortion correction and stereo correction.

[0121] Image preprocessing needs to be carried out based on the calibration results, using the correction functions in OpenCV to perform distortion correction and stereo correction on the left and right images.

[0122] Part 2: Improvements to Census matching cost.

[0123] Cost calculation is performed on the corrected image. First, an improved Census cost calculation is performed, and then it is fused with the AD cost and gradient cost.

[0124] like Figures 2-3 As shown, unlike the traditional Census cost calculation method which directly uses Hamming distance as the cost value, the improved Census method introduces a weight matrix of the window neighborhood, with a matrix size of 5×5, and multiplies the XOR result bitwise by the weight coefficient as the cost value.

[0125] If the absolute value of the difference between a neighboring pixel and the center pixel within the window is within the threshold T a and T b If the value is between 1 and 2, it indicates that the point is likely not a true disparity discontinuity point and the selected rectangular window is not suitable for the point. Therefore, the cost is increased by increasing the weight coefficient to |I(p)-I(q)| to reduce the probability of mismatch.

[0126] Within the window, except for the absolute value of the difference between neighboring pixels and the center pixel which is within the threshold T. a and T b Outside of the points between them, the weight coefficients of all other points are set to 1 to ensure that the true disparity discontinuities can still be matched normally.

[0127] like Figure 3As shown, although the window contains outliers, the XOR result of the outliers is 0. In this case, the improved Census method and the traditional method calculate the same cost.

[0128] Part Three: The Fusion of Three Costs

[0129] For different wave images, the fusion ratio of improved Census cost, AD cost and gradient cost is dynamically adjusted according to the strength of wave texture and the size of the parallax discontinuity region, so that the present invention can be applied to the matching of wave images in different scenarios.

[0130] Therefore, the present invention adopts the above-mentioned wave image matching method based on fusion matching cost, which solves the problem that wave image disparity maps are prone to having a large number of invalid points and mismatched points, improves the accuracy of wave stereo matching, and ensures the matching accuracy of depth discontinuity regions.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A wave image matching method based on fusion matching cost, characterized in that: Includes the following steps: S1: Use a binocular camera to acquire left and right eye images of the waves to determine the parallax range; S2: Preprocess the left and right eye images of the original waves acquired in step S1; S3: Obtain the matching cost of the wave image using the improved Census algorithm; Step S3 specifically includes the following steps: S31: Select a rectangular window with a fixed size of 5×5 and iterate through the corrected image; S32: Compare the grayscale values ​​of neighboring pixels within the window with those of the center pixel. If neighboring pixels q Not smaller than the center pixel p Marked as 1; If neighboring pixels q Smaller than the center pixel p If , then it is marked as 0; After all comparisons are completed, a binary sequence of length 25 consisting of 0s and 1s is obtained, represented as: in, Represents the center pixel p The bit string generated by window transformation, N p Represents pixels p The neighborhood, I(p) and I (q) They are respectively p and q pixel values, This is defined as an operator that compares neighboring pixels with the center pixel; S33: Transform the center pixel window values ​​of the left and right eye images Perform a bitwise XOR operation to obtain a new bit string of length 25. , The center pixel of the left eye image. The center pixel of the right eye image; S34: Set threshold T a Equals 8, threshold T b Equal to 12, will and The absolute values ​​of the thresholds are respectively T a and threshold T b Compare the results and determine the bit string. The weighting coefficients for each element are compared using the following formula: Where ω is the weighting coefficient, for The absolute value; S35: Convert the bit string The matching cost of the improved Census is obtained by multiplying the weights bitwise and summing the results. C Census (p,d) ; S4: Integrate the improved Census cost, AD cost, and gradient cost to obtain the cost space; S5: Cost aggregation is performed using a cross-domain approach; S6: Calculate the parallax of the wave image using the winner-take-all WTA strategy; S7: Perform left-right consistency checks on the disparity map in step S6, and perform hole filling and sub-pixel optimization processing.

2. The wave image matching method based on fusion matching cost according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11: Determine the area of ​​the wave to be photographed based on the baseline distance of the binocular camera. The width of the wave area to be photographed shall not exceed 10 times the baseline distance of the binocular camera. S12: Determine the parallax range based on the baseline distance of the binocular camera, the focal length of the lens, and the distance between the camera and the target. The relationship between the parameters is as follows: in, D The distance between the wave and the camera. d For parallax, B The distance to the camera baseline. f For camera focal length, The principal point is shown in the left and right views respectively. x coordinate.

3. The wave image matching method based on fusion matching cost according to claim 1, characterized in that: In step S2, image preprocessing includes distortion correction and stereo correction, and the specific steps are as follows: S21: Distortion includes radial distortion and tangential distortion. The distortion correction model is as follows: in, These are the normalized image coordinates after distortion. These are the distortion-free, normalized image coordinates. r The distance from a pixel in the image to the center point of the image. , , , All are radial distortion coefficients. All are tangential distortion coefficients; S22: Stereo correction, based on the principle of epipolar geometry, corrects the left and right eye images to the same horizontal epipolar line.

4. The wave image matching method based on fusion matching cost according to claim 3, characterized in that: Step S4 specifically includes the following steps: S41: Reduce AD ​​costs and improve Census Both the cost and gradient cost are normalized to the range [0, 1], as shown in the following formula: in, c The value is denoted by λ, and the weighting ratio is λ. S42: Cost of normalized AD, improvement Census The cost and gradient cost are fused together using the following formula: in This is the ultimate cost of the fusion; For gradient cost; For the cost of AD; For improvement Census cost; S43: Repeat steps S3 and S4 pixel by pixel and disparity to obtain the cost space.

5. The wave image matching method based on fusion matching cost according to claim 1, characterized in that: In step S5, the specific steps of cost aggregation are as follows: S51: Set the color threshold and distance threshold to determine the horizontal and vertical arm lengths of a single pixel; S52: For all pixels, sum the pixel values ​​on their horizontal arms and store the sum as a temporary value; S53: For all pixels, sum the temporary values ​​stored in the previous step along their vertical arms to obtain the final aggregated value of a single pixel. .

6. The wave image matching method based on fusion matching cost according to claim 5, characterized in that: In step S6, the winner-take-all (WTA) strategy is adopted to select the final aggregated cost of a single pixel. minimum disparity d As the initial disparity, an initial disparity map is constructed for the left eye image.

7. The wave image matching method based on fusion matching cost according to claim 5, characterized in that: Step S7 specifically includes the following steps: S71: Left-right consistency detection is used to process mismatched and occluded points in the disparity map, specifically including the following steps: S711: Calculate the disparity map on the right according to the method in step S6; S712: Compare the disparity values ​​of corresponding points in the left and right disparity maps: If the difference is greater than the set threshold, the disparity of that point is considered abnormal and it is removed. S72: Hole filling specifically includes calculating the mean value of the disparity values ​​near the hole area in the disparity map and using it as the new disparity value for the hole area; S73: Perform sub-pixel optimization to obtain the final disparity map of wave stereo matching.

Images