A crack image matching method based on binocular stereo vision and U-Net neural network

Abstract

The present invention relates to the field of surface crack detection in concrete structures, and in particular to a crack image matching method based on binocular stereo vision and a U-NET neural network, comprising the following steps: S1: using a binocular camera to obtain a left view and a right view of cracks on the surface of a concrete structure; S2: performing distortion correction and stereo correction on the left and right views; S3: using a trained U-Net semantic segmentation network to extract cracks in the corrected left and right views; S4: retrieving crack boundary points on the same horizontal line; S5: matching crack boundary points in the corrected left and right views that have the same retrieval order; S6: performing a correlation test on the neighborhoods where the matching points are located in the corrected left and right views, and relocating matching points that fail the test. The present invention extracts cracks in the corrected left and right views respectively by integrating binocular stereo vision with a deep learning algorithm, and matches crack boundary points on the same horizontal line.

Description

Technical Field

[0001] The present invention relates to the field of concrete structure surface crack detection, and specifically to a crack image matching method based on binocular stereo vision and a U-Net neural network. Background Art

[0002] Cracks are a common problem in concrete structures. Prompt detection and repair of cracks can eliminate potential threats and ensure structural performance. Binocular stereo vision-based detection technology offers the advantage of being non-contact, avoiding the inconvenience and interference associated with contact sensors. It also provides target depth information, offering high accuracy, low cost, and speed, making it suitable for precise crack detection.

[0003] Currently, crack detection based on binocular stereo vision primarily uses matching algorithms based on blocks, templates, and grayscale information to match the left and right images as a whole. While these matching results are highly accurate, the calculation process is time-consuming. Furthermore, when the external light source is unevenly distributed, the brightness and exposure levels of the images captured by the two cameras can vary significantly, affecting the matching results. Summary of the Invention

[0004] In order to solve the problem that the calculation process of crack detection in the prior art consumes a long time, the present invention provides a crack image matching method based on binocular stereo vision and U-Net neural network.

[0005] The present invention is achieved through the following technical solutions:

[0006] A crack image matching method based on binocular stereo vision and U-Net neural network, the operation steps are as follows:

[0007] (1) Use a binocular camera to obtain the left and right views of the cracks on the concrete structure surface;

[0008] (2) performing distortion correction and stereo correction on the left view and the right view respectively to obtain a corrected left view and a corrected right view;

[0009] (3) Using the U-Net semantic segmentation network to extract cracks in the rectified left and right images respectively;

[0010] (4) Extract the crack boundary and retrieve the crack boundary points on the same horizontal line from left to right to obtain the crack boundary points {L1, L2···L n} and the crack boundary points {R1, R2···R n};

[0011] (5) On the same horizontal line, all the crack boundary points in the corrected left view and all the crack boundary points in the corrected right view are matched to obtain the matching points {L1 R1, L2 R2, ···L n R n};

[0012] (6) Perform a correlation test on the neighborhood where the matching points are located, and re-match the matching points that fail the test.

[0013] Preferably, in step (1), before obtaining the left view and the right view of the surface crack of the concrete structure, the binocular camera is calibrated using a black and white checkerboard, the calibration plate is placed in front of the binocular camera and 12 pairs of images are obtained by changing different angles, and the intrinsic parameters and extrinsic parameters of the binocular camera are calculated using Zhang's calibration method.

[0014] Preferably, in step (2), the left view and the right view are subjected to distortion correction according to the camera internal parameters.

[0015] Preferably, in step (2), the left view and the right view are stereo-corrected according to the camera external parameters so that the same points in the left view and the right view are located on the same horizontal line.

[0016] Preferably, in step (3), the U-Net semantic segmentation network is first trained using an open source concrete structure surface crack image, and then, based on the trained U-Net semantic segmentation network, the corrected left view and the corrected right view are imported to perform crack segmentation respectively.

[0017] Preferably, in step (4), the Sobel edge detection operator is used to extract the boundaries of the crack areas in the corrected left view and the corrected right view; and on the same horizontal line, along the left to right direction, the crack boundary points {L1, L2···L n} and the crack boundary points {R1, R2···R n}.

[0018] Preferably, in step (5), the crack boundary points on the same horizontal line are paired in the retrieval order, that is, the i-th crack boundary point in the corrected left view corresponds to the i-th crack boundary point in the corrected right view, and the matching points {L1R1, L2R2, ···L n R n}, in order to match the crack boundary points on all horizontal lines.

[0019] Preferably, in step (6), during the inspection, the matching points {L i , R i}Matching point{L i}, matching point {Ri}, the normalized cross-correlation calculation is performed on the two regions based on the grayscale value. If the correlation between the two regions is greater than or equal to 0.5, the matching point is qualified; if the correlation between the two regions is less than 0.5, the matching point is unqualified.

[0020] Preferably, when the matching point test fails, the original matching point {R i} and re-determine the matching points within the 11×11 neighborhood of .

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention's crack image matching method, based on binocular stereo vision and a U-Net neural network, avoids traditional matching algorithms based on blocks, templates, and grayscale information, and eliminates the need to match the entire left and right views. Instead, by integrating binocular stereo vision with a deep learning algorithm, cracks are extracted from the corrected left and right views, respectively, and crack boundary points are matched on the same horizontal line. During use, only the binocular camera needs to capture an image of the same crack. The entire calculation process is fast and highly accurate, effectively matching cracks in the left and right views. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Flowchart of the present invention.

[0024] Figure 2 It is the algorithm flow chart of the present invention.

[0025] Figure 3 This is the image distortion correction and stereo correction process of the present invention.

[0026] Figure 4 This is the U-Net network architecture used in the present invention.

[0027] Figure 5 This is the crack matching process proposed by the present invention.

[0028] Figure 6 This is the neighborhood correlation test process of the left and right matching points in the present invention.

[0029] Figure 7 This is the relocation process of unqualified matching points in the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below with reference to specific embodiments, which are intended to explain the present invention rather than to limit it.

[0031] The present invention discloses a crack image matching method based on binocular stereo vision and U-Net neural network, referring to Figure 1 、 2 , including the following steps:

[0032] (1) The binocular camera was calibrated using a black and white checkerboard. The calibration plate was placed in front of the binocular camera and 12 pairs of images were obtained by changing different angles. Based on Zhang's calibration method, the Stereo Camera Calibrator toolbox in MATLAB was used to import the left camera image and the right camera image respectively and set the size parameters of the calibration plate to calibrate the internal and external parameters of the binocular camera. The binocular camera was used to capture the left and right views of the cracks on the surface of the concrete structure. The cracks could be captured in both the left and right views.

[0033] (2) According to the camera's internal parameters, the distortion of the left and right views is corrected. The specific steps are as follows: Use the undistortImage function in MATLAB to correct the radial distortion and tangential distortion of the image; According to the camera's external parameters, the left and right views are stereo corrected. Use the rectifyStereoImages function in MATLAB to rotate and translate the same points in the left and right views to be on the same horizontal line. Figure 3 The distortion correction and stereo correction process are demonstrated.

[0034] (3) The U-Net semantic segmentation network is trained using open-source images of cracks on the surface of concrete structures. Based on the fully trained U-Net semantic segmentation network, the corrected left view and the corrected right view are used as input, and cracks are extracted from the corrected left view and the corrected right view respectively. Figure 4 The U-Net network architecture is shown.

[0035] (4) Using the Sobel edge detection operator, the edge function in MATLAB is used, and the 'method' parameter is set to 'Sobel' to extract the regional boundaries of the cracks in the corrected left view and the corrected right view; and the crack boundary points on the same horizontal line in the two corrected views are retrieved, and the crack boundary points {L1, L2···L n} and the crack boundary points {R1, R2···R n}.

[0036] (5) Pair the crack boundary points on the same horizontal line in the retrieval order. The i-th point in the corrected left view corresponds to the i-th point in the corrected right view. The matching points {L1 R1, L2 R2, ···L n R n}; Match the crack boundary points on all horizontal lines as the crack matching process, Figure 5 The process of crack matching is demonstrated.

[0037] (6) For the matching point {L i , R i}, for the matching point {L i} and the matching point {R i} is used to perform correlation test on the 31×31 neighborhood where the matching points are located, and the normalized cross-correlation algorithm is used to relocate the matching points that fail the test. The specific steps are as follows:

[0038] First, the matching points {L i} and the matching point {R i} to perform correlation test in the neighborhood: extract matching points {L i} and matching point {R i}, the normalized cross-correlation calculation is performed on the two regions based on the grayscale value, and the calculation formula is:

[0039]

[0040] Where x and y represent the horizontal and vertical coordinates in the neighborhood, respectively, and all pixels in the neighborhood need to be traversed; F(x, y) and T(x, y) represent the grayscale values ​​of the pixels in the corrected right view and the corrected left view, respectively, and μ F With μ T Respectively represent the grayscale mean of the neighborhood of the corrected right view and the corrected left view, σ F and σ T is the grayscale variance of the neighborhood of the corrected right view and the corrected left view, Figure 6 The correlation test process is demonstrated.

[0041] Then, if the correlation between the two regions is greater than or equal to 0.5, the matching point {L i , R i} qualified; if the correlation between the two regions is less than 0.5, the original matching point {R i} is considered as a candidate matching point area, and 31×31 neighborhoods are extracted with each pixel in the candidate area as the center. The correlation between these neighborhoods and the neighborhood in the corrected left view is calculated, and the center point of the neighborhood with the highest correlation is used as the new matching point {R i}, Figure 7 The process of matching point relocalization is demonstrated.

[0042] The above description is merely a preferred embodiment of the present invention and is not intended to impose any limitation on the technical solution of the present invention. Those skilled in the art should understand that, without departing from the spirit and principles of the present invention, the technical solution can also be subjected to several simple modifications and replacements, and these modifications and replacements are also within the scope of protection covered by the claims.

Claims

1. A crack image matching method based on binocular stereo vision and U-Net neural network, characterized in that: The steps are as follows: (1) Use a binocular camera to obtain the left and right views of the cracks on the concrete structure surface; (2) performing distortion correction and stereo correction on the left view and the right view respectively to obtain a corrected left view and a corrected right view; (3) Using the U-Net semantic segmentation network to extract cracks in the rectified left and right images respectively; (4) Extract the crack boundary and retrieve the crack boundary points on the same horizontal line from left to right to obtain the crack boundary points {L1, L2···L n } and the crack boundary points {R1, R2···R n }; (5) On the same horizontal line, all the crack boundary points in the corrected left view and all the crack boundary points in the corrected right view are matched to obtain the matching points {L1 R1, L2 R2, ···L n R n }; (6) Perform a correlation test on the neighborhood where the matching points are located, and re-match the matching points that fail the test.

2. The crack image matching method based on binocular stereo vision and U-Net neural network according to claim 1, characterized in that: In step (1), before obtaining the left view and right view of the surface cracks of the concrete structure, the binocular camera is calibrated using a black and white checkerboard. The calibration plate is placed in front of the binocular camera and 12 pairs of images are obtained by changing different angles. The intrinsic and extrinsic parameters of the binocular camera are calculated using the Zhang calibration method.

3. The crack image matching method based on binocular stereo vision and U-Net neural network according to claim 2, characterized in that: In step (2), the left view and the right view are distorted according to the camera internal parameters.

4. The crack image matching method based on binocular stereo vision and U-Net neural network according to claim 2, characterized in that: In step (2), the left view and the right view are stereo rectified according to the camera external parameters so that the same points in the left view and the right view are located on the same horizontal line.

5. The crack image matching method based on binocular stereo vision and U-Net neural network according to claim 1, characterized in that: In step (3), the U-Net semantic segmentation network is first trained using open-source concrete structure surface crack images. Then, based on the trained U-Net semantic segmentation network, the corrected left view and the corrected right view are imported to perform crack segmentation respectively.

6. The crack image matching method based on binocular stereo vision and U-Net neural network according to claim 1, characterized in that: In step (4), the Sobel edge detection operator is used to extract the boundaries of the crack area in the corrected left view and the corrected right view; and on the same horizontal line, the crack boundary points {L1, L2···L n } and the crack boundary points {R1, R2···R n }.

7. The crack image matching method based on binocular stereo vision and U-Net neural network according to claim 1, characterized in that: In step (5), the crack boundary points on the same horizontal line are paired in the retrieval order, that is, the i-th crack boundary point in the corrected left view corresponds to the i-th crack boundary point in the corrected right view, and the matching points {L1 R1, L2R2, ···L n R n }, in order to match the crack boundary points on all horizontal lines.

8. The crack image matching method based on binocular stereo vision and U-Net neural network according to claim 1, characterized in that: In step (6), during the inspection, the matching points {L i , R i }Matching point{L i }, matching point {R i }, the normalized cross-correlation calculation is performed on the two regions based on the grayscale value. If the correlation between the two regions is greater than or equal to 0.5, the matching point is qualified; if the correlation between the two regions is less than 0.5, the matching point is unqualified.

9. The crack image matching method based on binocular stereo vision and U-Net neural network according to claim 8, characterized in that: When the matching point test fails, according to the normalized cross-correlation algorithm, the original matching point {R i } and re-determine the matching points within the 11×11 neighborhood of .

Images