A road crack width measuring method based on binocular stereo vision

Abstract

The application provides a road crack width measurement method based on binocular stereo vision, which comprises the following steps: firstly, left and right view images of a road crack target are collected by using a binocular stereo camera, and a set of stereo matching point pairs is obtained by performing stereo matching on the left and right view images; secondly, a space-pixel distance relationship in a road crack collection scene is obtained based on the set of stereo matching point pairs; then, a CAMII-UNet road crack segmentation network model which is designed according to the characteristics of road cracks and trained in a road crack data set is used to perform road crack segmentation on the left view image, and a pixel width of the segmented road crack in the left view image is calculated by using a crack width measurement method; finally, the real space width of the road crack is calculated by using the space-pixel distance relationship in the road crack collection scene. The application enhances the segmentation capability of the network for small cracks, effectively improves the accuracy of road crack segmentation, and realizes high-precision road crack width measurement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of road crack width measurement, in particular to a road crack width measurement method based on binocular stereo vision. BACKGROUND

[0002] After the traffic road surface is rolled by large vehicles or naturally worn, cracks and damages are easily generated, and the road condition will deteriorate over time, which often becomes a fatal factor threatening road safety. The formation of cracks is not only directly related to the structural capacity of infrastructure, but also significantly affects the serviceability of infrastructure. The existing manual road crack detection has the disadvantages of low efficiency, high missed detection and misdiagnosis rate, and poor timeliness, therefore, how to detect cracks in the early stage and continuously observe the crack changes to help the later preventive repair has become a research hotspot. In recent years, a large number of scholars have conducted in-depth research on crack detection and proposed many methods, which can be roughly divided into two categories: image processing and deep learning. The image processing method mainly includes threshold-based and edge detection-based methods, which mostly need prior knowledge and are manually selected and designed features. The deep learning method trains a convolutional neural network through a road crack dataset, so that the obtained network model has the ability to extract high-level crack features, and is suitable for crack detection in various natural scenes.

[0003] In the intelligent transportation system, road crack width measurement of the traffic scene is an important link in road quality evaluation. Crack width measurement can not only judge the durability of the road surface and other infrastructure, but also judge the severity of crack damage, which is an important basis for traffic facility managers to make reasonable decisions, and helps staff to analyze cracks, check hidden dangers in engineering construction, segment and locate cracks, and accurately extract crack information. In the existing crack width measurement methods, the most widely used method is to calculate the pixel width and pixel length of the crack through image processing algorithm, but the measurement at the pixel level has a large difference with the real crack width due to the near-wide and far-narrow effect, and it is also difficult to be applied in practical application. SUMMARY

[0004] In view of the technical problems of low accuracy of the existing crack segmentation algorithm and the difficulty of the existing image-based crack width measurement to be applied in practical use, the present application provides a road crack width measurement method based on binocular stereo vision, which uses the binocular stereo vision principle and combines the crack segmentation algorithm of deep learning to realize simple, accurate and real road crack width measurement.

[0005] The technical scheme of the present application is as follows:

[0006] A road crack width measurement method based on binocular stereo vision, the steps are as follows:

[0007] Step 1: Use a binocular stereo camera to acquire left and right viewpoint images of the road crack target;

[0008] Step 2: Use the SURF matching algorithm to perform stereo matching on the left and right viewpoint images to obtain a set of stereo matching point pairs;

[0009] Step 3: Select the two pairs of stereo matching points that are farthest apart in the two-dimensional pixel distance in the left viewpoint image from the set of stereo matching point pairs. Use Zhang Zhengyou calibration method to obtain the three-dimensional spatial coordinates of the two spatial points corresponding to the two selected pairs of stereo matching points respectively. Calculate the three-dimensional spatial distance between the two spatial points based on the three-dimensional spatial coordinates. Obtain the spatial-pixel distance relationship in the road crack acquisition scene based on the ratio of the three-dimensional spatial distance between the two spatial points to their two-dimensional pixel distance in the left viewpoint image.

[0010] Step 4: The left viewpoint image is segmented into a binary image by using the CAMII-UNet road crack segmentation network model, which conforms to the characteristics of road cracks and has been trained in the road crack dataset.

[0011] Step 5: Calculate the pixel width of the segmented road crack in the left viewpoint image using the crack width measurement method;

[0012] Step 6: Calculate the true spatial width of the road crack using the spatial-pixel distance relationship in the road crack acquisition scene, thus realizing the road crack width measurement.

[0013] Preferably, the set of stereo matching point pairs is S = {(P} Li ,P Ri ), i = 1, 2, ..., N}, where P Li (x Li ,y Li ) represents the stereo matching point pair (P) Li ,P Ri The two-dimensional coordinates of P in the left viewpoint image. Ri (x Ri ,y Ri ) represents the stereo matching point pair (P) Li ,P Ri The two-dimensional coordinates of the image in the right viewpoint.

[0014] Preferably, the method for selecting the two pairs of stereo matching points with the greatest two-dimensional pixel distance in the left viewpoint image from the set of stereo matching point pairs is as follows: among the N feature matching points P in the left viewpoint image... Li The algorithm uses the convex hull principle and the caliper algorithm to select the two feature points P that are farthest apart. L1 and P L2 Then, from the set of stereo matching point pairs S = {(PLi P Ri ), i = 1, 2,..., N} are found. L1 (x L1 ,y L1 ), P R1 (x R1 ,y R1 )) and (P L2 (x L2 ,y L2 ), P R2 (x R2 ,y R2 )).

[0015] Preferably, in step three, the three-dimensional space coordinates P1(X1, Y1, Z1) and P2(X2, Y2, Z2) of the two space points respectively corresponding to the selected two pairs of stereo matching points (P L1 (x L1 ,y L1 ), P R1 (x R1 ,y R1 )) and (P L2 (x L2 ,y L2 ), P R2 (x R2 ,y R2 )) are obtained by using Zhang Zhengyou calibration method; in the Zhang Zhengyou calibration method, the calibration board is a black and white square checkerboard, the checkerboard size is 15mm x 15mm, and in the calibration process, the left camera and the right camera each collect 16 images from different collection distances and collection angles of the calibration board in different directions, and the intrinsic and extrinsic parameters of the camera are obtained from the calibration images, wherein the intrinsic parameters include the pixel-level focal length f L , f R of the left camera and the right camera, the principal point coordinates (u L , v L ), (u R , v R ), the lens distortion (k 1L , k 2L ), (k 1R , k 2R ), and the skew variable γ L , γ R ; the extrinsic parameters include the rotation matrix and the translation matrix T = [t x t y t z ] T .

[0016] Preferably, in step three, the space-pixel distance relationship k under the road crack collection scene is:

[0017]

[0018]

[0019] Preferably, the CAMII-UNet road crack segmentation network model comprises an input structure, an encoder and a decoder; wherein the input structure is a multi-scale input image, comprising sequentially downsampling the original image I in the H and W directions to obtain downsampled images I1, I2, I3 and I4, I has a size of HxWxC, I1 is I2 is I3 is I4 is The encoder comprises a first residual convolution module, a first max-pooling module, a second residual convolution module, a second max-pooling module, a third residual convolution module, a third max-pooling module, a fourth residual convolution module, a fourth max-pooling module and a fifth residual convolution module; the decoder comprises a first upsampling module, a sixth residual convolution module, a second upsampling module, a seventh residual convolution module, a third upsampling module, an eighth residual convolution module, a fourth upsampling module, a ninth residual convolution module and a convolution layer;

[0020] The original image I is extracted through the first residual convolution module to obtain a first feature; the first feature is obtained through the CA attention mechanism module to obtain a first output feature;

[0021] The downsampled image I1 is extracted through one residual convolution to obtain a second feature, the first feature extracted through the first max-pooling module is combined with the second feature to obtain a third feature, and the third feature is obtained through the second residual convolution module to obtain a fourth feature; the fourth feature is obtained through the CA attention mechanism module to obtain a second output feature;

[0022] The downsampled image I2 is extracted through two residual convolutions to obtain a fifth feature, the fourth feature extracted through the second max-pooling module is combined with the fifth feature to obtain a sixth feature, and the sixth feature is obtained through the third residual convolution module to obtain a seventh feature; the seventh feature is obtained through the CA attention mechanism module to obtain a third output feature;

[0023] The downsampled image I3 is extracted through three residual convolutions to obtain an eighth feature, the seventh feature extracted through the third max-pooling module is combined with the eighth feature to obtain a ninth feature, and the ninth feature is obtained through the fourth residual convolution module to obtain a tenth feature; the tenth feature is obtained through the CA attention mechanism module to obtain a fourth output feature;

[0024] The down-sampled image I4 is extracted with the eleventh feature through four residual convolution modules, and the twelfth feature is obtained by merging and connecting the feature extracted by the fourth maximum pooling module and the eleventh feature; the first up-sampling feature is input into the first up-sampling module after passing through the fifth residual convolution module to obtain the first up-sampling feature;

[0025] The first up-sampling feature is merged and connected with the fourth output feature, and then sequentially passes through the sixth residual convolution module and the second up-sampling module to obtain the second up-sampling feature;

[0026] The second up-sampling feature is merged and connected with the third output feature, and then sequentially passes through the seventh residual convolution module and the third up-sampling module to obtain the third up-sampling feature;

[0027] The third up-sampling feature is merged and connected with the second output feature, and then sequentially passes through the eighth residual convolution module and the fourth up-sampling module to obtain the fourth up-sampling feature;

[0028] The fourth up-sampling feature is merged and connected with the first output feature, and then sequentially passes through the ninth residual convolution module and the convolution layer to obtain the final output feature.

[0029] Preferably, the residual convolution module comprises a first convolution layer, a second convolution layer and an add feature fusion layer, wherein the input feature sequentially passes through the first convolution layer and the second convolution layer, and the features extracted by the first convolution layer and the second convolution layer are input into the add feature fusion layer for feature fusion to obtain the output feature.

[0030] Preferably, the crack width measurement method comprises: after obtaining the binary image of the road crack segmentation, first, using the axis change method to obtain the crack skeleton line, removing the false branches by the discrete skeleton evolution method, and then removing the intersection points by using the connected domain to obtain the crack skeleton; secondly, using the canny algorithm to detect the crack edge; then, selecting M interest points on the crack skeleton with a step of 2 pixels, and calculating the local crack pixel width d i of each interest point to the crack edge; i Finally, selecting the maximum value d=max{d i |i=1,2,...,M} from the M d i s corresponding to the M interest points as the pixel width of the road crack in the entire image.

[0031] Preferably, the local crack pixel width d iThe method for calculating the real space width of the road crack is: first, obtaining the crack direction vector of the pixel position where the interest point is located; second, obtaining a local boundary point set of the crack edge within a certain radius range; then, dividing the local boundary point set into two boundary point candidate groups according to the similarity of the vector of the connection line between the interest point and the local boundary point and the crack direction vector in the direction; then, selecting the two points closest to the interest point from the first boundary point candidate group and the second boundary point candidate group respectively; finally, calculating the Euclidean distance between the pixel coordinates of the two points as the local crack pixel width d of the interest point i .

[0032] Preferably, the method for calculating the real space width of the road crack is: multiplying the space-pixel distance relationship k and the road crack pixel width d to obtain the real space width D of the road crack.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] 1) The road crack segmentation network CAMII-UNet of the present application adds a residual structure to the original VGG block for feature fusion, designs a multi-scale image input expansion network to expand the receptive field, and fuses a CA attention mechanism to fully capture the crack target information in the image, thereby enhancing the network's segmentation ability for small cracks, solving the problem of poor segmentation effect caused by the diversity of road cracks and the imbalance between classes, and effectively improving the precision of road crack segmentation.

[0035] 2) The present application uses CAMII-UNet to accurately segment the road cracks in the left-view image, uses the hybrid width measurement method to obtain the accurate pixel width of the segmented road cracks, combines the space-pixel distance relationship calculated based on the principle of binocular stereo vision, and indirectly calculates the actual space width of the road cracks, effectively solving the problem that the existing image processing type road crack width measurement method is difficult to achieve accurate measurement based on the two-dimensional pixel width to estimate the actual space width, and realizing high-precision actual width measurement of road cracks. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0037] Figure 1 is the overall flowchart of the present application;

[0038] Figure 2 is a schematic diagram of the binocular stereo acquisition system of the present application;

[0039] Figure 3 Structure diagram of the CAMII-UNet network of the present application;

[0040] Figure 4 Structure diagram of the residual convolution module of the present application;

[0041] Figure 5 Comparison diagram of road crack segmentation results of the present application;

[0042] Figure 6 Comparison diagram of road crack segmentation network feature layer visualization of the present application;

[0043] Figure 7 Schematic diagram of the phenomenon of near wide and far narrow in road crack image acquisition;

[0044] Figure 8 Schematic diagram of road crack binary graph width measurement processing of the present application; wherein, (a) is a crack binary graph, (b) is a crack original skeleton graph, (c) is a skeleton graph after pruning, (d) is a crack skeleton graph after removing intersection points, (e) is a crack edge graph, and (f) is a crack width measurement graph. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0046] As Figure 1As shown, the embodiment of the present application provides a road crack width measurement method based on binocular stereo vision. First, a binocular stereo camera is used to collect left and right view images containing road crack targets; stereo matching is performed on the left and right view images to obtain a set of stereo matching point pairs; second, the two pairs of stereo matching points with the farthest two-dimensional pixel distance in the left view image are selected from the obtained set of stereo matching point pairs, Zhang Zhengyou calibration method is used to obtain the three-dimensional space coordinates of the two space points corresponding to the selected two pairs of stereo matching points, the three-dimensional space distance between the two space points is calculated according to the three-dimensional space coordinates, and the space-pixel distance relationship under the road crack collection scene is obtained according to the ratio of the three-dimensional space distance of the two space points to their two-dimensional pixel distance in the left view image; third, the CAMII-UNet road crack segmentation network model designed according to the characteristics of road cracks and trained in the road crack data set is used to perform road crack segmentation on the left view image; then, the pixel width of the segmented road crack in the left view image is calculated using the crack width measurement method; finally, the real space width of the road crack is further calculated using the space-pixel distance relationship under the road crack collection scene, and road crack width measurement is realized. The specific steps are as follows:

[0047] Step one: Use a binocular stereo camera to collect left and right view images of road crack targets; as shown in Figure 2 , the binocular stereo camera acquisition system includes two 2448x2048 pixel resolution, 8mm fixed focus WOLVES WL1608-5MP lenses, 5 million pixel resolution Hikvision MV-CA050-11UC industrial cameras, an Intel Core i7-10750H CPU, 16GB of memory, an NVIDIA RTX2060 graphics card with 6G of video memory, a notebook computer, a set of binocular stereo camera supports, and a 15mmx15mm size black and white square chessboard. The notebook computer controls the binocular stereo camera shooting and data transmission through the USB3.0 interface.

[0048] Step two: use SURF matching algorithm to perform stereo matching on left and right view images to obtain a set of stereo matching point pairs; as shown in Figure 1 , use SURF matching algorithm to perform stereo matching on left and right view images to obtain a set of stereo matching point pairs S={(P Li ,P Ri ),i=1,2,...,N}, where P Li (x Li ,y Li ) represents the two-dimensional coordinates of the stereo matching point pair (P Li ,P Ri ) in the left view image, P Ri (xRi , y Ri ) represents the two-dimensional coordinates of the stereo matching point pair (P Li , P Ri ) in the right view image.

[0049] Step three: selecting two pairs of stereo matching points with the furthest two-dimensional pixel distance in the left view image from the set of stereo matching points, using Zhang Zhengyou calibration method to obtain the three-dimensional coordinates of the two spatial points corresponding to the selected two pairs of stereo matching points, calculating the three-dimensional spatial distance between the two spatial points according to the three-dimensional spatial coordinates, and obtaining the spatial-pixel distance relationship under the road crack collection scene according to the ratio of the three-dimensional spatial distance of the two spatial points to their two-dimensional pixel distance in the left view image.

[0050] As shown in Figure 1 , two pairs of stereo matching points with the furthest two-dimensional pixel distance in the left view image are selected from the set of stereo matching points: two feature points P Li and P L1 with the furthest distance are selected from N feature matching points P L2 in the left view image using the convex hull principle and the jamming algorithm; and then the two pairs of stereo matching points (P Li , P Ri ) and (P L1 , P L1 ) are found from the set of stereo matching points S = {(P L1 , P R1 ), i = 1, 2,..., N}. R1 R1 L2 L2 L2 R2 R2 R2

[0051] Zhang Zhengyou calibration method is used to obtain the three-dimensional coordinates of the two pairs of stereo matching points (P L1 , P L1 ) and (P L1 , P R1 ). R1 R1 L2 L2 L2 R2 R2 R2 ​​​​​​​​​​​​​​​The three-dimensional spatial coordinates of the two spatial points corresponding to the given points are P1(X1,Y1,Z1) and P2(X2,Y2,Z2), respectively. In the Zhang Zhengyou calibration method, the calibration board is a black and white square checkerboard with a size of 15mm×15mm. During the calibration process, the left and right cameras each acquire 16 images of the calibration board from different acquisition distances and angles. The intrinsic and extrinsic parameters of the cameras are obtained from these 16 calibration images. The intrinsic parameters include the pixel-level focal length f of the left camera. L The pixel-level focal length f of the right camera R Principal point coordinates (u L ,v L ), (u R ,v R Lens distortion (k) 1L ,k 2L ), (k 1R ,k 2R ), slope variable γ L γ R ;External parameters include rotation matrix Translation matrix T = [t] x t y t z ] T Tables 1 and 2 show the internal and external parameters of a randomly calibrated set of binocular stereo cameras.

[0052] Table 1 Internal parameters of the binocular stereo camera

[0053]

[0054] Table 2 External parameters of the binocular stereo camera

[0055]

[0056] Based on the three-dimensional spatial distance between two spatial points P1(X1,Y1,Z1) and P2(X2,Y2,Z2) Rather than P in the left viewpoint image L1 (x L1 ,y L1 ) and P L2 (x L2 ,y L2 ) two-dimensional pixel distance The spatial-pixel distance relationship k in the road crack acquisition scene is obtained by calculating the ratio:

[0057]

[0058] Table 3 shows the spatial-pixel distance relationship k calculated by the method of the present application at multiple different collection positions of the binocular stereo camera. As shown in Table 3, due to the need to move the camera when shooting different cracks, the camera height also changes slightly due to different ground conditions, and the calculated spatial-pixel distance relationship k also changes slightly. This also indicates that the pixel width of the crack cannot accurately evaluate the true damage state of the crack, and the true crack width measurement problem needs to be solved.

[0059] Table 3 spatial-pixel distance relationship k calculation example

[0060]

[0061] Step four: performing road crack segmentation on the left view image by the designed CAMII-UNet road crack segmentation network model which conforms to the characteristics of road cracks and is trained in a road crack data set. The road crack data set includes 511 image data containing road cracks selected from existing data sets CFD, Aigle-RN, Crack Tree200, Deep Crack, and 151 self-made road crack image data, wherein the training set, the validation set and the test set are divided according to the ratio of 7:2:1.

[0062] As Figure 3 shown, the CAMII-UNet road crack segmentation network of the present application adds a residual structure to the original VGG block for feature fusion, designs a multi-scale image input expansion network to expand the receptive field, and fuses a CA attention mechanism to fully capture the target information in the image. The CAMII-UNet road crack segmentation network model includes an input structure, an encoder and a decoder; wherein the input structure is a multi-scale input image, including sequentially downsampling the original image I in the H and W directions to obtain downsampled images I1, I2, I3 and I4, I has a size of HxWxC, I1 is I2 is I3 is I4 is The encoder includes a first residual convolution module, a first max pooling module, a second residual convolution module, a second max pooling module, a third residual convolution module, a third max pooling module, a fourth residual convolution module, a fourth max pooling module and a fifth residual convolution module; the decoder includes a first upsampling module, a sixth residual convolution module, a second upsampling module, a seventh residual convolution module, a third upsampling module, an eighth residual convolution module, a fourth upsampling module, a ninth residual convolution module and a convolution layer.

[0063] The original image I is processed by a first residual convolution module to extract the first feature; the first feature is then processed by a CA attention mechanism module to obtain the first output feature; the downsampled image I1 is processed by a single residual convolution to extract the second feature, the first feature is then merged with the feature extracted by the first max pooling module to obtain the third feature, the third feature is then processed by a second residual convolution module to obtain the fourth feature, the fourth feature is then processed by a CA attention mechanism module to obtain the second output feature; the downsampled image I2 is processed by two residual convolutions to extract the fifth feature, the fourth feature is then merged with the feature extracted by the second max pooling module to obtain the sixth feature, the sixth feature is then processed by a third residual convolution module to obtain the seventh feature, the seventh feature is then processed by a CA attention mechanism module to obtain the third output feature; the downsampled image I3 is processed by three residual convolutions to extract the eighth feature, the seventh feature is then merged with the feature extracted by the third max pooling module to obtain the ninth feature, the ninth feature is then processed by a fourth residual convolution module to obtain the... The tenth feature; the tenth feature is processed by the CA attention mechanism module to obtain the fourth output feature; the downsampled image I4 is processed by four residual convolutions to extract the eleventh feature, and the feature extracted by the tenth feature by the fourth max pooling module is merged and concatenated with the eleventh feature to obtain the twelfth feature; the twelfth feature is processed by the fifth residual convolution module and then input into the first upsampling module to obtain the first upsampled feature; the first upsampled feature is merged and concatenated with the fourth output feature and then processed by the sixth residual convolution module and the second upsampling module to obtain the second upsampled feature; the second upsampled feature is merged and concatenated with the third output feature and then processed by the seventh residual convolution module and the third upsampling module to obtain the third upsampled feature; the third upsampled feature is merged and concatenated with the second output feature and then processed by the eighth residual convolution module and the fourth upsampling module to obtain the fourth upsampled feature; the fourth upsampled feature is merged and concatenated with the first output feature and then processed by the ninth residual convolution module and the convolutional layer to obtain the final output feature.

[0064] like Figure 4 As shown, the residual structure added to the original VGG block for feature fusion, i.e., the residual convolution module, includes: a first convolutional layer, a second convolutional layer, and an add feature fusion layer. The input features pass through the first convolutional layer and the second convolutional layer in sequence, and the features extracted by the first convolutional layer and the features extracted by the second convolutional layer are input to the add feature fusion layer for feature fusion to obtain the output features.

[0065] Figure 5Fig. 6 is a road crack segmentation result comparison chart of the present application. The crack segmentation binary image obtained by the CAMII-UNet road crack segmentation network of the present application is compared with the crack segmentation binary images obtained by the other six networks. In the binary image, the value 1 corresponds to the crack area, and the value 0 corresponds to the background area. For the one-way crack image in the first column, the original crack is relatively small, and the green box part has very small crack branches. The CAMII-UNet network of the present application can segment the small cracks in this part. In contrast, the other six networks cannot well segment the small cracks in this part. For the one-way crack in the second column, the original crack has a relatively wide width, but there are many stone particles in the crack gap. The CAMII-UNet network and the Deeplab v3+ of the present application can well segment the crack in this part without holes. In contrast, the other five networks have many holes and different degrees of loss. For the blocky crack in the third column and the fourth column, the original crack has small cracks. The CAMII-UNet network and the Attention-UNet of the present application can segment the small cracks in the green box part. In contrast, the other five networks cannot completely segment the small cracks in this part. For the crack in the fifth column, the overall crack pattern is light and thin, and it is not easy to identify. The CAMII-UNet network of the present application can segment most of the cracks. In contrast, the other six networks have different degrees of loss. The crack segmented by the FCN is relatively wide and has poor segmentation effect on small cracks. The SegNet, Deeplab v3+, UNet and Attention-UNet have poor continuity and have many discontinuities. The PSPNet completely fails to segment the small cracks. For the crack in the sixth column, the original crack is relatively easy to identify. Except for the FCN and the PSPNet, the CAMII-UNet network and the other three networks can segment the cracks in the green box part. In summary, the CAMII-UNet network of the present application has good segmentation effect on the segmentation of one-way cracks, blocky cracks, cracks, small cracks and wide cracks.

[0066] Table 4 shows the network performance comparison of UNet and its improved networks Attention-UNet, CA-UNet and CAMII-UNet. As can be seen from Table 4, the CAMII-UNet network achieves the optimal value in five of the six indicators G, C, mIOU, R and F among the network performance comparison, wherein, compared with the base network UNet, G is improved by 0.12%, reaching 98.99%; C is improved by 0.85%, reaching 90.62%; mIOU is improved by 1.47%, reaching 84.44%; R is improved by 1.62%, reaching 81.74%; F is improved by 1.99%, reaching 82.30%. Although P does not exceed the performance of Attention-UNet and CA-UNet, P is improved by 2.36% compared with the base network UNet, reaching 82.86%. In summary, the network performance of the CAMII-UNet of the application is optimal.

[0067] Table 4 network performance comparison

[0068]

[0069] Figure 6 is a visualization comparison chart of the feature layers of the road crack segmentation network of the application. Figure 6 The 8-layer visualization feature maps of the UNet network and the CAMII-UNet network of the application are compared. Figure 6 (a) is a small crack image containing shoes that interfere with crack segmentation. As can be seen, the 7th layer and the 8th layer of UNet obviously focus on the shoe part, while the 7th layer and the 8th layer of CAMII-UNet focus more on the crack itself and are less affected by the shoes. Figure 6 (b) is a relatively wide crack image containing broken stones. As can be seen, the 1st layer, the 2nd layer and the 3rd layer of CAMII-UNet more effectively suppress the background and focus on the crack than UNet; the 5th layer, although belonging to the high-level feature part, still focuses on the edge of the crack; the 6th layer, the 7th layer and the 8th layer of CAMII-UNet also show better segmentation effect. Figure 6 (c) is a crack image containing a large shadow that interferes with crack segmentation. As can be seen, the CAMII-UNet network does not pay too much attention to the shadow part, and more attention is allocated to the crack area in the 3rd layer, the 5th layer, the 6th layer, the 7th layer and the 8th layer. Figure 6(d) The CAMII-UNet network shows obvious benefits in suppressing background features and focusing on crack features in the 3rd, 5th and 8th layers. In summary, the CAMII-UNet of the application can not only suppress the network from focusing on other targets other than cracks, but also retain clearer crack edge information in the deeper layers of the network, and has a more fine ability to extract crack features.

[0070] Step five: calculate the pixel width of the segmented road crack in the left view image using the crack width measurement method.

[0071] Figure 7 The phenomenon of near wide and far narrow in road crack image collection is given. Figure 7 An exemplary schematic diagram of a road crack image taken by two different height positions is shown, wherein the A position is 2L away from the ground, and the B position is L away from the ground. Figure 7 (a) shows the A position image, Figure 7 (b) shows the B position image, it can be seen that the crack range taken by the A position is larger, and the pixel width of the crack is smaller, while the crack range taken by the B position is smaller, and the pixel width of the crack is larger. Figure 7 (c) shows the binarization image of the A position image, Figure 7 (d) shows the binarization image of the B position image, Figure 7 (e) shows the local pixel visualization diagram of a certain area of the A position binarization image, and Figure 7 (f) shows the local pixel visualization diagram of the same area of the B position binarization image. It can be observed that in Figure 7 (e), the pixel width of the selected area of the A position image is 1-2 pixels, while in Figure 8 (f), the pixel width of the same selected area of the B position image is 2-3 pixels, which shows that the pixel width of the crack in the image is affected by the shooting height, and is near wide and far narrow, which cannot be used to represent the actual width of the crack, and is difficult to be put into practical application. Therefore, it is necessary to further use binocular stereo vision information to determine different shooting heights, convert the crack pixel width to the actual width, and eliminate the influence of different shooting heights on the measurement of the actual width of the crack.

[0072] Figure 8 The road crack binarization image width measurement method of the application is exemplarily shown. As shown in Figure 8 (a), the trained CAMII-UNet road crack segmentation network model segments the road crack in the left view image into a crack binarization image. First, the original skeleton line of the crack is obtained using the center axis change method, as shown in Figure 8(b) shows; the pruned skeleton is obtained by removing false branches through discrete skeleton evolution method, as shown in Figure 8 (c) shows; the cross point removed skeleton is obtained by removing cross points using connected domain, as shown in Figure 8 (d) shows. Secondly, the crack edge is detected using canny algorithm, as shown in Figure 8 (e) shows; then, M interest points on the crack skeleton are selected with a step of 2 pixels, and the local crack pixel width d i of each interest point to the crack edge is calculated separately. i Finally, the maximum value d = max{d i |i = 1, 2,..., M} from M d i corresponding to M interest points is selected as the pixel width of the road crack in the whole image, as shown in Figure 1 (f) shows.

[0073] The process of calculating the local crack pixel width d i of each interest point to the crack edge separately includes: firstly, obtaining the crack direction vector of the pixel position of the interest point; secondly, obtaining the local boundary point set of the crack edge within a certain radius range; then, according to the similarity of the vector of the connection line from the interest point to the local boundary point and the crack direction vector in the direction, the local boundary point set is divided into two boundary point candidate groups; after that, the two points closest to the interest point are selected from the first boundary point candidate group and the second boundary point candidate group respectively; finally, the Euclidean distance between the pixel coordinates of the two points is calculated as the local crack pixel width d i of the interest point.

[0074] Step six: the real space width of the road crack is calculated using the space-pixel distance relationship k in the road crack collection scene, and the road crack width measurement is realized.

[0075] The real space width D = kd of the road crack is obtained by multiplying the space-pixel distance relationship k and the road crack pixel width d.

[0076] As shown in ​ , the road crack width measurement method based on binocular stereo vision proposes a corresponding model, including image collection and stereo matching module, space-pixel distance relationship calculation module, road crack segmentation module, road crack pixel width measurement module, and road crack actual width calculation module.

[0077] Table 5 exemplarily shows the actual crack width measured by the crack width measurement method of the present application at different shooting heights. As can be seen, the space-pixel distance relationship k values of the 14 exemplary measurements shown are different, which indicates that they are made at different shooting heights, so the crack width measurement method of the present application is not limited to a single fixed shooting height.

[0078] Table 5 Actual crack width measured by the method of the present application

[0079]

[0080] The above description is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A road crack width measurement method based on binocular stereo vision, characterized by, The steps are as follows: Step one: using binocular stereo camera to collect left view image and right view image of road crack target; Step two: using SURF matching algorithm to perform stereo matching on left view image and right view image to obtain a set of stereo matching point pairs; Step three: selecting two pairs of stereo matching points with the farthest two-dimensional pixel distance in the left view image from the set of stereo matching point pairs, using Zhang Zhengyou calibration method to obtain the three-dimensional space coordinates of the two space points corresponding to the selected two pairs of stereo matching points, calculating the three-dimensional space distance between the two space points according to the three-dimensional space coordinates, and obtaining the space-pixel distance relationship under the road crack collection scene according to the ratio of the three-dimensional space distance of the two space points to the two-dimensional pixel distance thereof in the left view image; Step four: performing road crack segmentation on the left view image into a binary image through the designed CAMII-UNet road crack segmentation network model which conforms to the characteristics of road cracks and is trained in a road crack data set; Step five: calculating the pixel width of the segmented road crack in the left view image using the crack width measurement method; The crack width measurement method includes: after obtaining a binary image of the road crack segmentation, firstly, the crack skeleton line is obtained using the median transformation method, false branches are removed using the discrete skeleton evolution method, and then intersections are removed using connected components to obtain the crack skeleton; secondly, the crack edge is detected using the Canny algorithm; then, M interest points on the crack skeleton are selected with a step size of 2 pixels, and the local crack pixel width d from the location of each interest point to the crack edge is calculated individually. i Finally, from the M points of interest corresponding to the M d i Select the maximum value d = max{d i |i=1,2,...,M} represents the pixel width of the road crack in the entire image; The local crack pixel width d from the crack edge of each point of interest is calculated individually. i The method is as follows: First, obtain the crack direction vector at the pixel location of the point of interest; second, obtain the set of local boundary points within a certain radius of the crack edge; then, divide the set of local boundary points into two candidate boundary point groups based on the similarity in direction between the vector connecting the point of interest to the local boundary points and the crack direction vector; next, select the two points closest to the point of interest from the first and second candidate boundary point groups respectively; finally, calculate the Euclidean distance between the pixel coordinates of these two points as the local crack pixel width d of the point of interest. i ; Step six: calculating the real space width of the road crack using the space-pixel distance relationship under the road crack collection scene to realize road crack width measurement.

2. The binocular stereo vision-based road crack width measuring method according to claim 1, characterized in that, The set of stereo matching point pairs is S = {(P Li , P Ri ), i = 1, 2,..., N}, where P Li (x Li ,y Li ) represents the two-dimensional coordinates of the stereo matching point pair (P Li , P Ri ) in the left-view image, and P Ri (x Ri ,y Ri ) represents the two-dimensional coordinates of the stereo matching point pair (P Li , P Ri ) in the right-view image.

3. The binocular stereo vision-based road crack width measuring method according to claim 2, characterized in that, The method for selecting the two pairs of stereo matching points with the greatest two-dimensional pixel distance in the left view image from the set of stereo matching point pairs is as follows: From the N feature matching points P in the left view image... Li The algorithm uses the convex hull principle and the caliper algorithm to select the two feature points P that are farthest apart. L1 and P L2 Then, from the set of stereo matching point pairs S = {(P Li ,P Ri Find these two pairs of stereo matching points (P) in the {i = 1, 2, ..., N}. L1 (x L1 ,y L1 ),P R1 (x R1 ,y R1 )) and (P L2 (x L2 ,y L2 ),P R2 (x R2 ,y R2 )).

4. The binocular stereo vision-based road crack width measuring method according to claim 3, characterized in that, In step three, the Zhang Zhengyou calibration method is used to obtain the three-dimensional space coordinates P1(X1, Y1, Z1) and P2(X2, Y2, Z2) of the two space points respectively corresponding to the two pairs of selected stereo matching points (P L1 (x L1 ,y L1 ),P R1 (x R1 ,y R1 )) and (P L2 (x L2 ,y L2 ),P R2 (x R2 ,y R2 )) in the stereo image pair. L In the Zhang Zhengyou calibration method, the calibration board is a black and white square checkerboard, and the checkerboard size is 15mm*15mm. In the calibration process, the left camera and the right camera each capture 16 images of the calibration board from different distances and angles. The intrinsic and extrinsic parameters of the cameras are obtained from these calibration images, wherein the intrinsic parameters include the pixel-level focal length f R , f L , the principal point coordinates (u L , v R ), (u R , v 1L ), the lens distortion (k 2L , k 1R ), (k 2R , k L ), and the skew variable γ R , γ x ; and the extrinsic parameters include the rotation matrix and the translation matrix T=[t y t z t T ​ 5. The binocular stereo vision-based road crack width measuring method according to claim 4, characterized in that, In step three, the space-pixel distance relationship k under the road crack collection scene is: wherein, is the three-dimensional spatial distance of two spatial points P1(X1, Y1, Z1) and P2(X2, Y2, Z2), is the two-dimensional pixel distance of two spatial points P1(X1, Y1, Z1) and P2(X2, Y2, Z2) in the left view image P L1 (x L1 ,y L1 ) and P L2 (x L2 ,y L2 ) in the left view image P 6. The binocular stereo vision based road crack width measurement method according to claim 1, wherein, The CAMII-UNet road crack segmentation network model comprises an input structure, an encoder and a decoder; wherein the input structure is a multi-scale input image, comprising sequentially performing down-sampling on an original image I in H and W directions to obtain down-sampled images I1, I2, I3 and I4, I has a size of HxWxC, I1 is I2 is I3 is I4 is The encoder comprises a first residual convolution module, a first maximum pooling module, a second residual convolution module, a second maximum pooling module, a third residual convolution module, a third maximum pooling module, a fourth residual convolution module, a fourth maximum pooling module and a fifth residual convolution module; the decoder comprises a first up-sampling module, a sixth residual convolution module, a second up-sampling module, a seventh residual convolution module, a third up-sampling module, an eighth residual convolution module, a fourth up-sampling module, a ninth residual convolution module and a convolution layer; The original image I is extracted through the first residual convolution module; the first feature is obtained through the CA attention mechanism module; The down-sampled image I1 is extracted through one residual convolution; the feature extracted by the first maximum pooling module is combined with the second feature to obtain the third feature, and the third feature is obtained through the second residual convolution module; the fourth feature is obtained through the CA attention mechanism module; The down-sampled image I2 is extracted through two residual convolutions; the feature extracted by the second maximum pooling module is combined with the fifth feature to obtain the sixth feature, and the sixth feature is obtained through the third residual convolution module; the seventh feature is obtained through the CA attention mechanism module; The down-sampled image I3 is extracted through three residual convolutions; the feature extracted by the third maximum pooling module is combined with the eighth feature to obtain the ninth feature, and the ninth feature is obtained through the fourth residual convolution module; the tenth feature is obtained through the CA attention mechanism module; The down-sampled image I4 is extracted through four residual convolutions; the feature extracted by the fourth maximum pooling module is combined with the eleventh feature to obtain the twelfth feature; the first up-sampling feature is obtained after the fifth residual convolution module; The first up-sampling feature and the fourth output feature are combined and connected, and then sequentially pass through the sixth residual convolution module, the second up-sampling module to obtain the second up-sampling feature; The second up-sampling feature and the third output feature are combined and connected, and then sequentially pass through the seventh residual convolution module, the third up-sampling module to obtain the third up-sampling feature; The third up-sampling feature is connected with the second output feature, and then sequentially passes through an eighth residual convolution module and a fourth up-sampling module to obtain a fourth up-sampling feature; The fourth up-sampling feature is connected with the first output feature, and then sequentially passes through a ninth residual convolution module and a convolution layer to obtain a final output feature.

7. The binocular stereo vision-based road crack width measuring method according to claim 6, characterized in that, The residual convolution module comprises a first convolution layer, a second convolution layer and an add feature fusion layer, wherein the input feature sequentially passes through the first convolution layer and the second convolution layer, and the features extracted by the first convolution layer and the second convolution layer are input into the add feature fusion layer for feature fusion to obtain an output feature.

8. The binocular stereo vision based road crack width measurement method according to claim 1, wherein, The method for calculating the real space width of the road crack is: multiplying the space-pixel distance relationship k and the pixel width d of the road crack to obtain the real space width D=kd of the road crack.

Images