Bridge appearance damage binocular vision intelligent detection method and system fusing deep learning

Abstract

The application discloses a bridge apparent damage binocular vision intelligent detection method and system fusing deep learning, and comprises the following steps: constructing a training data set of each bridge part, bridge component and damage category respectively; based on an improved VGG16 model, constructing a part recognition model, a component recognition model and a damage recognition model respectively to obtain a three-layer pre-classification model; based on an improved encoder-decoder FCN model, constructing a pixel-level detection model of each damage category; then, training the multi-layer pre-classification model and the pixel-level detection model by using corresponding training data sets respectively, performing online image detection, and obtaining the damage number, shape contour and inclination direction of the pre-classified bridge apparent image; based on the center projection model of binocular stereo vision, performing three-dimensional reconstruction and quantitative measurement on the image shape contour of the damage to obtain the damage size. The application can output comprehensive apparent damage information such as position, category and size, and provide an objective basis for the service bridge apparent state evaluation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of bridge structure appearance information detection, and in particular to a bridge appearance damage binocular vision intelligent detection method and system fusing deep learning. BACKGROUND

[0002] In recent years, unmanned aerial vehicle (UAV) assisted machine vision based bridge detection methods are gradually emerging. Many studies combine deep learning methods to efficiently and accurately detect bridge appearance damage. However, the existing bridge detection systems still have the following problems in appearance information detection: (1) the appearance damage information extraction is not perfect or the information is not integrated, and the artificial detection results still need to be combined to further apply to bridge appearance state evaluation; (2) the specific damage characteristics of different components on different bridge parts are not considered, which leads to interference between similar damages at different bridge locations, reducing the damage recognition accuracy; (3) the influence weight of damage severity at different bridge locations is not considered, which leads to the fact that the measurement results cannot directly provide reference for damage quantitative evaluation; (4) the training ideas of related damage recognition or detection models are single, and the application is limited. SUMMARY

[0003] In view of the deficiencies of the prior art, the present application provides a bridge appearance damage binocular vision intelligent detection method and system fusing deep learning, and the specific technical solutions are as follows:

[0004] A bridge appearance damage binocular vision intelligent detection method fusing deep learning, the method comprises the following steps:

[0005] Step 1: acquire bridge appearance images through a binocular shooting system, and label the part of the bridge, the specific component of the bridge where the damage is located, and the damage category for each image, and classify the images according to the part of the bridge, and construct a training data set for each bridge part; then further classify the images of each bridge part according to the specific component of the bridge where the damage is located, and construct a training data set for each bridge component of each bridge part; finally, classify the images of each bridge component according to the damage category, and construct a training data set for each damage category of each bridge component of each bridge part;

[0006] Step two: build an improved VGG16 model, i.e. modify the number of neurons of the three fully connected layers of the VGG16 model to 1024, 512 and the number of recognition categories, then train the improved VGG16 model with the training data set of each bridge part respectively to obtain the part recognition model corresponding to each bridge part; then train the part recognition model corresponding to each bridge part with the training data set of each bridge member to obtain the member recognition model corresponding to the member; finally, train the member recognition model corresponding to the member with the training data set of each damage category of each bridge member of each bridge part to obtain the damage recognition model corresponding to each damage category of the bridge member; thereby obtaining a three-level pre-classification model;

[0007] Step three: build an improved encoder-decoder FCN model, i.e. use a VGG19-based encoder network as a feature extraction network, and superimpose the encoder maximum pooling index onto the corresponding same resolution decoder transpose convolution output for fusion, thereby detecting easily overlooked damage detail features; the encoder network generates a feature map according to the input bridge appearance image, then the feature map is input into the decoder network to generate a dense prediction map, and then the dense prediction map is processed by a Softmax layer to obtain the class probability of each pixel, completing the damage pixel-level detection; and train the improved encoder-decoder FCN model with the bridge appearance image of each damage category as the training set to obtain the pixel-level detection model corresponding to each damage category;

[0008] Step four: input the bridge appearance image to be predicted into the part recognition model to identify the bridge part corresponding to the image, then input the bridge appearance image to be predicted into the member recognition model corresponding to the bridge part to identify the member corresponding to the image; finally, input the bridge appearance image to be predicted into the damage recognition model corresponding to the member to output the damage category of the bridge appearance image to be predicted; input the bridge appearance image to be predicted into the pixel-level detection model corresponding to the damage category to output the damage quantity, shape contour and inclination direction of the bridge appearance image to be predicted.

[0009] Step five: build a center projection model based on binocular stereo vision, and use a binocular stereo vision model with intersecting optical axes as a damage positioning model; extract and search for matching point pairs between the bridge appearance image pairs to be measured by SIFT algorithm and nearest neighbor algorithm, then select three point pairs from the matching point pairs by a random algorithm and input them into the damage positioning model to obtain a spatial damage plane; the spatial damage plane and the corresponding pinhole camera model form a center projection model, then perform three-dimensional reconstruction on the damage shape contour output by the center projection model corresponding to the bridge appearance image to be measured in step four to complete damage quantitative measurement, and output the damage size of the bridge appearance image pair to be measured.

[0010] Further, when constructing the training data set based on the bridge appearance image, first, the obtained small amount of data is manually classified and calibrated to form a small training data set, then the improved VGG16 model is quickly trained using the small training data set, and the obtained three-level pre-classification model is used to predict the bridge appearance image which has not been classified, and according to the classification result, manual checking is completed, and the image checked by manual checking is supplemented into the small training data set, and the semi-automatic expansion of the original data set is completed.

[0011] Further, the pixel-level detection model has five types, which are respectively a pitted surface detection model, a broken hole detection model, a crack detection model, a exposed steel bar detection model and a concrete spalling detection model.

[0012] Further, during the training of the improved VGG16 model, the convolution process uses the ReLU function as the activation function, and the weights of the convolution layers close to the input end of the model are kept unchanged, that is, these convolution layers are frozen to retain part of the learned features and prevent parameter overfitting; the weights of the convolution layers close to the output end are constantly updated, that is, these convolution layers are unfrozen, so that the model has stronger learning ability and constantly learns new features.

[0013] Further, the improved VGG16 model has five convolution blocks, three of which close to the input end of the model are frozen, and two of which close to the output end of the model are unfrozen; max-pooling layers are used to connect and separate between the convolution blocks.

[0014] Further, after semi-automatic expansion of the original data set, data augmentation is performed on the existing bridge appearance image, and the augmented image is used to construct the training data set.

[0015] A bridge appearance damage binocular stereo vision intelligent detection system fusing a deep learning algorithm, which is used to realize a bridge appearance damage binocular vision intelligent detection method, and the system comprises:

[0016] A bridge image acquisition module, which acquires bridge appearance images by a binocular shooting system and uploads the images in real time;

[0017] A multi-level pre-classification module, which comprises a part recognition model, a component recognition model and a damage recognition model, and is used to recognize the shooting part, the component where the damage is located and the damage category of the bridge appearance image step by step;

[0018] A damage pixel-level detection module, which is used to detect the number, shape contour and inclination direction of a specific category of damage;

[0019] A classifier is configured to classify and correlate the bridge appearance images according to the output results of the previous stage model, call the corresponding next stage model, and complete the transmission of the bridge appearance images between the stages of the model during the training and verification process of the part recognition model, the component recognition model, the damage recognition model and the pixel-level detection model.

[0020] A damage measurement module is composed of a binocular vision-based damage positioning model and a central projection model, and is configured to reconstruct the damage morphology contour and quantitatively measure the damage size.

[0021] A database is configured to store the bridge appearance information feature images.

[0022] Further, the system further comprises a data enhancer configured to perform transformation and enhancement operation on the bridge appearance images to expand the number of bridge appearance images.

[0023] The beneficial effects of the present application are as follows:

[0024] (1) The present application constructs a multi-level pre-classification model based on the multi-level classification idea, comprehensively considers the damage features on different components of different bridge parts, avoids the mutual interference of the same damage in different positions, and effectively improves the damage recognition effect.

[0025] (2) The binocular stereo vision technology is applied to quantitatively measure the damage size, and the influence of the damage severity on the specific component of the specific bridge part is further considered, so that the condition rating of the specific bridge component can be determined according to the corresponding bridge inspection specification. The method of the present application first locates the bridge concrete surface by using binocular stereo vision, then projects the damage morphology contour on it by the central projection method, completes the reconstruction of the damage morphology contour, avoids the matching problem of the damage pixel points between the two views relative to the existing dense matching-based binocular stereo vision three-dimensional reconstruction method, effectively reduces the calculation cost, and improves the reconstruction accuracy.

[0026] (3) The intelligent detection of the bridge appearance damage is realized, and high precision and high efficiency are taken into account at the same time, comprehensive appearance damage information such as position, category and size is output, the digital twin of the bridge damage information is realized, and objective basis is provided for the service bridge appearance state evaluation.

[0027] (4) The expansion training data mechanism is provided, the parameters and performance of the multi-level pre-classification model and the damage detection model can be continuously updated and improved in the later application process, and the model precision and generalization ability are improved. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The flow chart of the bridge appearance damage binocular stereo vision intelligent detection method of the present application is shown in the figure.

[0029] Figure 2A total framework diagram of a multi-level pre-classification model based on VGG16;

[0030] Figure 3 An encoder-decoder FCN model network structure diagram for improvement;

[0031] Figure 4 A precision comparison diagram under different Epochs;

[0032] Figure 5 A center projection model diagram based on binocular stereo vision;

[0033] Figure 6 A pixel-level detection and measurement result example diagram of common crack damage. DETAILED DESCRIPTION

[0034] The purposes and effects of the present application will become more apparent from the following detailed description of the preferred embodiments of the present application, which are described in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application.

[0035] As shown in Figure 1 The bridge appearance damage binocular vision intelligent detection method based on fusion of deep learning, according to the present application, comprises the following steps:

[0036] S1: Construct a bridge appearance information detection dataset;

[0037] For each bridge image collected by the unmanned aerial vehicle equipped with a binocular camera system on site, a small preliminary dataset is obtained after artificial classification according to damage features. The improved VGG16 model is quickly trained using this dataset, and after adjusting the hyperparameters and network architecture, a better model based on this small dataset is obtained. The initial model is used to train the unclassified bridge appearance images, and after artificial verification of the classification results, the new data is supplemented into the existing dataset to obtain a semi-automatic expanded original dataset.

[0038] Bridge damage includes, but is not limited to, pitted surface, pothole, exposed reinforcement, crack, spalling, etc. The encoder-decoder FCN damage segmentation dataset completed by calibration also contains non-damage images. The parts, components of the bridge and the damage directory corresponding to each component are shown in Table 1. The bridge appearance damage detection dataset constructed in step S1 is randomly divided into 5:1, the former is used as the training set to train the model, and the latter is used as the test set to evaluate the model quality. Finally, the training dataset of each bridge part, the training dataset of each bridge component of each bridge part, and the training dataset of each damage category of each bridge component of each bridge part are obtained.

[0039] Table 1 Classification directory of bridge appearance information

[0040]

[0041]

[0042] More importantly, the application performs data augmentation on the training set, including angle rotation, translation, skew transformation, scaling, and horizontal flip operation on the image, to reduce the overfitting level of the recognition result.

[0043] S2: constructing a three-level pre-classification model based on the improved VGG16 model;

[0044] As shown in Figure 2 , the original VGG16 model is improved, and a convolutional neural network model is constructed using the "Tensorflow+Python" system framework. The model is composed of five convolutional blocks and three fully connected layers, and the specific convolutional layers are separated by pooling layers. Table 2 lists the structure of each layer of the convolutional neural network.

[0045] Table 2 VGG neural network structure

[0046]

[0047]

[0048] Note: n represents the number of classes of the classification network.

[0049] VGG has five blocks, each of which is composed of 2-3 convolutional layers and a pooling layer. The size of the convolution kernel used in the convolutional layer is 3x3. The convolutional layer neuron convolution calculation formula is:

[0050]

[0051] In the formula, X is a two-dimensional vector with an area of (M, N), ω ij is the convolution kernel, b is the output feature bias term, and σ is the activation function, which uses the ReLU function in this model.

[0052] The multi-level pre-classification model has three levels. The first level is a part recognition model of the bridge part, the second level is a component recognition model of the bridge component, and the third level is a damage recognition model of each damage category. To improve the efficiency of the model, the first level classification network is first trained and optimized, and the second and third level networks are referred to the first level. This is because the total classification number increases from the first level to the third level, and the classification difficulty is progressive. Moreover, the classification objects of the second and third levels are also the bridge surface information, and the overall classification task is similar. Under the same other experimental conditions, after using model fine-tuning, the model performance is optimized to a certain extent.

[0053] To improve the accuracy of the model, two unfreezing strategies of the top convolutional layers and pooling layers in the pre-trained VGG are studied. Compared with the original VGG result, when the convolutional block 5 is unfrozen, the validation accuracy of the model is significantly improved from 49% to 69.4%, and when the convolutional block 4 and the convolutional block 5 are unfrozen, the validation accuracy reaches 95.5%.

[0054] When the number of neurons remains the original setting, unfreezing the convolutional block 4 and 5 is higher than unfreezing only the block 5 by about 0.25 in training and validation accuracy. When the number of neurons of the two dense layers is changed, the model performance will also be slightly different; when the number of neurons of the dense layer is 1024+256, the validation set accuracy is the highest, reaching 96%. The results show that the improved VGG compared with the original VGG in the task of bridge surface defect classification, the validation accuracy is improved by about 45%, and the detailed data is shown in Table 3.

[0055] Therefore, the number of neurons of the first two dense layers of the model is modified from 4096 to 1024 and 256, and the training speed of the model is accelerated.

[0056] Table 3 Comparison of original and improved VGG performance

[0057]

[0058]

[0059] To improve the training accuracy of the model, some of the bottom convolutional layers are frozen to retain some learned features and prevent overfitting due to too many parameters; at the same time, the head convolutional layers are unfrozen to make the model have stronger learning ability and constantly learn new features.

[0060] Epoch represents the number of training rounds of the data set. According to the size of the data set, the performance of the GPU and the model, the optimization direction is determined. Properly increasing the Epoch can improve the accuracy and make the model training process more stable; but too large Epoch may lead to overfitting or invalid training of the model that has reached the optimal. Analysis of the training results shows that since the number of first-level classifications is only three, the overall convergence is fast, and an Epoch of 100 can obtain a higher accuracy. The number of second-level component classification models is larger, which is 11 categories (bridge deck system), 7 categories (lower structure) and 5 categories (upper structure). When Epoch = 100, the model oscillates greatly, the accuracy is about 0.8 and is not stable, indicating that the network is still underfitting. When the bridge deck system and the upper structure Epoch = 150, and the lower structure Epoch = 200, the learning curve is more stable, and the accuracy after stabilization is improved by about 10%. Table 4 shows the hyperparameter settings of the improved VGG16 model, Figure 4 The accuracy comparison chart under different Epoch.

[0061] Table 4 VGG model hyperparameter settings

[0062] Hyperparameters Values Epoch 30-200 Initial learning rate 1 x 10 -4 ~ 4 x 10 -6 ]]> Decay rate 0.8 / 0.9 Steps per epoch 5 (epoch) Batch size 128 / 256

[0063] The bridge apparent information image and its corresponding three-level information are used as input data and output data of the model. The number of iterations for training is set to 100-200 times, the training result parameters are saved, and the results are output in the following format for different recognition results: "part-member-damage category".

[0064] The effectiveness of the convolutional neural network model is verified by selecting different categories of bridge apparent images. The model recognition result is compared with the actual classification. The recognition accuracy needs to reach more than 80%, otherwise the parameters and hyperparameters of the model are adjusted until the error meets the requirements.

[0065] After the model training, the bridge apparent information image to be detected is input into the updated bridge apparent damage three-level pre-classification model to determine the final damage category.

[0066] S3: Based on the improved encoder-decoder FCN model, a pixel-level detection model corresponding to each damage category is constructed.

[0067] This embodiment considers three aspects: common, has a great influence on the service state of the bridge and has a great weight in the evaluation of the service state of the bridge. Five damages of exposed reinforcement, crack, pit, spalling and honeycomb pitting are selected as research objects that need to obtain more in-depth detection information. In the aforementioned multi-level pre-classification work, damage classification information has been obtained, and pictures containing the five common damages are extracted and distinguished from a large number of original input pictures as the basis for work.

[0068] The traditional VGG19 design only uses the top feature map for prediction, resulting in the neglect of shallow feature maps containing more detailed information. Such design is not conducive to small target detection. In view of the wide application of encoder-decoder framework CNN in semantic segmentation, a pre-trained VGG19 classification network is used as an encoder, and a new encoder-decoder FCN is built with a transpose convolution layer for damage pixel-level detection. Figure 3The encoder-decoder FCN structure diagram is shown. Among them, the encoder network removes the softmax layer at the end of VGG19, replaces the three fully connected layers with convolutional layers, and the corresponding output is also changed from the number of categories to the feature map related to the damage. The feature map is expanded to the original input size by 5 times of transposed convolution upsampling, and the jump connection structure is used to fuse the results of the first 4 times of transposed convolution upsampling with the corresponding size of the encoder maximum pooling output, so that the network can adapt to the information required for identifying small targets such as cracks and exposed reinforcement. Since pre-classification can determine the types of damage contained in the image, the model only needs to perform a pixel-level detection task on the five common types of damage. Table 5 lists the structure of each layer of the encoder-decoder FCN.

[0069] Table 5 Encoder-decoder FCN structure

[0070]

[0071]

[0072] A parameter transfer learning strategy is adopted, and the convolutional layer parameters in the encoder part of the FCN are initialized by the pre-trained VGG19 weights to save training time and optimize learning efficiency. In addition, the filter weights of the deconvolution layer in the decoder part are initialized with a truncated normal distribution with a mean of 0 and a standard deviation of 0.01, and the initial value of its bias is set to a constant zero vector. The bridge appearance information image and the corresponding pixel-level label are used as training data to train each FCN model for 20 epochs, and the batch size is set to 2 (considering the GPU memory constraint).

[0073] According to the damage recognition result in the multi-level pre-classification, the bridge appearance image to be detected is input into the corresponding damage pixel-level detection model, and the damage number, shape contour and inclination direction are returned. Among them, the damage shape contour is the image point pixel coordinate set {(u i ,v i )i=…}.

[0074] The encoder-decoder structure increases the model depth while also increasing the computational cost, but the overall impact on the damage detection speed is not great. The input data of the encoder-decoder FCN in the present application is five kinds of damage obtained by VGG classification, which reduces the input of other damage and non-damage images. On the one hand, the workload of the encoder-decoder FCN is reduced from the pre-screening of the input, improving the efficiency of pixel detection, and on the other hand, the interference of other damage is reduced.

[0075] In order to measure the prediction bias, cross-entropy is used as the loss function to evaluate the similarity between the target value (i.e. the labeled value) and the predicted logarithmic value, as shown in the following formula:

[0076]

[0077] where J(0) is the loss function, a i represents the target value of the i-th pixel, p i represents the predicted value of the i-th pixel, and N is the total number of pixels in the image.

[0078] The training optimizer uses Adam, which combines the advantages of two optimization algorithms, AdaGrad and RMSProp, and its formula is:

[0079]

[0080] m t = β1m t-1 + (1 - β1)g t ,

[0081]

[0082]

[0083] where β1 and β2 represent the exponential decay rate of the first and second moment estimates, respectively set to 0.9 and 0.999; t is the iteration index; m t and v t represent the exponential moving average of the first and second moments of the gradient, and represent the bias correction for the first and second order momentum, respectively; θ t-1 and θ t represent the weights before and after iteration; ε = 10 -8 is a small default number to maintain the stability of the numerical value during optimization; η represents the learning rate.

[0084] Considering that the learning rate is a key hyperparameter that determines whether the training process converges and the convergence speed, in order to determine a suitable initial value, three groups of experiments were conducted with 1e-3, 1e-4 and 1e-5, respectively. In each training process, an exponential step decay function was used to anneal the learning rate at each training step to quickly reach a faster local optimum. The corresponding formula is as follows:

[0085]

[0086] where η0 is the initial learning rate, r d is the decay rate, and t is the iteration step; is the floor operation, which returns the maximum integer less than or equal to the input value.

[0087] In the art, the model is usually evaluated by three indexes of IOU (Intersection over Union), precision, recall. IOU represents the ratio of the overlapping area of the real value region and the predicted region to the union area. Its formula can be expressed as:

[0088]

[0089] The precision rate refers to the proportion of positive samples contained in all predicted positive samples, and its formula is:

[0090]

[0091] The recall rate refers to the proportion of correctly identified positive samples in all positive samples, and its formula is:

[0092]

[0093] The F1 score is also an index for measuring the accuracy, which takes into account both precision and recall, and its formula is:

[0094]

[0095] Where, the detection of non-A damage but identified as A is considered as false positive (FP); A damage but not correctly identified is false negative (FN). The detection of correctly identified sample is considered as true positive (TP). Since the semantic segmentation task is more demanding than the classification task, the correct identification criterion is more stringent. Only when the detection region and the real region defect type match, and the IoU of the two regions is not less than 0.7, it is considered as correct identification.

[0096] After obtaining the precision and recall values, the P-R curve is drawn with the two as the vertical and horizontal coordinates, and the area surrounded by the curve and the coordinate axis is the AP (average precision) value of a certain type of disease. The average of the AP values of all categories is the mAP value, and the formula can be expressed as:

[0097]

[0098] Where, n is the total number of categories, and AP i represents the AP value of the i-th category. The mAP is used as an index to judge the overall recognition accuracy of the model. The correct recognition rate needs to reach more than 80%, otherwise the parameters and hyperparameters of the model are adjusted, or the network layer is fine-tuned, and part of the convolution layer is frozen and unfrozen, until the error meets the requirements.

[0099] S4: input the bridge apparent image to be predicted into the part recognition model, recognize the bridge part corresponding to the image, then input the bridge apparent image to be predicted into the component recognition model corresponding to the bridge part, recognize the component corresponding to the image; finally, input the bridge apparent image to be predicted into the damage recognition model corresponding to the component, output the damage category of the bridge apparent image to be predicted; input the bridge apparent image to be predicted into the pixel-level detection model corresponding to the damage category, and the model outputs the damage quantity, shape contour and inclination direction of the bridge apparent image to be predicted.

[0100] S5: reconstruct the damage shape contour output in step S4 based on the center projection model of binocular stereo vision, and quantitatively measure the damage size;

[0101] Taking the bridge apparent image pair and the damage shape contour output in step S4 as objects, the camera corresponding to the center projection model is named as the main camera (C1), and the other is the positioning camera (C2) for convenience of the following description. First, the spatial position of the damage is determined by the damage positioning model based on binocular stereo vision. As shown in the figure, the specific process of damage positioning is as follows: Figure 5

[0102] SIFT feature extraction algorithm is applied to extract feature points of two images in the stereo image pair I = {I1, I2}, which are F1 = {(pi, fi) | i = 1…M} and F2 = {(pj, fj) | j = 1…N} respectively, wherein fi and fj are local feature descriptors of the i-th feature point position pi of the image I1 and the j-th feature point position pj of the image I2 respectively. 1,i 1,i 2,j 2,j 1,i 2,j 1,i 2,j

[0103] On the basis of accurately determining the feature points, the nearest neighbor search algorithm is used to search and calculate the shortest Euclidean distance of the descriptors of the reference feature points (on the image I1) and the target feature points (on the image I2), and the corresponding two feature points are taken as a pair of matched points. The matching result is a set of feature point pairs {(pi, p2) | pi e I1, p2 e I2}.

[0104] Random algorithm is applied to select three non-collinear feature points p1, p2 and p3 on the structural surface of the reference image, together with the homologous points p1', p2' and p3' of the three points on the other image to form a three-point pair. The three non-collinear space points P1, P2 and P3 corresponding to the three-point pair are calculated through the binocular stereo vision model, and the determination of the spatial position of the crack is completed.

[0105] ​​​​​​​​​Taking point P1 in the computational space as an example, the main camera coordinates of point P1 in the world coordinate system are (X... p1 ,Y p1 Z p1 ), and its image points The transformation relationship can be expressed as follows:

[0106]

[0107] Where A1 is the intrinsic parameter matrix of the master camera, and its parameters are... It is the principal point C1. pixel coordinates, f l It is the focal length of the main camera, k. l and l l Representing coordinate systems in u l axis and v l The physical length of each pixel along the axial direction, where γ1 represents u. l axis and v l The tilt coefficient of the axis is generally 0; I3 is a 3×3 identity matrix, O 3×1 This represents a 3×1 zero vector.

[0108] At the same time, P1(X) p1 ,Y p1 Z p1 It is also connected through a homography moment H. The transformation relationship is established, and the corresponding formula is as follows:

[0109]

[0110] Where A2 is the intrinsic parameter matrix of the positioning camera, and R and t are its rotation matrix and translation vector relative to the main camera, respectively. -1 Let R be the inverse matrix of the rotation matrix. From the above two equations, we get:

[0111]

[0112] The binocular imaging device was calibrated to obtain the internal parameters and relative pose of the two cameras. The calibrated focal length f of the main camera was then used. l The distance from the optical center of the pinhole model to the imaging plane is taken as the distance, and u on the imaging plane is considered. l axis and v l The tilt factor of the axis, based on the principal point of calibration. The location is determined by converting image pixel coordinates to physical coordinates, rather than simply fixing it at the image center, in order to more closely approximate the imaging process and reduce reconstruction errors. If point pairs p1 and p1′, p2 and p2′, and p3 and p3′ are determined, their spatial points P1, P2, and P3 in the C1 measurement coordinate system can be calculated respectively, thus completing the localization of the plane where the damage is located.

[0113] Then the central projection model is used to reconstruct the damage shape profile. Unlike the traditional three-dimensional reconstruction method based on dense matching results to generate dense point clouds, the present application simulates the C1 camera imaging process with a pinhole camera model, and takes the C1 camera optical center as the projection origin, and the space plane determined by P1, P2 and P3 as the image bearing surface, and projects the image damage shape profile output in step S4 onto the image bearing surface to reconstruct the real damage shape profile. The central projection reconstruction of damage information effectively avoids the pairing problem of the pixel points of the bridge apparent image pair damage shape profile, reduces the calculation cost, and improves the reconstruction accuracy.

[0114] Before that, the image damage shape profile pixel points and the image bearing surface are unified in the same reference system, that is, the pixel coordinates (u i ,v i ) are converted into C1 coordinates (x i ,y i ,z i ). The transformation relationship between the camera image coordinate system and the camera coordinate system is as follows:

[0115]

[0116] After coordinate conversion, central projection calculation is performed. The intersection of the straight line from the optical center of the central projection model to the image points of the damage shape profile after coordinate conversion and the image bearing surface is calculated, and the reconstruction of the real damage shape profile {(X i ,Y i ,Z i )|i=1…} is completed.

[0117] Finally, based on the reconstructed space points of the damage shape profile, the damage shape feature size is quantitatively measured. Taking a crack as an example, the quantitative measurement of distance type size, given the real shape profile of the crack, such as the crack skeleton and the edge, the crack width can be calculated by the distance between the two edge points closest to the skeleton points, and the length of the crack shape can be calculated by accumulating the Euclidean distance of adjacent skeleton points, to complete the quantitative measurement of crack size. Taking a hole as an example, the quantitative measurement of area type size, according to the real shape profile of the hole, the centroid of the closed contour is calculated, and a triangle is formed by the centroid and the adjacent two reconstructed points on the contour, and the actual area of the hole is approximated by accumulating the areas of these triangles.

[0118] Finally, S6: combining the multi-level pre-classification results and pixel-level detection results output in step S4 with the damage size obtained in step S5, and finally outputting comprehensive apparent damage detection information including position, category and size.

[0119] The bridge apparent damage binocular stereo vision intelligent detection system fusing the deep learning algorithm is used for realizing a bridge apparent image damage intelligent detection method.

[0120] The bridge image acquisition module acquires the bridge apparent image by the binocular shooting system and uploads the bridge apparent image in real time.

[0121] The multi-level pre-classification module comprises a part recognition model, a component recognition model and a disease recognition model and is used for recognizing the shooting part, the damage component and the damage category of the bridge apparent image step by step.

[0122] The damage pixel-level detection module is used for detecting the quantity, the shape contour and the inclination direction of the damage of a specific category.

[0123] The classifier is used for classifying and associating the bridge apparent image according to the output result of the previous model, calling the corresponding next model and completing the transmission of the bridge apparent image between the models in the training and verification process of the part recognition model, the component recognition model, the damage recognition model and the damage pixel-level detection model.

[0124] The damage measurement module is composed of a damage positioning model based on binocular vision and a central projection model and is used for damage shape contour reconstruction and damage size quantitative measurement.

[0125] The database is used for storing the bridge apparent information feature image.

[0126] The data enhancer is used for performing transformation enhancement operation on the bridge apparent image and expanding the quantity of the bridge apparent image.

[0127] Embodiment

[0128] In this embodiment, 7200 bridge apparent images containing non-damage classification collected by a drone carrying a binocular shooting system are selected, and 1200 images are randomly extracted to obtain a small preliminary data set by manual classification. The VGG16 model is quickly trained by using the obtained small preliminary data set, the structure of the VGG16 model is optimized after adjusting the neuron number and the hyperparameter of the VGG16 model, and a better model based on the small data set is obtained.

[0129] The remaining 6000 unclassified images are trained by using the optimized VGG16 model, the new data is expanded to the existing data set after the result is manually checked, and the original data set is established. The shooting part, the component type and the damage category information of each bridge apparent image in the data set can be obtained by using the original data set, and finally, the multiple classified data files are constructed into the bridge apparent information detection initial data set.

[0130] The completed bridge appearance information detection initial data set is randomly divided into training set and validation set in proportion, obtaining a training set containing 6000 images and a validation set containing 1200 images. The improved VGG16 model is trained using the training set, and the model performance is evaluated through the validation set. After training, the multi-level recognition accuracy of each model can reach more than 90%, and the overfitting level is low, so that the trained multi-level pre-classification model can be obtained. Using the multi-level pre-classification model, 14842 bridge appearance images containing common damage categories are processed, obtaining 3352 exposed tendon images, 5916 crack images, 1558 pit images, 2740 spalling images and 1276 honeycomb surface images. Each image maintains the original size, i.e. 2048x1536 dpi, without compression. Then these images are pixel-level labeled to construct the damage pixel-level detection database of exposed tendon, crack, pit, spalling and honeycomb surface, etc.

[0131] The improved encoder-decoder FCN model is trained on each type of damage pixel-level detection database, with 500 iterations, and the training results are saved to obtain each damage pixel-level detection model of the bridge appearance image. Taking the crack pixel-level detection model as an example, the training results are shown in Table 3. The FCN model with an initial learning rate of 1e-4 achieves the highest precision, recall and F1 score, which are 83.10%, 85.74% and 84.14% respectively. The FCN model is used as the default crack damage pixel-level detection model to extract the crack number information, shape contour and inclination direction from the image.

[0132] Table 6 Training results of crack pixel-level detection model

[0133] Learning rate (x 10 -4 )]]> Precision (%) Recall (%) F1 -score (%) 0.1 80.48 80.67 80.47 1 83.10 85.74 84.14 10 79.53 79.84 78.43

[0134] The remaining bridge appearance images are used to verify the effectiveness of the model. The effective detection of damage needs to reach more than 80%, otherwise the parameters and hyperparameters of the model are adjusted until the accuracy meets the requirements. In this example, the values of the set hyperparameters are shown in Table 5. After training, the average damage detection accuracy of each training set reaches 89%, and the average defect recognition accuracy of each validation set reaches 81%, which meets the requirements.

[0135] Table 7 Hyperparameter settings of the example model

[0136] Hyperparameters Values Learning rate 1 x 10 -4 ]] Epoch 50 Batch 2 Steps per Epoch 500 Validation steps 25

[0137] In the damage pixel-level detection task, the damage detection results predicted by the model are displayed and compared with the corresponding pixel-level labeled images. When there are two or more damage types in the bridge appearance image, the segmented areas with different colors and depths are used to represent them. Taking cracks as an example, the pixel-level detection results and corresponding size results of different crack morphologies are as follows: Figure 6The number, shape, and orientation of the cracks are further obtained by judging whether a single pixel belongs to a crack or not. The model can perform well for irregular shapes and even for large area damages (e.g. pitting) and small size damages (e.g. cracks, exposed reinforcement) in the target. Figure 6 The model can also perform well for damages with complex shapes or complex backgrounds, such as cross cracks and complex background cracks.

[0138] According to the bridge inspection specification, the maximum width limit of the structural cracks of the main girder is 0.2 mm, and the limit of the concrete cracks of the main pier is 0.3 mm. In order to meet the measurement requirements, the size of the crack of the silk mill (0.1 mm) needs to be quantified. In this example, the error of the crack measurement result can be stably controlled within 0.1 mm, and the maximum error is 0.093 mm, which can meet the measurement accuracy requirements.

[0139] Those skilled in the art can understand that the above description is only preferred examples of the application and is not used to limit the application, although the application is described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions recorded in the foregoing examples or make equivalent replacements for part of the technical features. Any modification, equivalent replacement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A binocular visual intelligent detection method for apparent bridge damage integrating deep learning, characterized in that, The method includes: Step 1: Acquire visual images of the bridge using a binocular imaging system, and label each image with the location of the bridge being photographed, the specific bridge component where the damage occurred, and the damage category. Then, categorize the images according to the bridge location, constructing a training dataset for each bridge location. Next, further categorize the images of each bridge location according to the specific bridge component where the damage occurred, constructing a training dataset for each bridge component within each bridge location. Finally, categorize the images of each bridge component according to the damage category, constructing a training dataset for each bridge component within each bridge location, and for each damage category. Step 2: Construct an improved VGG16 model by modifying the number of neurons in the three fully connected layers of the VGG16 model to 1024 and 256, respectively, and adjusting the number of recognition categories. Then, train the improved VGG16 model using the training dataset for each bridge part to obtain a part recognition model for each bridge part. Next, train the part recognition model for each bridge part using the training dataset for each bridge component to obtain a component recognition model for each component. Finally, train the component recognition model for each component using the training dataset for each damage category of each bridge component in each bridge part to obtain a damage recognition model for each damage category of each bridge component. This results in a three-level pre-classification model. Step 3: Construct an improved encoder-decoder FCN model. Specifically, use a VGG19-based encoder network as the feature extraction network. The encoder's max-pooling index is superimposed onto the corresponding transposed convolutional output of the decoder at the same resolution for fusion, thereby detecting easily overlooked damage details. The encoder network generates a feature map based on the input bridge appearance image. This feature map is then input into the decoder network to generate a dense prediction map. The dense prediction map is then processed by a Softmax layer to obtain the class probability of each pixel, completing pixel-level damage detection. The improved encoder-decoder FCN model is trained using bridge appearance images for each damage category as the training set, resulting in a pixel-level detection model for each damage category. Step 4: Input the bridge appearance image to be predicted into the part recognition model to identify the bridge part corresponding to the image. Then, input the bridge appearance image to be predicted into the component recognition model corresponding to the bridge part to identify the component corresponding to the image. Finally, input the bridge appearance image to be predicted into the damage recognition model corresponding to the component to output the damage category of the bridge appearance image to be predicted. Input the bridge appearance image to be predicted into the pixel-level detection model corresponding to the damage category to output the number of damages, shape contours, and tilt direction of the bridge appearance image to be predicted. Step 5: Construct a central projection model based on binocular stereo vision, using a binocular stereo vision model with intersecting optical axes as the damage localization model; extract and search for matching point pairs between the bridge appearance image pairs to be measured using the SIFT algorithm and the nearest neighbor algorithm, and then randomly select three point pairs from the matching point pairs and input them into the damage localization model to obtain a spatial damage plane; the spatial damage plane and the corresponding pinhole camera model form a central projection model, and then perform three-dimensional reconstruction on the damage morphology contour of the bridge appearance image to be measured corresponding to the central projection model, as output in Step 4, to complete the quantitative damage measurement and output the damage size of the bridge appearance image pair to be measured.

2. The binocular visual intelligent detection method for bridge apparent damage fused with deep learning according to claim 1, characterized in that, When constructing a training dataset based on bridge appearance images, the first step is to manually classify and label the small amount of data obtained to form a small training dataset. Then, the improved VGG16 model is quickly trained using the small training dataset, and the obtained three-level pre-classification model is used to predict the bridge appearance images that have not yet been classified. The classification results are manually verified, and the manually verified images are added to the small training dataset to complete the semi-automatic expansion of the original dataset.

3. The binocular visual intelligent detection method for bridge apparent damage fused with deep learning according to claim 1, characterized in that, There are five pixel-level detection models: surface roughness detection model, pothole detection model, crack detection model, exposed rebar detection model, and concrete spalling detection model.

4. The binocular visual intelligent detection method for bridge apparent damage fused with deep learning according to claim 1, characterized in that, During the training of the improved VGG16 model, the ReLU function is used as the activation function in the convolution process, and the weights of some convolutional layers near the model input are kept unchanged, that is, these convolutional layers are frozen to retain some learned features and prevent parameter overfitting; the weights of some convolutional layers near the output are continuously updated, that is, these convolutional layers are unfrozen, so that the model has a stronger learning ability and can continuously learn new features.

5. The binocular visual intelligent detection method for bridge apparent damage fused with deep learning according to claim 4, characterized in that, The improved VGG16 model has a total of 5 convolutional blocks. The three convolutional blocks closest to the model input are frozen, and the two convolutional blocks closest to the model output are unfrozen. The convolutional blocks are connected and separated by a max-pooling layer.

6. The binocular visual intelligent detection method for bridge apparent damage fused with deep learning according to claim 2, characterized in that, After semi-automatic augmentation of the original dataset, data augmentation is performed on the existing bridge appearance images, and the augmented images are used to construct the training dataset.

7. A binocular stereo vision intelligent detection system for bridge apparent damage integrating deep learning algorithms, characterized in that, This system is used to implement the binocular visual intelligent detection method for bridge apparent damage according to any one of claims 1 to 6. The system includes: The bridge image acquisition module uses a binocular camera system to capture images of the bridge's appearance and upload them in real time. The multi-level pre-classification module includes a location recognition model, a component recognition model, and a damage recognition model, which are used to identify the location where the bridge appearance image was captured, the component where the damage occurred, and the damage category step by step. The damage pixel-level detection module is used to detect the number, shape, and tilt direction of damage of a specific type. The classifier is used to classify and associate bridge appearance images based on the output of the previous level model during the training and validation of the part recognition model, component recognition model, damage recognition model, and damage pixel-level detection model. It then calls the corresponding next level model to complete the transfer of bridge appearance images between the various levels of models. The damage measurement module consists of a binocular vision-based damage localization model and a central projection model, which are used for damage morphology contour reconstruction and quantitative measurement of damage size. A database used to store characteristic images of bridge appearance.

8. The bridge apparent damage binocular stereo vision intelligent detection system integrating deep learning algorithms according to claim 7, characterized in that, The system also includes a data augmenter for transforming and enhancing bridge appearance images, thus expanding the number of bridge appearance images.

Images