Three-level recognition method and recognition system of bridge surface information based on deep learning

Abstract

The present invention discloses a three-level recognition method and recognition system for bridge appearance information based on deep learning. The method first classifies a training dataset into different levels to obtain a training dataset for each bridge part, a training dataset for each bridge component, and a training dataset for each disease category. The method then uses the aforementioned training datasets to train an improved VGG16 model, obtaining a part recognition model for each bridge part, a component recognition model for each component, and a disease recognition model for each disease category, forming a three-level recognition model. Finally, the bridge appearance image to be predicted is input into the part recognition model to identify the bridge part corresponding to the image. The component recognition model corresponding to the corresponding bridge part is then input into the disease recognition model to identify the component corresponding to the image. Finally, the disease recognition model corresponding to the corresponding component is input into the disease recognition model to output the disease category of the image to be predicted containing the bridge appearance information. The present invention can provide comprehensive and accurate bridge service status information, including component location and disease type.

Description

Technical Field

[0001] The present invention relates to the field of bridge structure surface information detection, and in particular to a three-level recognition method and recognition system for bridge surface information based on deep learning. Background Art

[0002] In recent years, many scholars at home and abroad have conducted research on new methods for non-destructive testing of bridge defects. Convolutional neural networks have been widely used in the detection of crack defects in bridges, roads, and tunnels. However, the existing bridge inspection system has the following problems in the application of surface information recognition: (1) The existing identification methods involve a single type of defect, most of which focus on typical defects such as cracks or specific locations such as bridge piers, and cannot identify multiple defects in different locations; (2) In the evaluation of bridge service status, the same defect on different components corresponds to different impact weights. The existing methods do not comprehensively identify and determine the location information of the defect. The results still need to be manually identified before the location of the defect can be scored according to the bridge inspection specifications; (3) The surface characteristics of the same defect on different components are different. Direct single-level recognition causes interference between images, resulting in low recognition accuracy; (4) The update and improvement of the model during the application process is not considered, and the application scope is limited. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention proposes a three-level recognition method and recognition system for bridge surface information based on deep learning. The specific technical solutions are as follows:

[0004] A three-level identification method for bridge surface information based on deep learning, the method specifically includes:

[0005] S1: Obtain a large number of bridge surface images and annotate each image with the bridge location, the specific bridge component where the defect is located, and the defect category. Classify the images according to the bridge location to construct a training dataset for each bridge location. Then, further classify the images of each bridge location according to the specific bridge component where the defect is located to construct a training dataset for each bridge component in each bridge location. Finally, classify the images of each bridge component according to the defect category to construct a training dataset for each defect category for each bridge component in each bridge location.

[0006] S2: Construct an improved VGG16 model, that is, modify the number of neurons in the three fully connected layers of the VGG16 model to 1024, 512, and the number of recognition categories, and then train the improved VGG16 model with the training dataset of each bridge part respectively to obtain a part recognition model corresponding to each bridge part; then use the training dataset of each bridge component to train the part recognition model of the corresponding bridge part to obtain a component recognition model of the corresponding component; finally, use the training dataset of each disease category of each bridge component of each bridge part to train the component recognition model of the corresponding component to obtain a disease recognition model of each disease category of the corresponding bridge component; thus, a three-level recognition model is obtained;

[0007] S3: Input the bridge appearance image to be predicted into the part recognition model to identify the bridge part corresponding to the image, and then input the bridge appearance image to be predicted into the component recognition model corresponding to the bridge part to identify the component corresponding to the image; finally, input the bridge appearance image to be predicted into the defect recognition model corresponding to the component, and output the defect category of the bridge appearance image to be predicted.

[0008] Furthermore, the parts of the bridge include the bridge deck system, the superstructure, and the substructure; each part corresponds to a component identification model;

[0009] The bridge deck system includes the following components: signboards, isolation nets, railings, curbstones, green belts, drains, bridge deck pavement, sidewalks, expansion devices, and bridgehead settlement;

[0010] The superstructure includes the following components: steel box girder, arch bridge, beam bridge, drainage hole, stone bridge, small box girder;

[0011] The substructure includes the following components: ear spar, piers, abutments, supports, and foundations; each component corresponds to a component identification model;

[0012] Each component corresponds to a disease identification model.

[0013] Furthermore, when the improved VGG16 model is trained, the convolution process uses the ReLU function as the activation function, and keeps the weights of some convolutional layers close to the input end of the model unchanged, that is, freezes these convolutional layers, so as to retain some learned features and prevent parameter overfitting; the weights of some convolutional layers close to the output end are kept updated, that is, unfreezes these convolutional layers, so that the model has stronger learning ability and continuously learns new features.

[0014] Furthermore, the improved VGG16 model has a total of 5 convolution blocks, three convolution blocks close to the model input end are frozen, and two convolution blocks close to the model output end are unfrozen; the convolution blocks are separated by max-pooling connections.

[0015] Furthermore, in S1, after obtaining a large number of images containing bridge appearance information, data enhancement is first performed on the images, and then a training data set is constructed using the enhanced images.

[0016] A three-level recognition system for bridge surface information based on deep learning, which is used to implement the recognition method, includes:

[0017] The bridge image upload module is used as an image upload port to collect the bridge surface image;

[0018] A data enhancer is used to perform transformation and enhancement operations on the bridge appearance image to expand the number of bridge appearance images;

[0019] The three-level recognition module includes a location recognition model, a component recognition model, and a defect recognition model, which are used to gradually identify the shooting location of the bridge surface image, the component where the defect is located, and the defect category;

[0020] The classifier is used to classify and associate the bridge surface image according to the recognition results of the previous level model during the training and verification of the part recognition model, component recognition model, and disease recognition model, and call the corresponding next level model for recognition to complete the transmission of the bridge surface image between models at all levels.

[0021] The beneficial effects of the present invention are as follows:

[0022] (1) The present invention constructs a multi-level three-level identification model based on the three-level classification concept, comprehensively considers the influence of the location of the defect and the component type on the defect characteristics, directly corresponds to the relevant evaluation indicators of the bridge inspection specifications, and effectively improves the defect identification effect.

[0023] (2) For the constructed multi-level recognition model, the data set is graded and reorganized layer by layer, the similarity of the input data is improved, the process of extracting data features by the model is greatly optimized, and the mutual interference between disease morphologies in different locations is effectively avoided, so that the model has higher recognition accuracy.

[0024] (3) After the number of hidden layer neurons is modified and the convolution layer is frozen and unfrozen, the convolutional neural network has high model recognition accuracy and fast speed.

[0025] (4) Provide a mechanism for expanding training data. This method and system are highly scalable and can continuously improve and update the bridge surface information classification model in later applications, thereby improving image recognition accuracy and generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of a three-level identification method for bridge surface information based on deep learning provided by the present invention.

[0027] Figure 2 This is a schematic diagram of the overall framework of the part recognition model, component recognition model, disease recognition model and the corresponding improved VGG16 model provided by the present invention.

[0028] Figure 3 Schematic diagram of changing the number of neurons in the fully connected layer of the improved VGG16 model and freezing some convolutional blocks during training.

[0029] Figure 4 This is an example diagram of the model training results for the drainage system.

[0030] Figure 5 Output of classification results for appearance information recognition

[0031] Figure 6 This is a comparison chart of accuracy under different Epochs. DETAILED DESCRIPTION

[0032] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0033] like Figure 1 As shown, the three-level identification method of bridge appearance information based on deep learning of the present invention specifically includes:

[0034] Step 1: Obtain a large number of bridge surface images and annotate each image with the bridge location, the specific bridge component where the defect is located, and the defect category. Classify the images according to the bridge location to construct a training dataset for each bridge location. Then, further classify the images of each bridge location according to the specific bridge component where the defect is located to construct a training dataset for each bridge component in each bridge location. Finally, classify the images of each bridge component according to the defect category to construct a training dataset for each defect category for each bridge component in each bridge location.

[0035] Bridge defect categories include but are not limited to cracks, damage, exposed reinforcement corrosion, honeycombed surfaces, etc., and also include images without defects.

[0036] In this embodiment, the parts of the bridge include the bridge deck system, the superstructure, and the substructure; each part corresponds to a component identification model.

[0037] The bridge deck system includes the following components: signboards, isolation nets, railings, curbs, green belts, drainage outlets, bridge deck pavement, sidewalks, expansion devices, and bridge head settlement.

[0038] The superstructure includes the following components: steel box girder, arch bridge, beam bridge, drainage hole, stone bridge, and small box girder.

[0039] The substructure includes the following components: ear wing beam, piers, abutments, supports, and foundations; each component corresponds to a component identification model.

[0040] Each component corresponds to a disease identification model.

[0041] The bridge locations, corresponding components, and corresponding component damage categories are shown in Table 1.

[0042]

[0043]

[0044] To further expand the training dataset, we can first perform data augmentation on the existing bridge surface images, including rotation, translation, shearing, scaling, and horizontal flipping. Finally, we randomly split the resulting dataset into a 5:1 split, with the former serving as the training set for model training and the latter as the test set for evaluating model quality.

[0045] S2: Construct an improved VGG16 model, that is, modify the number of neurons in the three fully connected layers of the VGG16 model to 1024, 512 and the number of recognition categories, and then use the training data set of each bridge part to train the improved VGG16 model respectively to obtain a part recognition model corresponding to each bridge part; then use the training data set of each bridge component to train the part recognition model corresponding to the bridge part to obtain a component recognition model for the corresponding component; finally, use the training data set of each disease category of each bridge component of each bridge part to train the component recognition model of the corresponding component to obtain a disease recognition model for each disease category of the corresponding bridge component; thus, a three-level recognition model is obtained.

[0046] like Figure 2 As shown in the figure, the improved VGG16 model is composed of 5 convolution blocks and an output layer. Each convolution block consists of 2 to 3 convolution layers superimposed with a pooling layer. The convolution calculation formula of the convolution layer neuron is:

[0047]

[0048] Where X is a two-dimensional vector with the region (M, N), ω ij is the convolution kernel, b is the output feature additional bias term, σ is the activation function, and this model uses the ReLU function.

[0049] As feature information continues to increase, in order to reduce the amount of computation, VGG sets a pooling layer to reduce the image size to remove redundant information. In this embodiment, a 2×2 maximum pooling algorithm with a step size of 2 is used for downsampling. This means that only the maximum value of the pixel in the 2×2 pixel grid is retained as the representative feature. Its function can be expressed as:

[0050] f pool =Max(X m,n ,X m+1,n ,X m,n+1 ,X m+1,n+1 )(0≤m≤M,0≤n≤N) (2)

[0051] When building the improved VGG16 model, the input is determined to be a 300×300×3 image, and the output is a 38×38×512 feature map matrix, denoted as conv4_3. The specific convolution and pooling operations are as follows: the input is convolved twice with a kernel size of 3×3-64, followed by a maxpool operation, a convolution operation with a kernel size of 3×3-128 twice, a maxpool operation, a convolution operation with a kernel size of 3×3-256 three times, a maxpool operation, a convolution operation with a kernel size of 3×3-512 three times, and a maxpool operation, a convolution operation with a kernel size of 3×3-512 three times, to obtain the above feature map matrix conv4_3.

[0052] After convolution and pooling, the multidimensional input is flattened into a one-dimensional vector using the Flatten layer and then processed by the three subsequent Dense layers. To improve model efficiency, the number of neurons in the first two Dense layers was reduced from 4096 to 1024 and 512 to speed up model training. The number of output nodes in the third Dense layer is set to the number of categories n, which varies with each sub-model. By comparing the probabilities of each node, the network selects the node with the highest probability, thereby obtaining a multi-class prediction result. The network structure of the entire improved VGG16 model is shown in Table 2.

[0053] Table 2 VGG neural network structure

[0054]

[0055] like Figure 3 As shown in the figure, in order to improve the model training accuracy, the weights of some convolutional layers close to the model input are kept unchanged, that is, these convolutional layers are frozen, so as to retain some learned features and prevent parameter overfitting; the weights of some convolutional layers close to the output are kept updated, that is, these convolutional layers are unfrozen, so that the model has stronger learning ability and can continuously learn new features.

[0056] In this embodiment, during training, the bridge surface information image and its corresponding three-level information are used as the input data and output data of the model, the cross entropy function is selected as the loss function, and RMSprop is used as the optimizer to set a custom learning rate decay based on the loss change to accelerate the training convergence speed. The number of training iterations is set to 100-200 times, and the training result parameters are saved. At the same time, the results are output in the following format for different recognition results: "part-component-disease". Convolution blocks 4 and 5 are unfrozen to speed up the model training speed. A conditional learning rate decay mechanism is introduced, and the initial learning rate is set to 4×10 -6 . If the validation set loss does not decrease, wait for 5 epochs. If the loss still does not decrease, reduce the learning rate with a decay rate of 0.8 and increase the number of epochs to 100 to improve training efficiency. After the above adjustments, the training curve of the model is more stable and has stronger generalization ability. For each classification sub-model, Epoch finally takes values ​​such as 100, 150, and 200. When the model performs well, a smaller epoch is used. When the smaller epoch cannot achieve good accuracy, it is increased to 150 or 200. Then, freeze the convolution base and perform the first training. After the training is completed, unfreeze the two head convolution blocks of the convolution base and perform a second joint training with the custom dense layer.

[0057] At the same time, a validation set is selected to verify the three recognition models, and the model recognition results are compared with the actual classification. The recognition accuracy must reach more than 80%, otherwise the parameters and hyperparameters of the model are adjusted until the error meets the requirements, thereby obtaining a three-level recognition model.

[0058] S3: Input the image containing the bridge appearance information to be predicted into the part recognition model to identify the bridge part corresponding to the image, and then input the image containing the bridge appearance information to be predicted into the component recognition model corresponding to the bridge part to identify the component corresponding to the image; finally, input the image containing the bridge appearance information to be predicted into the defect recognition model corresponding to the component, and output the defect category of the image containing the bridge appearance information to be predicted.

[0059] The present invention also provides a three-level recognition system for bridge surface information based on deep learning, which is used to implement the recognition method. The system includes:

[0060] The bridge image upload module is used as an image upload port to collect the bridge surface image;

[0061] A data enhancer is used to perform transformation and enhancement operations on the bridge appearance image to expand the number of bridge appearance images;

[0062] The three-level recognition module includes a location recognition model, a component recognition model, and a defect recognition model, which are used to gradually identify the shooting location of the bridge surface image, the component where the defect is located, and the defect category;

[0063] The classifier is used to classify and associate the bridge surface image according to the recognition results of the previous level model during the training and verification of the part recognition model, component recognition model, and disease recognition model, and call the corresponding next level model for recognition to complete the transmission of the bridge surface image between models at all levels.

[0064] The following is a specific example to verify the effectiveness of the recognition method and recognition system of the present invention.

[0065] A database of bridge surface information was collected and processed, capturing images of key bridge components, including piers, webs, supports, and joints. A total of 38,026 surface information images were selected, all in JPEG format with a 2048×1536 dpi original image size. The training set contained 4,535 images, including 2,368 images of the bridge deck, 723 images of the superstructure, and 1,444 images of the substructure. The validation set contained 1,170 images, including 595 images of the bridge deck, 180 images of the superstructure, and 395 images of the substructure. The remaining images served as the test set.

[0066] The "Tensorflow+Python" system framework was built using a Win10 system computer, and the VGG16 model was used as the core for improvement.

[0067] Each category of images is used as input data of the improved VGG16 model, and the corresponding image classification is used as output data for model training. The number of iterations is 100-200 times. The training results are saved to obtain a three-level recognition model for bridge appearance information.

[0068] The remaining images are used to verify the effectiveness of the model. The effective recognition rate of the disease must reach more than 80%. Otherwise, the number of iterations of the model is adjusted until the recognition rate meets the requirements. In this embodiment, the average accuracy of the three-level image recognition is 84%. Figure 4 shown.

[0069] The image to be detected is input into the model for recognition, and the classification result of the image after recognition is automatically output, such as Figure 5 As shown in the figure, when an image to be inspected is input into the image upload module, the system first performs a first-level classification and determines that the image belongs to the "bridge deck system." The classifier then substitutes the image into the second-level classification model for the bridge deck system and determines that the image belongs to the "railing" model. The classifier then substitutes the image into the third-level classification model for the bridge deck system and determines that the image belongs to the "crack" model. Finally, the system automatically combines the three classification results and outputs the final result: "bridge deck system - railing - crack."

[0070] During the training process, the number of training rounds is selected as 150 to ensure the classification accuracy of the model. Taking the second-level classification model as an example, Figure 6 As shown in the figure, for the second-level classification models corresponding to the superstructure, substructure, and bridge floor, the training accuracy is greatly improved when Epoch 150 is selected, reaching more than 90%, and the results have good convergence, which is suitable for the application process.

[0071] In order to further demonstrate the beneficial effects of the improved VGG16 model of the present invention, the training results of the three-level model composed of the traditional VGG16 model are trained using the training data set, as well as the training results when one convolution block B5 is unfrozen and two convolution blocks B4 and B5 are unfrozen, as shown in Table 3.

[0072] Table 3. Comparison of original and adjusted VGG performance

[0073]

[0074]

[0075] Table 3 shows that in Case 2, only convolutional block B5 was unfrozen, resulting in an accuracy of less than 70%. However, in Cases 3 through 7, both convolutional blocks B4 and B5 were unfrozen, achieving accuracy exceeding 90%. Therefore, unfreezing convolutional blocks B4 and B5 was chosen. In Cases 3 through 6, by assigning different numbers of neurons to the last two dense layers, it was found that 1024 and 256 neurons in the last two layers achieved the highest recognition accuracy. In Case 7, an attempt to add a new convolutional block resulted in a decrease in recognition performance. Therefore, no new blocks were added, and the corresponding parameters in Case 6 were used for model construction.

[0076] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art will still be able to modify the technical solutions described in the foregoing examples or substitute equivalents for some of the technical features therein. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the invention shall be included within the scope of protection of the invention.

Claims

1. A three-level identification method for bridge appearance information based on deep learning, characterized by: The method specifically includes: S1: Obtain a large number of bridge surface images and annotate each image with the bridge location, the specific bridge component where the defect is located, and the defect category. Classify the images according to the bridge location to construct a training dataset for each bridge location. Then, further classify the images of each bridge location according to the specific bridge component where the defect is located to construct a training dataset for each bridge component in each bridge location. Finally, classify the images of each bridge component according to the defect category to construct a training dataset for each defect category for each bridge component in each bridge location. S2: Construct an improved VGG16 model, that is, modify the number of neurons in the three fully connected layers of the VGG16 model to 1024, 512, and the number of recognition categories, and then train the improved VGG16 model with the training dataset of each bridge part respectively to obtain a part recognition model corresponding to each bridge part; then use the training dataset of each bridge component to train the part recognition model of the corresponding bridge part to obtain a component recognition model of the corresponding component; finally, use the training dataset of each disease category of each bridge component of each bridge part to train the component recognition model of the corresponding component to obtain a disease recognition model of each disease category of the corresponding bridge component; thus, a three-level recognition model is obtained; S3: Input the bridge appearance image to be predicted into the part recognition model to identify the bridge part corresponding to the image, and then input the bridge appearance image to be predicted into the component recognition model corresponding to the bridge part to identify the component corresponding to the image; finally, input the bridge appearance image to be predicted into the defect recognition model corresponding to the component, and output the defect category of the bridge appearance image to be predicted.

2. The three-level identification method for bridge appearance information based on deep learning according to claim 1 is characterized in that: The parts of the bridge include the bridge deck system, the superstructure, and the substructure; each part corresponds to a part identification model; The bridge deck system includes the following components: signboards, isolation nets, railings, curbstones, green belts, drains, bridge deck pavement, sidewalks, expansion devices, and bridgehead settlement; The superstructure includes the following components: steel box girder, arch bridge, beam bridge, drainage hole, stone bridge, small box girder; The substructure includes the following components: ear spar, piers, abutments, supports, and foundations; each component corresponds to a component identification model; Each component corresponds to a disease identification model.

3. The three-level identification method of bridge appearance information based on deep learning according to claim 1 is characterized in that: During training of the improved VGG16 model, the convolution process uses the ReLU function as the activation function, and keeps the weights of some convolutional layers close to the model input unchanged, that is, freezes the three convolutional blocks close to the model input, so as to retain some learned features and prevent parameter overfitting; keeps the weights of some convolutional layers close to the output continuously updated, that is, unfreezes the two convolutional blocks close to the model output, so that the model has stronger learning ability and continuously learns new features; The convolution blocks are separated by max-pooling connections.

4. The three-level identification method for bridge appearance information based on deep learning according to claim 1 is characterized in that: In S1, after obtaining a large number of images containing bridge appearance information, data enhancement is first performed on the images, and then a training data set is constructed using the enhanced images.

5. A three-level recognition system for bridge appearance information based on deep learning, characterized by: The system is used to implement the three-level identification method of bridge surface information based on deep learning as described in any one of claims 1 to 4, and the system includes: The bridge image upload module is used as an image upload port to collect the bridge surface image; A data enhancer is used to perform transformation and enhancement operations on the bridge appearance image to expand the number of bridge appearance images; The three-level recognition module includes a location recognition model, a component recognition model, and a defect recognition model, which are used to gradually identify the shooting location of the bridge surface image, the component where the defect is located, and the defect category; The classifier is used to classify and associate the bridge surface image according to the recognition results of the previous level model during the training and verification of the part recognition model, component recognition model, and disease recognition model, and call the corresponding next level model for recognition to complete the transmission of the bridge surface image between models at all levels.

Images