An intelligent bridge disease identification method based on an improved residual structure

Abstract

The present invention provides an intelligent method for identifying bridge diseases based on an improved residual structure. First, a historical picture dataset of bridge diseases is obtained, converted into a pixel matrix, and then an enhanced data copy is obtained using a data augmentation algorithm and merged into a training set. Then, a convolutional neural network is built, and the built convolutional neural network model is trained, verified, and tested to obtain the recognition result of the target bridge disease types. The present invention improves on the original ResNet neural network, designs a new neural network ResNet+ for bridge disease identification, optimizes the residual structure between networks, and improves the accuracy of bridge disease identification.

Description

Technical Field

[0001] The present invention belongs to the field of infrastructure health detection, and particularly relates to an intelligent bridge disease identification method based on an improved residual structure. Background Art

[0002] From 1990 to 2021, an average of 25,700 new highway bridges were built in China every year. Bridges have become an indispensable and important part of transportation infrastructure, and the existence of bridges is of great significance for connecting regions separated by geographical locations.

[0003] However, during the service life of bridges, due to the combined effects of load fatigue, environmental corrosion, material aging and other factors, it inevitably leads to the accumulation of bridge structure damage and the attenuation of resistance, thereby reducing the ability to resist disasters. How to improve the structural safety and durability of in-service bridges and thus extend the service life of bridges has become a huge challenge and an urgent task faced by related fields. In this regard, researchers have developed a series of detection methods for bridge diseases to timely obtain the state information during the service period of bridges.

[0004] Among them, the damage identification method based on deep learning has achieved certain research results in the field of bridge structural health detection. However, due to reasons such as complex picture backgrounds and long shooting distances, the detection accuracy obtained is often low, the effect is not ideal, and it cannot be widely applied in actual projects. Therefore, the intelligent damage identification method urgently needs to be improved at the algorithm level to improve the accuracy and robustness of identification. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the present invention proposes an intelligent bridge disease identification method based on an improved residual structure, which can improve the intelligent identification accuracy of bridge diseases.

[0006] The object of the present invention is achieved by the following technical solutions:

[0007] An intelligent bridge disease identification method based on an improved residual structure, the method comprising the following steps:

[0008] Step 1: Obtain a historical picture dataset of bridge diseases, where the dataset includes pictures of the service states of bridge components at key structural positions, and label disease tags to organize them into an initial dataset;

[0009] Step 2: Convert the initial data in Step 1 into pixel values of three pixel channels of R, G, and B, and convert them into a pixel matrix according to the positions of the pixel values in the picture;

[0010] Step 3: Use a data augmentation algorithm to perform data augmentation on the images in the initial dataset to obtain the pixel matrix of the augmented data copy; merge the pixel matrix of the initial data and the pixel matrix of the augmented data copy to obtain the training dataset;

[0011] Step 4: Construct a ResNet+ convolutional neural network, and use the training dataset obtained in Step 3 to train the ResNet+ convolutional neural network to generate a weight model of the ResNet+ convolutional neural network;

[0012] The ResNet+ convolutional neural network is based on the ResNet neural network. Inside a single residual block, it uses a class residual connection to replace the single 3x3 convolutional kernel between residual blocks, thereby increasing the receptive field of each network layer and strengthening the feature expression ability of the network on fine-grained disease images;

[0013] Step 5: Convert the bridge disease image data to be recognized into pixel values of the R, G, and B pixel channels, and convert them into a pixel matrix according to their positions in the image, and then input them into the weight model of the ResNet+ convolutional neural network obtained in Step 4 to obtain the category of the bridge disease to be recognized.

[0014] Further, in Step 2, the Cutmix data augmentation algorithm is used to perform data augmentation on the images in the initial dataset, and the calculation formula is as follows:

[0015]

[0016]

[0017] where x A and x B are pixel matrices composed of two randomly selected original bridge disease images, M is a coefficient matrix, where x A , W is the width of the image, H is the height of the image, C is the number of channels, and M ∈ {0,1} W×H is a matrix with H rows and W columns and element values between 0 and 1; is the pixel matrix after data augmentation; y A and y B refer to the original disease labels, λ refers to a number specified between 0 and 1, refers to the newly generated label.

[0018] Furthermore, the ResNet+ convolutional neural network main body includes 16 residual blocks. Each residual block includes a 1x1 convolutional kernel for dimensionality reduction of disease image data, three ResNet+ residual convolutional kernels for feature extraction of disease image data, and a 1x1 convolutional kernel for dimensionality increase and restoration of disease image data;

[0019] The specific steps to replace the single 3x3 convolutional kernel between residual blocks with class residual connections within a single residual block are as follows:

[0020] (1) Divide the output features of the 1x1 convolutional kernel for dimensionality reduction of disease image data into s groups according to the number of channels n. The number of channels of each group of features is w, i.e., n = s × w. Denote each group of features after equal division as x i , where i ∈ {1, 2,...., s};

[0021] (2) Also divide the original residual convolutional kernels into s groups according to the number of channels. The output channels of each group are w. Denote the convolutional operation of each group as Conv i ();

[0022] (3) For each group of features x i after grouping, except for the first group without convolutional operation, other groups all correspond to the convolutional operation Conv i (). Denote y i as the output of the convolutional operation Conv i (). Then, starting from the second group, before each convolutional operation Conv i (), connect the output y i-1 of the previous group with the features x i of the current group as the input of Conv i (). And so on until the last group of features. It can be expressed by the formula as follows:

[0023]

[0024] (4) Concatenate the outputs corresponding to each group and input them into the 1x1 convolutional kernel at the last layer for dimensionality increase and restoration of disease image data. Fuse these multi-scale features to obtain the output of this residual block.

[0025] Furthermore, the layout structure of the ResNet+ neural network is as follows:

[0026]

[0027] Furthermore, the bridge components at the key structural positions in step one include bridge deck pavement, railing, capping beam, cross beam, pier, foundation, and wing wall.

[0028] The beneficial effects of the present invention are as follows:

[0029] The present invention is an improvement based on the original ResNet neural network. Within a single residual block, a class residual connection is used to replace the single 3x3 convolutional kernel between residual blocks, optimizing the residual structure between networks, thereby increasing the receptive field of each network layer and strengthening the feature expression ability of the network on fine-grained disease images, thus improving the accuracy of bridge disease recognition. Brief Description of the Drawings

[0030] Figure 1 is a flowchart of the method of the present invention;

[0031] Figure 2 is a comparison between the picture processed by the Cutmix data augmentation algorithm and the original picture;

[0032] Figure 3 is a flowchart of training using ResNet+ neural network;

[0033] Figure 4 is a comparison chart of the recognition accuracy after applying ResNet+ neural network and the accuracy of other convolutional neural networks. Detailed Embodiments

[0034] The present invention will be described in detail below according to the drawings and preferred embodiments. The purpose and effects of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0035] As Figure 1 shown, the intelligent bridge disease recognition method based on the improved residual structure of the present invention specifically includes the following steps:

[0036] S1: Take multi-faceted and three-dimensional photos of the bridge to be health-detected to ensure that the service status of bridge components at key structural positions can be obtained; the shooting positions include but are not limited to bridge deck paving, railing, bent cap, cross beam, pier, foundation, and wing wall.

[0037] S2: Transfer the pictures taken in S1 into the computer, mark their disease labels according to the specific component conditions obtained by shooting, and store them in a table according to the disease picture storage path - disease label name to organize an initial data set;

[0038] S3: Use the Cutmix data augmentation algorithm to process the initial data set obtained in S2, and generate a copy of the data set after data augmentation each time it is processed.

[0039] The data augmentation algorithm is based on the following formula:

[0040]

[0041]

[0042] Among them, is the original pixel matrix, W is the width of the image, H is the height of the image, C is the number of channels, and M ∈ {0, 1} W×H is a matrix with H rows, W columns, and element values between 0 and 1; is the pixel matrix after data augmentation; y refers to the original disease label, λ refers to the specified number between 0 and 1, refers to the newly generated label.

[0043] S4: According to the principle of graphic imaging, convert the original data in S2 and the enhanced data copy obtained in S3 into pixel values of three pixel channels R, G, and B, and convert them into the form of a pixel matrix according to the positions of the pixel values.

[0044] S5: Based on the Pytorch deep learning tool, construct a ResNet + convolutional neural network, and train it with the pixel matrix obtained in S4. The training process is as Figure 3 shown, and a weight model of the ResNet + convolutional neural network is generated. The training process includes:

[0045] (1) First, give the target classification label corresponding to each pixel matrix according to the bridge disease information in each initial data picture.

[0046] (2) Calculate the transfer values of the pixel matrix in each hidden layer of the ResNet + neural network.

[0047] (3) Calculate the difference between the final output result of the ResNet + neural network and the target classification label as the output of the loss function.

[0048] (4) Denote the tolerance value of the output of the loss function as α. If the difference between the final output and the target classification label exceeds α, then re - input the final output into the hidden layer of the ResNet + neural network for calculation, and calculate its error gradient to update the weight parameters of each hidden layer of the ResNet + neural network.

[0049] (5) When the value output by the loss function is within the tolerance value α, fix the neural network weight parameters and save the weight model of the ResNet + convolutional neural network.

[0050] The main body of the constructed ResNet + neural network consists of 16 residual blocks. In one residual block, it includes: a 1x1 convolutional kernel for dimensionality reduction of disease picture data, three ResNet + residual convolutional kernels for feature extraction of disease picture data, and a 1x1 convolutional kernel for dimensionality restoration of disease picture data.

[0051] Table 1 ResNet+ Neural Network Layout Structure

[0052]

[0053]

[0054] Based on the original residual block of the ResNet neural network, ResNet+ neural network improves the structure of the residual block and replaces the three ResNet+ residual convolution kernels with a class residual structure. The specific improvement steps are as follows:

[0055] a. First, use the parameter s to represent how many groups the bridge disease feature map in the convolution process is divided into;

[0056] b. Next, for the output features of the 1X1 convolution layer in the front of the residual convolution kernel, assuming its number of channels is n, ResNet+ divides it into s groups of features according to the number of channels, and the number of channels of each group of features is w, that is, n = s×w. Denote each group of features after equal division as x i , where i ∈ {1, 2,...., s};

[0057] c. Then, for the original residual convolution kernel, also divide it into s groups according to the process in step b, and the output channels of each group are w. Denote the convolution operation of each group as Conv i ();

[0058] d. For each group of features x i after grouping, except for the first group without convolution operation, other groups all correspond to the convolution operation Conv i (), where i ∈ {1, 2,...., s}. Denote y i-1 as the output of the convolution operation Conv i (). Then, starting from the second group, before each convolution operation Conv i (), the output y i-1 of the previous group will be connected with the features x i of the current group for residual connection and used as the input of Conv i (). And so on until the last group of features. It is expressed by the formula as follows:

[0059]

[0060] e. Finally, splice the outputs corresponding to each group in channels, and then input them into the last 1X1 convolution layer to fuse these multi-scale features to obtain the output of this residual structure.

[0061] S6: Convert the bridge disease data to be recognized into pixel values of three pixel channels, namely R, G, and B, and convert them into a pixel matrix form according to the positions of the pixel values, and input them into the training set weight model. The model outputs the categories of bridge diseases.

[0062] To verify the accuracy of the model prediction, calculate the proportion of the pictures correctly predicted by the model in all the training pictures, which is used as the evaluation index of the model recognition result.

[0063] Embodiment

[0064] Main computer hardware environment: Core TM i9-9900K CPU@3.60GHZ, Nvidia RTX 2080ti; Main computer software environment: Windows 10 operating system, using Python3.7.1, Pytorch 1.7.1, opencv-Python; Shooting hardware: IPhone 11 DXOMARK Camera.

[0065] In this embodiment, a total of 5 types of bridge diseases are sorted out, namely cracks, exposed reinforcement, spalling, pitted surface, and cavities. The distribution of the original data set is shown in Table 2.

[0066] Table 2 Data distribution of the original data set

[0067] Disease name Crack Exposed reinforcement Spalling Honeycombing Hole Quantity 2292 334 720 56 39

[0068] Use the Cutmix data augmentation algorithm to compare the processed disease pictures with the original pictures as Figure 2 shown. The data distribution of the data set after data augmentation is shown in Table 3.

[0069] Table 3 Data distribution of the data set after data augmentation

[0070] Disease name Crack Exposed reinforcement Spalling Honeycombing Hole Quantity 2292 500 1000 500 500

[0071] Use the third-party processing library opencv of python 3.7.1 to read pictures, read the pictures in the initial data set respectively, read the values of their R, G, and B channels, and convert them into a pixel matrix form according to the positions of the pixel values. Input the pixel matrix into the ResNet+ neural network and train the network, including:

[0072] (1) First, give the target classification label corresponding to each pixel matrix according to the bridge disease information in each initial data picture.

[0073] (2) Calculate the transfer values of the pixel matrix in each hidden layer of the ResNet+ neural network.

[0074] (3) Calculate the difference between the final output result of the ResNet+ neural network and the target classification label as the output of the loss function.

[0075] (4) Denote the tolerance value of the output of the loss function as α. If the difference between the final output and the target classification label exceeds α, then re-input the final output into the hidden layer of the ResNet+ neural network for calculation, and calculate its error gradient to update the weight parameters of each hidden layer of the ResNet+ neural network.

[0076] (5) When the value of the output of the loss function is within the tolerance value α, fix the weight parameters of the neural network and save the weight model of the ResNet+ convolutional neural network.

[0077] (6) —— Collect new bridge disease pictures for testing to evaluate the accuracy of the model. At the same time, replace the neural network with ResNet, Vgg, and alexnet in turn. After testing, the bridge disease recognition accuracy after improving the residual structure has been significantly improved. Compared with the original neural network, the highest average accuracy has increased by 0.11. The changes in the accuracy of each neural network with the progress of training cycles are as Figure 4 shown.

[0078] (6) Those of ordinary skill in the art can understand that the above are only preferred examples of the invention and are not used to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, for those skilled in the art, they can still modify the technical solutions described in the foregoing examples, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, etc. made within the spirit and principle of the invention shall be included within the protection scope of the invention.

Claims

1. An intelligent identification method for bridge diseases based on an improved residual structure, characterized in that, The method includes the following steps: Step 1: Obtain a historical picture dataset of bridge diseases. The dataset includes pictures of the service status of bridge components at key structural positions, and disease labels are marked and organized into an initial dataset; Step 2: Convert the initial data in the initial dataset in Step 1 into pixel values of three pixel channels of R, G, and B, and convert them into a pixel matrix according to the positions of the pixel values in the picture; Step 3: Use a data augmentation algorithm to perform data augmentation on the pictures in the initial dataset to obtain the pixel matrix of the augmented data copy; Merge the pixel matrix of the initial data and the pixel matrix of the augmented data copy to obtain a training dataset; Step 4: Construct a ResNet+ convolutional neural network, and use the training dataset obtained in Step 3 to train the ResNet+ convolutional neural network to generate a weight model of the ResNet+ convolutional neural network; The ResNet+ convolutional neural network is based on the ResNet neural network. Inside a single residual block, a class residual connection is used to replace the single 3x3 convolutional kernel between residual blocks, thereby increasing the receptive field of each network layer and strengthening the feature expression ability of the network on fine-grained disease images; Step 5: Convert the bridge disease image data to be recognized into pixel values of three pixel channels of R, G, and B, and convert them into a pixel matrix according to the positions of the pixel values in the picture, and then input them into the weight model of the ResNet+ convolutional neural network obtained in Step 4 to obtain the category of the bridge disease to be recognized; The specific steps of using a class residual connection to replace the single 3x3 convolutional kernel between residual blocks inside a single residual block are as follows: 1) Divide the output features of the 1x1 convolutional kernel for dimensionality reduction of disease image data into s groups evenly according to the number of channels n. The number of channels of each group of features is w, that is, n = s × w. Denote each group of features after even division as x i , i ∈ {1, 2,...., s}; 2) Divide the original residual convolution kernel into s groups according to the number of channels. The output channels of each group are w, and the convolution operation of each group is denoted as Conv i (). 3) For each group of features x after grouping i , except that the first group does not have a convolution operation, other groups all correspond to the convolution operation Conv i (), let y i be the output of the convolution operation Conv i (), then starting from the second group, before each convolution operation Conv i (), the output y i-1 of the previous group is connected to the features x i of the current group by a residual connection and used as the input of Conv i (), and so on until the last group of features. It is expressed by the formula as follows: 4) Perform channel splicing on each group of corresponding outputs, input the 1x1-sized convolutional kernel for dimensionality reduction and restoration of disease picture data in the last layer, and fuse these multi-scale features to obtain the output of the residual block.

2. The intelligent identification method for bridge diseases based on an improved residual structure according to claim 1, characterized in that, In Step 3, the Cutmix data augmentation algorithm is used to perform data augmentation on the pictures in the initial dataset, and the calculation formula is as follows: Among them, x A , x B are pixel matrices composed of two randomly selected original bridge disease images, M is a coefficient matrix, where W is the width of the image, H is the height of the image, C is the number of channels, M ∈ {0, 1} W×H is a matrix with H rows, W columns and element values between 0 and 1; is the pixel matrix after data augmentation; y A , y B refer to the original disease labels, λ refers to the specified number between 0 and 1, refers to the newly generated label.

3. The intelligent identification method for bridge diseases based on an improved residual structure according to claim 1, characterized in that, The main body of the ResNet+ convolutional neural network includes 16 residual blocks. Each residual block includes a 1x1-sized convolutional kernel for dimensionality reduction of disease picture data, three ResNet+ residual convolutional kernels for feature extraction of disease picture data, and a 1x1-sized convolutional kernel for dimensionality reduction and restoration of disease picture data.

4. The intelligent identification method for bridge diseases based on an improved residual structure according to claim 3, characterized in that, The layout structure of the ResNet+ convolutional neural network is as follows: The number of output channels of the conv1 layer is 64, the output size is 256×256 and 128×128, and the convolution type is 7×7 conv, stride 2; The number of output channels of the conv2_x layer is 64, the output size is 128×128, the convolution type is 3×3 max pool, stride 2, ResNet+ block×3; The number of output channels of conv3_x is 128, the output size is 64×64, and the convolution type is ResNet+ block×4; The number of output channels of the conv4_x layer is 256, the output size is 32×32, and the convolution type is ResNet+ block×6; The number of output channels of conv5_x is 512, the output size is 16×16, and the convolution type is ResNet+block×3; The number of output channels of the last layer is 1024, the output size is 16×16, and the convolution types are average pool and softmax.

5. The intelligent identification method for bridge diseases based on an improved residual structure according to claim 1, characterized in that, The bridge components at the key structural positions in Step 1 include bridge deck paving, railing, capping beam, cross beam, pier, foundation, and wing wall.

Images