Steel structure bridge corrosion and crack disease measurement method and system

Abstract

The present invention relates to a method and system for measuring corrosion and crack diseases of steel structure bridges, comprising the following steps: collecting a data set of corrosion and crack images of steel structure bridges, dividing the data set into a training set, a test set, and a validation set, wherein the data set contains no less than 70% of disease images; constructing and training an Improved Max‑DeepLab neural network with enhanced end-to-end panoramic segmentation: training the Improved Max‑DeepLab neural network with the data set to obtain the category to which the disease belongs and the area where the disease is located; obtaining the width of the crack disease and the area of ​​the corrosion disease respectively according to the predicted category to which the disease belongs and the area where the disease is located. The Improved Max‑DeepLab neural network with enhanced end-to-end panoramic segmentation effectively realizes high-precision distinguishing and quantitative measurement of the two diseases of corrosion and cracks of steel structure bridges, and can accurately and efficiently detect and quantify corrosion and crack defects, with a similarity of more than 49%, providing an effective means for quality assurance and maintenance of steel structure bridges.

Description

Technical Field

[0001] The present invention relates to the field of image semantic segmentation and intelligent construction technology, and in particular to a method and system for measuring corrosion and crack damage of steel structure bridges based on deep learning. Background Art

[0002] Steel structure bridges, as an important part of modern transportation infrastructure, carry the safe passage of vehicles and pedestrians. However, during the long-term service process, due to various reasons, steel structure bridges will have various diseases, which not only affect their performance, but also may pose a threat to traffic safety. 1. Steel is prone to corrosion and rust in a humid and aerobic environment. Especially in the connection parts, welds and places where the protective coating of the bridge is damaged, the corrosion and rust phenomenon is more serious. Long-term corrosion and rust will lead to a reduction in the cross-section and strength of the steel, which will affect the overall safety of the bridge; 2. Steel structure bridges are prone to fatigue damage under the action of cyclic loads. Fatigue cracks usually start from stress concentration points and continue to expand with the repeated action of loads. If not discovered and repaired in time, cracks may cause partial failure of the bridge or even overall collapse; 3. Steel structure bridges may deform when subjected to external loads, accelerating the development of diseases; 4. Environmental factors such as temperature, humidity, wind, rain, snow, etc. will have an impact on steel structure bridges. For example, temperature changes may cause temperature stress on the bridge, exacerbating the expansion of cracks; rain and snow may cause water on the bridge surface to freeze, increasing the load on the bridge; wind loads may also affect the stability of the bridge.

[0003] These defects will not only seriously affect the overall structural performance of the bridge, resulting in a decrease in the bridge's bearing capacity, but also reduce the bridge's safety performance in severe cases and increase the risk of traffic accidents. They will also gradually increase the difficulty of bridge maintenance, shorten the service life of steel structure bridges, and increase the cost of maintenance and replacement. They may also have a negative impact on the social economy and social life. For example, the closure of a bridge for maintenance may cause traffic congestion, affecting people's travel and daily life. Therefore, the large-scale normalized measurement of steel structure bridge defects is of great significance.

[0004] Corrosion and cracks are two common types of defects in steel structure bridges. Ultrasonic testing has become one of the main methods for measuring corrosion and crack defects in steel structure bridges. Ultrasonic testing equipment mainly includes ultrasonic transmitters, receivers, signal processing systems, and display and recording equipment. The transmitter generates high-frequency ultrasonic signals and transmits the signals to the surface of the steel structure bridge through a coupling agent; the receiver receives the ultrasonic signals reflected by the defects; the signal processing system filters and amplifies the received signals; the display and recording equipment displays the processed signals in the form of waveforms, images, etc. for analysis by the inspectors. However, the material type and properties of steel structure bridges have a direct impact on the effect of ultrasonic testing. For example, some alloy elements and impurities may cause sound wave scattering and reduce the detection accuracy. In addition, ultrasonic testing is costly and has real-time effects, and can only be controlled afterwards, which limits its role. In addition, the maintenance and calibration of the equipment also require professional knowledge and skills, which increases the difficulty of use and maintenance.

[0005] In recent years, with the rapid development of intelligent technology, its application in the measurement of steel structure bridge defects has become more and more extensive. Image segmentation technology has been widely used in fields such as computer vision and natural language processing. Applying this technology to the detection of corrosion and crack diseases in steel structure bridges is an ideal intelligent detection method, which can realize real-time quality monitoring and quantitative analysis of corrosion and cracks. However, the two types of diseases in steel structure bridges, corrosion type and crack type diseases, are easy to cause confusion because they show similar symptoms visually in the early stage, such as surface discoloration or local damage. In the existing intelligent detection methods, it is difficult to achieve satisfactory results for the measurement of these two diseases. In the existing intelligent detection methods, for example, the traditional microscopic crack recognition network is used in the quality inspection of 3D printed bridge decks based on semantic segmentation. Although some crack and bubble defect images can be located, the positioning and detection of corrosion defects and crack disease images, especially in the early stage, are similar, and the detection accuracy is low. Therefore, for the detection of corrosion and crack diseases in steel structure bridges, it is urgent to use a stable, real-time, and high-accuracy deep learning algorithm for detection and measurement. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the above technologies and provide a stable, real-time, high-accuracy, intelligent steel structure bridge corrosion and crack disease measurement method and system.

[0007] In order to solve the above technical problems, the technical solution of the present invention is:

[0008] In a first aspect, the present invention provides a method for measuring corrosion and crack damage of a steel structure bridge, the method comprising the following steps:

[0009] Step 1: Collection of steel structure bridge corrosion and crack image datasets

[0010] The inspection robot is used to inspect the bridge section, and the steel structure bridge is photographed through the camera. The inspection robot stores the video file taken every 100 meters, and then repeats this cycle to obtain a 20-minute video file for each area. Then, the video file obtained in each area is framed, and the frame extraction process is as follows:

[0011] 1) Use video editing software to import the obtained regional video file for frame extraction, set the frame rate to 1 frame / second, and extract 1 frame of image per second;

[0012] 2) After the setting is completed, extract frames from the video file, generate 1200 frames of images for every 20 minutes of video file, save the images generated by extracting frames, and set the format to PNG;

[0013] The images obtained by sampling frames are screened to include images with corrosion diseases, images with crack diseases, and images without diseases, and are annotated frame by frame. A data set is constructed using images obtained by multiple frame sampling, and the data set is divided into a training set, a test set, and a validation set, wherein the images with diseases account for no less than 70% of the data set;

[0014] Step 2: Build and train the Improved Max-DeepLab neural network for end-to-end panoramic segmentation:

[0015] The entire network of the Improved Max-DeepLab neural network adopts the Transformer framework, including an encoder and a decoder;

[0016] The encoder consists of a preprocessing module, a 3-layer linear bottleneck module, a 4-layer linear bottleneck module, a 6-layer axial attention bottleneck module, and a dual-path Transformer module;

[0017] The input image is processed by the preprocessing module, and then passes through a 3-layer linear bottleneck module and a 4-layer linear bottleneck module to extract features to obtain a pixel feature map A. The pixel feature map A enters a 6-layer axial attention bottleneck module and then enters the pixel path of the dual-path Transformer module. The dual-path Transformer module also inputs a global memory, which is a trainable parameter.

[0018] The decoder includes a plurality of stacked decoder bottleneck Bottle neck modules, each of which includes five 1X1 convolutions and one 3X3 convolution. On the pixel path, each stacked decoder bottleneck Bottle neck module has two inputs, one is the feature output obtained in the previous layer, and the other is the pixel feature map A output by the 4-layer linear bottleneck module;

[0019] On the one hand, the feature output obtained in the previous layer is subjected to a 1X1 convolution for feature fusion and then up-sampled to obtain a new feature B; on the other hand, the feature output obtained in the previous layer is subjected to a 1X1 convolution to obtain a new feature D;

[0020] The pixel feature map A undergoes a 1X1 convolution for feature fusion and then upsampling to obtain new features, and then undergoes a 3X3 convolution and a 1X1 convolution for feature fusion to obtain a new feature C;

[0021] The new features B, C, and D are added together in a matrix and then fused through a 1X1 convolution. This process is repeated again, and features of different scales are extracted in multiple stacked decoder bottleneck modules. The decoded feature E is then fused through bilinear adjustment and simple summation.

[0022] The decoded feature E is processed by a 3X3 dilated convolution (the number of dilated features is 2) and then matrix-multiplied with the mask features extracted by two fully connected layers on the global path to obtain a mask image. Finally, the final mask image is output after a batch normalization module, and the category of the disease is predicted.

[0023] Use the data set from step 1 to train the Improved Max-DeepLab neural network to obtain the disease category and the area where the disease is located;

[0024] Step 3: Calculate the size of the disease

[0025] According to the predicted category of the disease and the area where the disease is located, the width of the crack disease and the area of ​​the corrosion disease are obtained respectively.

[0026] Furthermore, the preprocessing module includes a 7×7 standardized convolution, a group normalization, a Relu activation function and a pooling layer connected in sequence;

[0027] The linear bottleneck module has three 1X1 convolutions, one 3X3 depth convolution, and four batch normalization modules built in. The input feature map is sequentially processed by 1X1 convolution, batch normalization module, Relu activation function, 3X3 depth convolution, batch normalization module, Relu activation function, 1X1 convolution, and batch normalization module to obtain a fused feature map Q.

[0028] At the same time, the input feature map is directly processed by the third 1X1 convolution and batch normalization module, and the result is matrix added with the fused feature map Q. After the linear activation function, the output feature map of a linear bottleneck module is obtained.

[0029] Furthermore, the attention calculation formula in the bottleneck module of the axial attention is:

[0030]

[0031] Among them, y o represents the output at the random initial position o of the input after pooling the projected input; N is the number of position grids formed by the output of the preprocessing module; softmax p represents the softmax function at the p = (a, b)th position; q represents the query, k is the key value, and v is the value; the learnable vector r represents the relative position encoding from p = (a, b) to o = (i, j); the superscript T represents the transposition; k p is the key value at position p, v p is the value at position p, is a learnable vector about the value from position p to point o; is a learnable vector about the key value from position p to point o; is a learnable vector for key queries from position p to point o.

[0032] Furthermore, the dual-path Transformer module has two built-in paths, one is a pixel path, and the other is a global path. A cross-attention mechanism is used between the two paths. On the pixel path, the pixel features are processed in sequence by a P2P axial attention module, a 1X1 convolution module, a 3X3 convolution module, a batch normalization module, a P2M attention module, a 3X3 convolution module, and a batch normalization module, and then matrix-added with the original input of the pixel path, and the output pixel features are obtained by a Relu activation function. On the global path, the output of the global memory after being processed in sequence by a fully connected layer, a batch normalization module, a Relu activation function, a fully connected layer, and an M2P&M2M attention module is matrix-added with the initial value of the global memory to obtain feature Z. After being processed by the Relu activation function, feature Z enters the feature fusion network module on the one hand, and performs a matrix addition operation with the output of the feature fusion network module on the other hand, and finally is processed by the Relu activation function to obtain the output mask category and mask feature.

[0033] The output result of the first batch normalization module in the pixel path is used as the output of the M2P&M2M attention module in the global path, and the output result of the second fully connected layer in the global path is used as the input of the P2M attention module in the pixel path.

[0034] Furthermore, the similarity metric is used as an evaluation indicator during the training process. The similarity metric is:

[0035] PQ=RQ×SQ,

[0036]

[0037] Among them, y i is the true value, is the predicted value, is the probability of predicting the correct category (i.e., RQ), Dice(m i , ) indicates the prediction mask and the truth mask m i The Dice coefficient (i.e. SQ) between them; PQ represents panoramic quality, sim represents similarity; RQ represents recognition quality, and SQ represents segmentation quality; c i is the category; “^” on the letter indicates prediction;

[0038] The mask categories are crack and corrosion, using an array consisting of M elements Make the total similarity between the predicted result and the true value the largest, that is: predict the arrangement for K is the number of positive masks, i and j are integers in the range of 1-K, and M is the total number of masks;

[0039] Then the loss function L PQ for:

[0040]

[0041]

[0042] in, Indicates that no object matches. represents the positive cross entropy, represents negative cross entropy, α is the balance coefficient, and its value is (0,1); They represent the probability of predicting the correct category, the probability of no object matching, and the predicted mask when the total similarity is maximum.

[0043] Furthermore, the specific process of calculating the disease size is:

[0044] According to the predicted disease category and the area where the disease is located, the high-precision measurement of cracks is achieved by combining bilateral filtering, morphological processing, Canny edge detection, k+1-means clustering, and eight-direction search method to obtain the predicted pixel width of the cracks;

[0045] According to the predicted pixel width of the crack, the actual measured width of the crack defect is obtained by using the conversion ratio between the pixel and the actual size, i.e., the pixel scale S;

[0046] The actual area F of the corrosion disease is obtained by multiplying the predicted area of ​​the corrosion disease by the square of the pixel scale S;

[0047] The pixel scale S is corrected by using the calibration algorithm, and the actual measured width of crack damage and the actual area of ​​corrosion damage after calibration are obtained with the optimal pixel scale S*.

[0048] In a second aspect, the present invention provides a steel structure bridge corrosion and crack disease measurement system, the system comprising:

[0049] Image acquisition module, using inspection robots to capture video images of steel structure bridges;

[0050] The image processing module is used to extract frames from the video images of the image acquisition module, mark them, and divide the data set;

[0051] Improved Max-DeepLab neural network for end-to-end panoptic segmentation to obtain the disease category and disease area;

[0052] The defect size calculation module uses bilateral filtering, morphological processing, Canny edge detection, k+1-means clustering, and eight-direction search method to process the output results of the Improved Max-DeepLab neural network to obtain the predicted pixel width of the crack and the predicted area of ​​the corrosion defect; obtain the conversion ratio S between the pixel and the actual size, correct S using the calibration algorithm, and use the optimal pixel scale S* to combine the predicted pixel width of the crack and the predicted area of ​​the corrosion defect to obtain the actual measured width of the crack after calibration and the actual area of ​​the corrosion defect after calibration;

[0053] The feedback processing module sets the measurement parameter threshold, and compares the actual measured width of the crack disease after calibration and the actual area of ​​the corrosion disease after calibration with the corresponding measurement parameter threshold. When the measurement parameter threshold is exceeded, the bridge structure is abnormal, and feedback is given to check the bridge status; when the measurement parameter threshold is not exceeded, the bridge is considered qualified.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The method of the present invention adopts the Improved Max-DeepLab neural network with enhanced end-to-end panoramic segmentation to effectively realize the high-precision differentiation and quantitative measurement of corrosion and cracks of steel structure bridges. It can accurately and efficiently detect and quantify corrosion and crack defects, with a similarity of more than 49%, providing an effective means for quality assurance and maintenance of steel structure bridges. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the structure of the Improved Max-DeepLab neural network in the present invention.

[0057] Figure 2 It is a schematic diagram of the structure of the preprocessing module of the Improved Max-DeepLab neural network in the present invention;

[0058] Figure 3 Schematic diagram of the structure of the linear bottleneck module of the Improved Max-DeepLab neural network in the present invention;

[0059] Figure 4 It is a structural diagram of the dual-path Transformer module of the Improved Max-DeepLab neural network in the present invention. DETAILED DESCRIPTION

[0060] The present invention is further explained below in conjunction with the embodiments and drawings, but this is not intended to limit the scope of protection of the present application.

[0061] Example 1

[0062] The present invention provides a method for measuring corrosion and crack damage of steel structure bridges. The overall method is as follows: preparing a data set of corrosion and crack images of steel structure bridges, training bridge damage images using an enhanced end-to-end panoramic segmentation Improved Max-DeepLab neural network, and further realizing quantitative measurement of steel structure bridge damage images. The method comprises the following steps:

[0063] 1. Collection of steel structure bridge corrosion and crack image dataset

[0064] The inspection robot is used to inspect the bridge section and shoot the steel structure bridge through the camera. The inspection robot stores the video file every 100 meters, and then repeats the process to obtain a 20-minute video file for each area. Then, the video file obtained in each area is framed to obtain the image data of the required steel structure bridge defect area. The frame extraction process is as follows:

[0065] 1. Use video editing software, such as Adobe Premiere Pro, Final Cut Pro, etc., to import the obtained area file for frame extraction.

[0066] 2. Set the frame rate to 1 frame / second, and 1 frame image will be extracted every second.

[0067] 3. After the settings are completed, extract frames from the video file. Every 20 minutes of video file can generate 1200 frames of images. Save the images generated by extracting frames and set the format to PNG.

[0068] 4. 8,000 steel structure bridge corrosion images and crack images and 4,000 conventional steel structure bridge images obtained by multiple frame extraction, a total of 12,000 images. In this embodiment, each image in the data set is a PNG image of 512 x 512 pixels.

[0069] The dataset is divided into training set, test set and validation set. The training set and validation set are used to train the Improved Max-DeepLab neural network for end-to-end panoramic segmentation.

[0070] 2. Build and train the Improved Max-DeepLab neural network for enhanced end-to-end panoramic segmentation:

[0071] The entire network of the Improved Max-DeepLab neural network adopts the Transformer framework, and its overall structure is as follows Figure 1 As shown, it includes an encoder and a decoder;

[0072] The encoder consists of a preprocessing module, a 3-layer linear bottleneck module, a 4-layer linear bottleneck module, a 6-layer axial attention bottleneck module, and a dual-path Transformer module;

[0073] The input image is processed by the preprocessing module, and then passes through a 3-layer linear bottleneck module and a 4-layer linear bottleneck module to extract features to obtain a pixel feature map A. The pixel feature map A enters a 6-layer axial attention bottleneck module and then enters the pixel path of the dual-path Transformer module. The dual-path Transformer module also inputs a global memory, which is a trainable parameter.

[0074] The decoder includes a plurality of stacked decoder bottleneck Bottle neck modules, each of which includes five 1X1 convolutions and one 3X3 convolution. On the pixel path, each stacked decoder bottleneck Bottle neck module has two inputs, one is the feature output obtained in the previous layer, and the other is the pixel feature map A output by the 4-layer linear bottleneck module;

[0075] On the one hand, the feature output obtained in the previous layer is subjected to a 1X1 convolution for feature fusion and then up-sampled to obtain a new feature B; on the other hand, the feature output obtained in the previous layer is subjected to a 1X1 convolution to obtain a new feature D;

[0076] The pixel feature map A undergoes a 1X1 convolution for feature fusion and then upsampling to obtain new features, and then undergoes a 3X3 convolution and a 1X1 convolution for feature fusion to obtain a new feature C;

[0077] The new features B, C, and D are added together in a matrix and then fused through a 1X1 convolution. This process is repeated again, and features of different scales are extracted in multiple stacked decoder bottleneck modules. The decoded feature E is then fused through bilinear adjustment and simple summation.

[0078] The decoded feature E is processed by a 3X3 dilated convolution (the number of dilated features is 2) and then matrix-multiplied with the mask features extracted by two fully connected layers on the global path to obtain a mask image. Finally, the final mask image is output after a batch normalization module, and the category of the disease is predicted.

[0079] The Improved Max-DeepLab neural network is trained using the data set from step 1 to obtain the category of the disease and the area where the disease is located.

[0080] Next, each module of Improved Max-DeepLab is further introduced;

[0081] In the encoder, the collected images are first input into the preprocessing module for processing. The input features are respectively extracted through a 3-layer linear bottleneck module and a 4-layer linear bottleneck module. The pixel features obtained after extraction enter a 6-layer axial attention bottleneck module. The pixel features obtained after calculation and a global memory (actually a trainable parameter) of the input enter the dual-path Transformer module.

[0082] The present invention introduces a preprocessing module into the network, such as Figure 2As shown in the figure, the preprocessing module includes 7×7 standardized convolution, group normalization, Relu activation function and pooling layer connected in sequence, which can solve the gradient explosion problem and prevent data overfitting, has stable training dynamics, does not depend on the statistics between batches, does not depend on the batch size, is insensitive to the batch size, is particularly suitable for situations where the batch size is limited, and improves the performance on small data sets, simplifies the deployment and maintenance of neural networks, and can

[0083] It accelerates the training efficiency of neural networks and has better results in distinguishing between corrosion and cracks.

[0084] The linear bottleneck module is as follows Figure 3 As shown, three 1X1 convolutions, one 3X3 convolution, and four batch normalization processing modules are built in. The input feature map is processed by 1X1 convolution dimension increase and batch normalization, 3X3 depth convolution and batch normalization, 1X1 convolution dimension reduction and batch normalization to obtain a fused feature map Q. The first two convolution activation functions use the Relu function, and the last dimension reduction convolution uses a linear activation function instead of the Relu function. The input image is subjected to 1X1 convolution and batch normalization and matrix addition with the obtained feature map Q, and the output feature map of a layer of linear bottleneck module is obtained through a linear activation function. In the present invention, three layers of linear bottleneck modules and four layers of linear bottleneck modules are cascaded, first increasing the dimension and then reducing the dimension, and a linear activation function is used to reduce the number of input image feature parameters, reduce the amount of calculation, and extract more image feature information, effectively preventing the feature from losing too much information in low-dimensional space and causing information collapse.

[0085] The bottleneck module of the axial attention, the attention calculation formula in this module is:

[0086]

[0087] Among them, y o represents the output at the random initial position o of the input after pooling the projected input; N is the number of position grids formed by the output of the preprocessing module; softmax p represents the softmax function at the p = (a, b)th position; q represents the query, k is the key value, and v is the value; the learnable vector r represents the relative position encoding from p = (a, b) to o = (i, j); the superscript T represents the transposition; k p is the key value at position p, v p is the value at position p, is a learnable vector about the value from position p to point o; is a learnable vector about the key value from position p to point o; is the learnable vector for key query from position p to point o; m is the number of columns in N.

[0088] The dual path converter module is as follows Figure 4 As shown;

[0089] The module has two built-in paths, one is the pixel path and the other is the global path. A cross-attention mechanism is used between the two paths (the input between the attention modules can come from the output on another path). The pixel features include a P2P axial attention module, a 1X1 convolution, two 3X3 convolutions, two batch normalization modules and a P2M attention module. The input pixel features are weighted by the P2P axial attention module, and the obtained pixel features are fused by a 1X1 convolution, a 3X3 convolution and a batch normalization module. The fused features are input to a P2M attention module for calculation. The calculation formula of this module at pixel position a is:

[0090]

[0091] in, represents the output of attention at pixel position a, softmax n Represents a softmax function of length N on the global path;

[0092] represents a query at pixel position a, represents the key value at global position n, Represents the value at the global position n. The image features obtained after calculation are added to the initial values ​​of the pixel features by matrix and then enter the 3X3 convolution module and batch normalization module for feature fusion. The fused features are input into the Relu function and the pixel features are output.

[0093] The global path includes two fully connected layers, a batch normalization module, an M2P&M2M attention module, and a feature fusion network module. The input global memory is extracted through two fully connected layers, and the obtained features are input into an M2P&M2M attention module for calculation. The calculation formula of this module at position b is:

[0094]

[0095] in, represents the output of attention at the global position b,

[0096] softmax n Indicates that the length of the global path is The softmax function, represents a query at global position b;

[0097] represents the key value of the pixel position and the global position matrix at n,

[0098] v pm Represents the value of the pixel position and the global position matrix at n.

[0099] On the global path, the output of the global memory after being processed by a fully connected layer, a batch normalization module, a Relu activation function, a fully connected layer, and an M2P&M2M attention module is matrix-added with the initial value of the global memory to obtain feature Z. After being processed by the Relu activation function, feature Z enters the feature fusion network module on the one hand, and performs a matrix addition operation with the output of the feature fusion network module on the other hand. Finally, it is processed by the Relu activation function to obtain the output mask category and mask feature.

[0100] In the present invention, a 3×3 dilated convolution (the number of dilated holes is 2) is added for feature fusion, which can not only increase the size of the receptive field but also reduce the information loss of the image.

[0101] The similarity metric is used as the evaluation indicator during the training process. The similarity metric is:

[0102] PQ=RQ×SQ,

[0103]

[0104] Among them, y i is the true value, is the predicted value, is the probability of predicting the correct category (i.e., RQ), Represents the prediction mask and the truth mask m i The Dice coefficient (i.e. SQ) between them; PQ represents panoramic quality, sim represents similarity; RQ represents recognition quality, and SQ represents segmentation quality; c i is the category; “^” on the letter indicates prediction;

[0105] The mask categories are cracks and corrosion, and a permutation σ∈θM consisting of M elements is used to maximize the total similarity between the predicted result and the true value, that is: the predicted permutation for K is the number of positive masks, i and j are integers in the range of 1-K, and M is the total number of masks;

[0106] Then the loss function L PQ for:

[0107]

[0108] in, Indicates that no object matches. represents the positive cross entropy, represents negative cross entropy, α is the balance coefficient, and its value is (0,1); They represent the probability of predicting the correct category, the probability of no object matching, and the predicted mask when the total similarity is maximum.

[0109] The Improved Max-DeepLab neural network of the present invention directly predicts a set of non-overlapping masks and their corresponding semantic labels. The output masks and labels are optimized using the panoramic quality (PQ) style index. The goal of the training model is to maximize the similarity between the ground truth mask and the predicted mask, retain rich image semantic information in high-resolution features, and be able to predict the maximum probability category. The corrosion and crack disease type data sets are input into the ImprovedMax-DeepLab for masking operations, and the masked images of the two are compared with the real images, that is, PQ (panoramic quality), that is, accuracy, to complete the discrimination and measurement of the two types of diseases. On the other hand, the present invention uses cross attention instead of the standard self-attention mechanism, and has lower computational complexity and better performance than Max-DeepLab on multiple semantic segmentation data sets, such as the Crack 500 data set and the Cracktree200 data set. The present invention trains 150 epochs, and when the training model reaches convergence, that is, the model training gradient is close to 0. ImprovedMax-DeepLab is evaluated on the validation set with DETR, Axial-DeepLab, Panoptic-DeepLab, Max-DeepLab-s, etc., and the test results on the validation set are significantly improved, as shown in Table 1. Table 1 shows the test results of the COCO dataset:

[0110] Table 1

[0111]

[0112] Furthermore, the data set is enhanced:

[0113] Perform data augmentation on the labeled data set:

[0114] ① Each image is horizontally flipped with a probability of 50%, and the corresponding label is also horizontally flipped in the same way. ② Randomly crop a fixed-size area from the horizontally flipped image and label, and the cropped image is set to 90% of the original image. The cropped image is the same size as the label. ③ Then randomly rotate the cropped image and label with a rotation angle of +5 degrees. ④ Randomly scale the rotated image and label, and set the scaling factor to 1.0. ⑤ Adjust the brightness and contrast of the scaled image to simulate images under different brightness conditions, and set the brightness to 1.1.

[0115] The images and labels after the above data enhancement processing are enhanced to obtain the enhanced dataset R * .

[0116] The enhanced dataset R * The training set, validation set, and test set are allocated in a ratio of 8:2.

[0117] The crack defect image of the steel structure bridge to be detected is input into the Improved Max-DeepLab neural network. After obtaining the output, the defect size is calculated and the measurement parameter threshold is set. The actual measured width of the crack disease after calibration and the actual measured area of ​​the corrosion disease after calibration are compared with the corresponding measurement parameter thresholds. When the measurement parameter threshold is exceeded, the bridge structure is abnormal and feedback is given to check the bridge status; when the measurement parameter threshold is not exceeded, the bridge is considered qualified.

[0118] Example 2

[0119] The steel structure bridge corrosion and crack disease measurement system of this embodiment includes:

[0120] Image acquisition module, using inspection robots to capture video images of steel structure bridges;

[0121] The image processing module is used to extract frames from the video images of the image acquisition module, mark them, and divide the data set;

[0122] Improved Max-DeepLab neural network for end-to-end panoptic segmentation to obtain the disease category and disease area;

[0123] The defect size calculation module uses bilateral filtering, morphological processing, Canny edge detection, k+1-means clustering, and eight-direction search method to process the output results of the Improved Max-DeepLab neural network to obtain the predicted pixel width of the crack and the predicted area of ​​the corrosion defect; obtain the conversion ratio S between the pixel and the actual size, correct S using the calibration algorithm, and use the optimal pixel scale S* to combine the predicted pixel width of the crack and the predicted area of ​​the corrosion defect to obtain the actual measured width of the crack after calibration and the actual area of ​​the corrosion defect after calibration;

[0124] The feedback processing module sets the measurement parameter threshold, and compares the actual measured width of the crack disease after calibration and the actual area of ​​the corrosion disease after calibration with the corresponding measurement parameter threshold. When the measurement parameter threshold is exceeded, the bridge structure is abnormal, and feedback is given to check the bridge status; when the measurement parameter threshold is not exceeded, the bridge is considered qualified.

[0125] The specific process of determining S is:

[0126] Place a ruler in the image with a length of L. The pixel length of the ruler in the image is measured to be L. The pixel scale S can be calculated by the following formula:

[0127] S=L / P

[0128] Among them, S represents the conversion ratio between pixels and actual size, that is, the pixel scale, L represents the actual length of the ruler, and P represents the pixel length of the ruler in the image.

[0129] It should be noted that the calculation of pixel scale requires that the length of the ruler in the image be accurately measured and that the ruler and the crack defect be on the same plane.

[0130] The pixel scale S is corrected using a calibration algorithm, which is:

[0131] Step 1 uses the following error function E(S) to measure the difference between the pixel area and the actual area:

[0132]

[0133] Where Bi represents the actual area of ​​the i-th defect, obtained by annotation; S represents the pixel scale; Ai represents the pixel area of ​​the i-th defect, and N is the total number of predicted defects in all samples in the test set;

[0134] Step 2: Minimize the error function E(S), take the derivative of E(S) and set the derivative equal to 0 to obtain the optimal pixel scale S * ,

[0135]

[0136] The actual measured width of crack damage after calibration and the actual area of ​​corrosion damage after calibration are obtained with the optimal pixel scale S*.

[0137] Calculation of corrosion area: Multiply the predicted pixel area of ​​corrosion F0 by the square of the pixel scale S* to obtain the calibrated actual area F of the corrosion disease, which is calculated using the following formula:

[0138] F=F0×S* 2

[0139] Among them, F0 represents the predicted pixel area of ​​erosion, the unit is pixel square.

[0140] The specific process of crack width measurement:

[0141] According to the predicted category of the disease and the area where the disease is located, the following processing is performed after the crack disease type is determined. The high-precision measurement of the crack width is achieved by combining bilateral filtering, morphological processing, Canny edge detection, k+1-means clustering, and eight-direction search method.

[0142] 1) Using bilateral filtering, firstly, Gaussian blur is used to remove interfering pixels and retain the real micro crack information;

[0143] 2) Then, the micro cracks are refined by morphological processing to obtain a single-pixel micro crack skeleton map. At this time, all pixel values ​​in the micro crack skeleton map are 0 or 1, which is a micro crack skeleton binary map, called a single-pixel micro crack skeleton map; the Canny edge detection algorithm is used to find the image edge and obtain the micro crack edge map by calculating the gray gradient of the single-pixel micro crack skeleton map, and the micro crack edge map obtained by the Canny edge detection is logically ANDed with the single-pixel micro crack skeleton map pixel by pixel;

[0144] 3) k+1-means clustering:

[0145] Select k+1 cluster center points from the results of the logical AND operation, take the k+1th cluster center point as the comparison parameter, and calculate the variance with the results of the first k cluster centers, further enhance the data representation for optimization calculation, and obtain the optimal result. The specific steps are:

[0146] Step 3-1: Randomly initialize k+1 points in the input image set, and select k points as the initial target center points of the k+1-means algorithm;

[0147] Step 3-2: Use the Euclidean distance calculation formula to calculate the geometric distance from the given image set to the initialization target center point;

[0148] Step 3-3: Assign each data point to the nearest cluster according to the principle of shortest distance;

[0149] Step 3-4: Calculate the geometric center of each cluster in the new round. The calculation formula is as follows:

[0150]

[0151] Where P x , P y Respectively represent the horizontal and vertical coordinates of the geometric center point to be calculated for the current cluster; P ix , P iy They represent the horizontal and vertical coordinates of the i-th point in the current cluster respectively; p represents the number of points in the current cluster;

[0152] Use the geometric center of the sample data in each cluster as the cluster center of the new classification;

[0153] Step 3-5: Repeat steps 3-2, 3-3 and 3-4 until the algorithm converges, that is, the center point no longer changes, and output the clustering result;

[0154] Step 3-6: Optimize the obtained clustering results, and use the obtained k center points and the k+1th center point to take the variance. The formula is:

[0155]

[0156] Among them, x k+1 represents the coordinates of the k+1th center point, x j (j=1,2,…k) represents the coordinates of the kth cluster center. Select the cluster center with the smallest variance and find the optimal cluster center;

[0157] Find the optimal clustering result;

[0158] 4) Skeleton extraction: Perform skeleton extraction on the result after k+1-means clustering. The specific skeleton extraction process is as follows:

[0159] Loop through all foreground pixels and mark for deletion those that meet any of the following conditions:

[0160] 2≤N(F1)≤6

[0161] S(F1)=1

[0162] F2F4F6=0

[0163] F4F6F8=0

[0164] F2F4F8=0

[0165] F2F6F8=0

[0166] Where N(F1) represents the number of foreground pixels among the 8 pixels adjacent to F1, S(F1) represents the cumulative number of 0-1 appearing in F2-F9-F2 pixels, where 0 represents background and 1 represents foreground; F represents pixel; the complete pixel position distribution of F1-F9 is:

[0167] <![CDATA[F9]]> <![CDATA[F2]]> <![CDATA[F3]]> <![CDATA[F8]]> <![CDATA[F1]]> <![CDATA[F4]]> <![CDATA[F7]]> <![CDATA[F6]]> <![CDATA[F5]]>

[0168] Until no pixel is marked as deleted, the output result is the skeleton image after the binary image is refined;

[0169] 5) Calculate the micro crack width using the eight-direction search method:

[0170] For the skeleton image after binary image refinement, each micro crack point is used as the initial target center point for width measurement, and an eight-direction search is performed in the skeleton image after binary image refinement extracted from the initial target center point; if it is a micro crack point, the number of pixels in that direction is increased by 1 until the micro crack boundary is reached, and the number of micro crack pixel points in the eight directions of "0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°" where a center point is located is counted respectively, and recorded as n1, n2, n3, n4, n5, n6, n7, n8; the number of micro crack points in the four groups of directions of 0° and 180°, 45° and 225°, 90° and 270°, and 135° and 315° are merged, and the merging method is: d1=n1+n5, d2=2 1 / 2 (n2+n6), d3=n3+n7, d4=2 1 / 2 (n4+n8), take D=min{d1,d2,d3,d4}, and record the direction in which D is obtained, where d1, d2, d3, d4 are the number of pixels in the directions of the four merged straight lines respectively, calculate w=w0×D, where w0 is the actual physical size represented by each pixel, and w is the actual width of the micro crack under the center point of the target; update to the next center point until the widths at all center points are measured.

[0171] The eight-direction search method can easily extract the center point of the crack pixel and the end point of the crack width for any crack curve with strange shapes. Most of the crack shapes seen in daily life are irregular. This method solves the problem of inaccurate crack width measurement by the traditional eight-direction search method.

[0172] Specifically, check the corrosion area and the width of the crack. If the ratio of the corrosion pixel area to the lens detection area does not exceed the measurement parameter threshold (1%), the steel bridge structure is considered qualified. If the width of the crack is less than 0.25mm, the steel bridge structure is considered qualified. If the ratio of the corrosion pixel area to the lens detection area exceeds the measurement parameter threshold (1%), if the width of the crack is greater than 0.25mm, an inspection is required.

[0173] Steel bridge structures are also considered acceptable if the images are not identified as corrosion and crack defects.

[0174] The hardware devices used by the system include the following components:

[0175] Processor: As the core component of the present invention, the processor is responsible for controlling and managing the operation of the entire system, including data acquisition, data processing, image recognition and other functions, and needs to have sufficient computing power and parallel processing capabilities to meet real-time requirements. The processor can be in different forms such as single-chip microcomputers, microprocessors, computers, etc. to meet the needs of different application scenarios;

[0176] Memory: Memory can be used to store collected data and historical data for subsequent processing and analysis. It has the characteristics of high speed, high reliability and scalability to meet the needs of long-term stable operation of the system;

[0177] Database: The back-end system needs to use a database to store and manage collected data, historical data, analysis results and other information;

[0178] Network interface: The back-end system needs to use a network interface to exchange and communicate data with the front-end system and other external systems. It has the characteristics of high speed, high stability and high security to ensure the reliability and security of data transmission.

[0179] The main feature of the computer back-end system memory is that a computer program is stored thereon, and the computer program can be executed by a processor to implement the various steps of the above-mentioned steel structure bridge corrosion and crack disease measurement method.

[0180] The computer back-end system database has the main feature that the database is configured to store and manage computer application data, including various data types and structures, which are applied to each step of the above-mentioned steel structure bridge corrosion and crack disease measurement method.

[0181] The computer back-end system network interface has the main feature of realizing communication and data transmission between computers. The network interface can provide various communication protocols and data transmission methods to meet the communication and data transmission requirements of different application scenarios and different needs, and is applied to each step of the above-mentioned steel structure bridge corrosion and crack disease measurement method.

[0182] The method of the present invention can accurately identify and quantify the corrosion and crack images of steel structure bridges, with a panoramic quality of more than 49%, which is more than 1% higher than the existing method. It has significant progress and can overcome the problems of difficult distinction between corrosion and cracks in steel structure bridges and low detection accuracy.

[0183] In this embodiment, the decoder uses two stacked decoder bottleneck modules for decoding.

[0184] The system of the present invention introduces automation and intelligent technology to improve the accuracy of measuring the corrosion type and crack type of steel structure bridges. The present invention can realize fast and large-scale steel structure bridge corrosion and crack image data acquisition, can extract features from a large amount of acquired data and quickly identify corrosion and cracks in steel structure bridges, and realize fast and accurate detection of steel structure bridge diseases, solving the problem of low accuracy of corrosion type and crack type recognition rate of traditional segmentation models in the early stage, and increasing the recognition efficiency of steel structure bridge diseases. And through the feedback mechanism to guide staff to make adjustments, the system can adapt to various different needs and application scenarios. This high degree of customizability gives the system a wide range of applicability and practicality. It can not only be applied to steel structure bridge disease detection, but also can be extended to other fields, such as surface defect detection in tunnel disease detection, track disease detection and other fields.

[0185] The system of the present invention provides the possibility for realizing larger-scale detection of corrosion-type diseases and crack-type diseases of transportation facilities, further promoting the development of this field. It has great significance and broad application prospects, and also provides strong support and guarantee for the development of steel structure bridge detection technology.

[0186] The present invention adopts Improved Max-DeepLab neural network to greatly improve the accuracy of recognition and quantification, so that the system can achieve real-time high-accuracy quantification, extract features from both shallow features and deep features, and ensure the quality of feature information. At the same time, it also solves the problem of the influence of shallow features on deep features caused by the network level being too deep, making the model more stable and reliable. And through the stacking and reuse of linear bottleneck modules, Improved Max-DeepLab can effectively learn the spatial relationship and semantic information between features, and improve the accuracy of semantic segmentation. The present invention adds a residual learning method to effectively adjust the number of channels of the image to achieve different demand operations of dimensions, and at the same time, the image feature information is processed with low-frequency information and high-frequency information, which greatly ensures the quality of high-frequency information, improves the overall details of the image, ensures the maximum detail texture quality of the image, and enhances the expressiveness of feature information in the image.

[0187] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A method for measuring corrosion and crack damage of steel structure bridges, characterized in that: The method comprises the following steps: Step 1: Collection of steel structure bridge corrosion and crack image datasets The inspection robot is used to inspect the bridge section, and the steel structure bridge is photographed through the camera. The inspection robot stores the video file taken every 100 meters, and then repeats this cycle to obtain a 20-minute video file for each area. Then, the video file obtained in each area is framed, and the frame extraction process is as follows: 1) Use video editing software to import the obtained regional video file for frame extraction, set the frame rate to 1 frame / second, and extract 1 frame of image per second; 2) After the setting is completed, extract frames from the video file, generate 1200 frames of images for every 20 minutes of video file, save the images generated by extracting frames, and set the format to PNG; The images obtained by sampling frames are screened to include images with corrosion diseases, images with crack diseases, and images without diseases, and are annotated frame by frame. A data set is constructed using images obtained by multiple frame sampling, and the data set is divided into a training set, a test set, and a validation set, wherein the images with diseases account for no less than 70% of the data set; Step 2: Build and train the Improved Max-DeepLab neural network for end-to-end panoramic segmentation: The entire network of the Improved Max-DeepLab neural network adopts the Transformer framework, including an encoder and a decoder; The encoder consists of a preprocessing module, a 3-layer linear bottleneck module, a 4-layer linear bottleneck module, a 6-layer axial attention bottleneck module, and a dual-path Transformer module; The input image is processed by the preprocessing module, and then passes through a 3-layer linear bottleneck module and a 4-layer linear bottleneck module to extract features to obtain a pixel feature map A. The pixel feature map A enters a 6-layer axial attention bottleneck module and then enters the pixel path of the dual-path Transformer module. The dual-path Transformer module also inputs a global memory, which is a trainable parameter. The decoder includes a plurality of stacked decoder bottleneck modules, each of which includes five 1X1 convolutions and one 3X3 convolution. On the pixel path, each stacked decoder bottleneck module has two inputs, one is the feature output obtained in the previous layer, and the other is the pixel feature map A output by the 4-layer linear bottleneck module. On the one hand, the feature output obtained in the previous layer is subjected to a 1X1 convolution for feature fusion and then up-sampled to obtain a new feature B; on the other hand, the feature output obtained in the previous layer is subjected to a 1X1 convolution to obtain a new feature D; The pixel feature map A undergoes a 1X1 convolution for feature fusion and then up-samples to obtain new features, and then undergoes a 3X3 convolution and a 1X1 convolution for feature fusion to obtain a new feature C. The new features B, C, and D are added together in a matrix and then fused through a 1X1 convolution. This process is repeated again, and features of different scales are extracted in multiple stacked decoder bottleneck modules. The decoded feature E is then fused through bilinear adjustment and simple summation. The decoded feature E is processed by a 3X3 dilated convolution and then matrix-multiplied with the mask feature extracted by two fully connected layers on the global path to obtain a mask image; finally, the final mask image is output after a batch normalization module, and the category of the disease is predicted; the number of holes in the 3X3 dilated convolution is 2; Use the data set from step 1 to train the Improved Max-DeepLab neural network to obtain the disease category and the area where the disease is located; Step 3: Calculate the size of the disease According to the predicted category of the disease and the area where the disease is located, the width of the crack disease and the area of ​​the corrosion disease are obtained respectively.

2. The method for measuring corrosion and crack damage of steel structure bridges according to claim 1 is characterized in that: The preprocessing module includes a 7×7 standardized convolution, a group normalization, a Relu activation function and a pooling layer connected in sequence; The linear bottleneck module has three 1X1 convolutions, one 3X3 depth convolution, and four batch normalization modules built in. The input feature map is sequentially processed by 1X1 convolution, batch normalization module, Relu activation function, 3X3 depth convolution, batch normalization module, Relu activation function, 1X1 convolution, and batch normalization module to obtain a fused feature map Q. At the same time, the input feature map is directly processed by the third 1X1 convolution and batch normalization module, and the result is matrix added with the fused feature map Q. After the linear activation function, the output feature map of a linear bottleneck module is obtained.

3. The method for measuring corrosion and crack damage of steel structure bridges according to claim 1 is characterized in that: The attention calculation formula in the bottleneck module of the axial attention is: Among them, y o represents the output at the random initial position o of the input after pooling the projected input; N is the number of position grids formed by the output of the preprocessing module; softmax p represents the softmax function at the p = (a, b)th position; q represents the query, k is the key value, and v is the value; the learnable vector r represents the relative position encoding from p = (a, b) to o = (i, j); the superscript T represents the transposition; k p is the key value at position p, v p is the value at position p, is a learnable vector about the value from position p to point o; is a learnable vector about the key value from position p to point o; is a learnable vector for key queries from position p to point o.

4. The method for measuring corrosion and crack damage of steel structure bridges according to claim 1 is characterized in that: The dual-path Transformer module has two built-in paths, one is a pixel path, and the other is a global path. A cross-attention mechanism is used between the two paths. On the pixel path, the pixel features are processed by a P2P axial attention module, a 1X1 convolution module, a 3X3 convolution module, a batch normalization module, a P2M attention module, a 3X3 convolution module, and a batch normalization module in sequence, and then matrix addition operation is performed with the original input of the pixel path, and the output pixel features are obtained by Relu activation function; on the global path, the output of the global memory after being processed by a fully connected layer, a batch normalization module, a Relu activation function, a fully connected layer, and an M2P&M2M attention module in sequence is matrix added with the initial value of the global memory to obtain feature Z. After being processed by the Relu activation function, feature Z enters the feature fusion network module on the one hand, and performs matrix addition operation with the output of the feature fusion network module on the other hand, and finally is processed by the Relu activation function to obtain the output mask category and mask feature; The output result of the first batch normalization module in the pixel path is used as the output of the M2P&M2M attention module in the global path, and the output result of the second fully connected layer in the global path is used as the input of the P2M attention module in the pixel path.

5. The method for measuring corrosion and crack damage of steel structure bridges according to claim 1 is characterized in that: The similarity metric is used as the evaluation indicator during the training process. The similarity metric is: PQ=RQ×SQ, Among them, y i is the true value, is the predicted value, is the probability of predicting the correct category, Represents the prediction mask and the truth mask m i The Dice coefficient between them; PQ represents the panoramic quality, sim represents the similarity; RQ represents the recognition quality, SQ represents the segmentation quality; c i is the category; "^" on the letter indicates prediction; The mask categories are crack and corrosion, using an array consisting of M elements Make the total similarity between the predicted result and the true value the largest, that is: predict the arrangement for K is the number of positive masks, i and j are integers in the range of 1-K, and M is the total number of masks; Then the loss function L PQ for: in, Indicates that no object matches. represents the positive cross entropy, represents negative cross entropy, α is the balance coefficient, and its value is (0,1); They represent the probability of predicting the correct category, the probability of no object matching, and the predicted mask when the total similarity is maximum.

6. The method for measuring corrosion and crack damage of steel structure bridges according to claim 1 is characterized in that: The specific process of calculating the disease size is: According to the predicted disease category and the area where the disease is located, the high-precision measurement of cracks is achieved by combining bilateral filtering, morphological processing, Canny edge detection, k+1-means clustering, and eight-direction search method to obtain the predicted pixel width of the cracks; According to the predicted pixel width of the crack, the actual measured width of the crack defect is obtained by using the conversion ratio between the pixel and the actual size, i.e., the pixel scale S; The actual area F of the corrosion disease is obtained by multiplying the predicted area of ​​the corrosion disease by the square of the pixel scale S; The pixel scale S is corrected by using the calibration algorithm, and the actual measured width of crack damage and the actual area of ​​corrosion damage after calibration are obtained with the optimal pixel scale S*.

7. A steel structure bridge corrosion and crack disease measurement system implementing the steel structure bridge corrosion and crack disease measurement method according to claim 1, characterized in that: The system comprises: Image acquisition module, using inspection robots to capture video images of steel structure bridges; The image processing module is used to extract frames from the video images of the image acquisition module, mark them, and divide the data set; Improved Max-DeepLab neural network for end-to-end panoramic segmentation to obtain the disease category and disease area; The defect size calculation module uses bilateral filtering, morphological processing, Canny edge detection, k+1-means clustering, and eight-direction search method to process the output results of the ImprovedMax-DeepLab neural network to obtain the predicted pixel width of the crack and the predicted area of ​​the corrosion defect; obtain the conversion ratio S between the pixel and the actual size, correct S using the calibration algorithm, and use the optimal pixel scale S* to combine the predicted pixel width of the crack and the predicted area of ​​the corrosion defect to obtain the actual measured width of the crack after calibration and the actual area of ​​the corrosion defect after calibration; The feedback processing module sets the measurement parameter threshold, and compares the actual measured width of the crack disease after calibration and the actual area of ​​the corrosion disease after calibration with the corresponding measurement parameter threshold. When the measurement parameter threshold is exceeded, the bridge structure is abnormal, and feedback is given to check the bridge status; when the measurement parameter threshold is not exceeded, the bridge is considered qualified.

Images