Bridge underwater crack detection method based on deep learning and movable platform

Abstract

The invention discloses a deep learning-based bridge underwater crack detection method and a movable platform, and the method comprises the steps: making a crack test block, calibrating an underwater camera, making a sealed cabin, obtaining and enhancing an underwater image, expanding the underwater image, making an underwater data set, and adding an AF-FPN structure on the basis of a yov5 network. The method comprises the following steps: detecting and positioning a crack of an underwater image in an underwater data set according to an improved yolov5 network, obtaining a target area where the crack is located, replacing all standard convolutional layers in TransUNet with deep separable convolutional layers, forming a DS-TransUNet network, introducing a SeNet attention mechanism into the DS-TransUNet network to form a DS-SeNet-TransUNet network, training the DS-SeNet-TransUNet network, and obtaining the target area where the crack is located. And based on the trained DS-SeNet-TransUNet network, identifying and verifying a target area where the crack is located and acquiring quantitative information of the crack, acquiring a to-be-detected bridge underwater image, and according to the trained DS-SeNet-TransUNet network, detecting the to-be-detected bridge underwater image. The underwater image crack detection and measurement precision is improved, and the detection efficiency is improved based on the movable platform.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a deep learning-based bridge underwater crack detection method and a movable platform. Background Art

[0002] Bridges play an important role in the transportation system. The underwater part of the bridge structure has a complex environment. It is faced with water scouring, corrosion by aquatic organisms, ship collision, and dry-wet cycle at the junction of water and air. These factors can easily cause defects in the underwater part, thus having a negative impact on the safety of the entire bridge. At present, underwater bridge inspection mainly relies on manual and underwater camera detection of underwater cracks. This traditional method has great defects. As the most dangerous underwater disease, crack detection and feature extraction are very important tasks and links.

[0003] As the research on deep learning continues to deepen, it has also achieved good applications in the field of feature detection. Some scholars have used deep learning technology to study the identification of underwater cracks in bridges, but there are few mobile applications that can be deployed for underwater crack disease detection in bridges, and there are many problems that have not been solved. Most of the disease identification process needs to be uploaded to the computer to complete, which is very inconvenient for inspectors to operate on site. Design an application software that can identify underwater cracks using mobile devices. For example, Zhang Kaiyi proposed a mobile terminal corn disease detection application, Zhao Jianmin and others also proposed a potato leaf disease identification system based on neural networks, and He Yangyin proposed a bridge disease identification system and application solution. He Yangyin's research made the mobile terminal identification function more complete and specific. The above studies have all achieved mobile phone or mobile terminal disease identification, but there is little research on underwater crack disease identification application systems.

[0004] Some scholars have conducted extensive research on crack detection. Zhou Zhong and others began to use the latest YOLO series neural network to detect cracks and optimize them. Ghostnet was used to replace the CSPDarknet backbone network of the YOLO series network so that cracks can be identified and detected more quickly and accurately. Qiu Xinjie and others used underwater robots as a platform to detect cracks of different forms based on the optimized semantic segmentation algorithm to meet the requirements of real-time and accuracy.

[0005] The above studies have excellent accuracy and speed in crack detection, with some even reaching over 95% accuracy, and some studies can achieve real-time detection. However, there is a lack of a centralized functional mobile platform that can classify and detect underwater cracks, perform semantic segmentation and extract features, and measure data at the same time. As a result, crack detection cannot be performed in real time or remotely, and the detection results are of low accuracy and incomplete information. Summary of the invention

[0006] The present invention provides a deep learning-based bridge underwater crack detection method and a movable platform to overcome the above-mentioned technical problems.

[0007] A method for detecting underwater cracks in bridges based on deep learning, comprising:

[0008] Step 1: Preparation of crack test blocks, which includes, under a standard laboratory environment, placing a PE protective film in a mold, filling the mold with concrete to make a test block, and curing the test block.

[0009] Step 2: Underwater camera calibration and sealed cabin production. The underwater camera calibration is used to obtain a real coordinate system based on the underwater camera coordinate system and the image coordinate system, wherein the image coordinate system is established according to the captured underwater image, and the parameters of the underwater camera are recalibrated based on the real coordinate system. The parameters include internal parameters and external parameters, wherein the internal parameters include camera focal length, principal point coordinates and distortion parameters, wherein the external parameters include camera position and camera attitude. The underwater camera is a structured light camera. The sealed cabin production is used to produce an underwater sealed cabin, and the underwater camera is embedded in the underwater sealed cabin and the waterproofness is verified.

[0010] Step 3: underwater image acquisition and enhancement. The underwater image acquisition includes placing the cured test blocks in an underwater environment in turn, and obtaining a crack image set of the test blocks in the underwater environment according to the calibrated underwater camera in the underwater sealed cabin. The underwater image enhancement includes enhancing the contrast and clarity of the crack images in the crack image set based on the MSRCR image enhancement algorithm and the limited contrast adaptive histogram equalization method, and marking the enhanced crack image set.

[0011] Step 4: underwater image expansion, wherein the underwater image expansion includes constructing a deep convolutional generative adversarial network, optimizing the deep convolutional generative adversarial network, training the optimized deep convolutional generative adversarial network based on the annotated crack image set and obtaining a virtual underwater crack image set containing cracks, annotating the virtual underwater crack image set,

[0012] Step 5: underwater data set preparation, wherein the underwater data set preparation includes merging the annotated virtual underwater fracture image set into the annotated fracture image set and expressing them as the underwater data set.

[0013] Step 6, improve the yolov5 network, the improved yolov5 network includes adding an AF-FPN structure on the basis of the yolov5 network, the AF-FPN structure is to add a feature enhancement module and an adaptive attention module on the basis of the pyramid network, the feature enhancement module is used to enhance the feature pyramid, and the adaptive attention module is used to reduce the loss of context information in feature channels and high-level feature maps; according to the improved yolov5 network, the cracks of the underwater images in the underwater data set are detected and located, and the target area where the cracks are located is obtained,

[0014] Step 7: Improve the TransUNet network. The improved TransUNet network is to replace all standard convolutional layers in TransUNet with depthwise separable convolutional layers to form a DS-TransUNet network. The SeNet attention mechanism is introduced into the DS-TransUNet network to form a DS-SeNet-TransUNet network. The DS-SeNet-TransUNet network is trained based on the underwater dataset after crack detection and positioning.

[0015] Step 8: Based on the trained DS-SeNet-TransUNet network, identify and verify the target area where the crack is located and obtain the quantitative information of the crack, which includes the length, width, and tortuosity of the crack.

[0016] Step 9: Obtain the underwater image of the bridge to be detected, and detect the underwater image of the bridge to be detected based on the trained DS-SeNet-TransUNet network.

[0017] Preferably, the optimization of the deep convolutional generative adversarial network includes:

[0018] S1. Replace all pooling layers in the deep convolutional generative adversarial network with micro-step convolution layers.

[0019] S2, delete the fully connected layer in the deep convolutional generative adversarial network,

[0020] S3. The activation function of the last network layer in the deep convolutional generative adversarial network uses the tanh function, and the other network layers use the ReLU function as the activation function.

[0021] Preferably, the training of the DS-SeNet-TransUNet network includes training the DS-SeNet-TransUNet network according to a cosine annealing learning rate adjustment method.

[0022] Preferably, the trained DS-SeNet-TransUNet network can also be embedded in an underwater cable robot device, which has a lighting and ranging device. After enhancing the underwater image acquired by the underwater cable robot device, crack detection is performed according to the trained DS-SeNet-TransUNet network.

[0023] A mobile platform for underwater crack detection, including a client, a server, and a model deployment module.

[0024] The model deployment module is used to deploy the trained DS-SeNet-TransUNet network model. TensorFlow Serving is used as the model deployment system. Onnx is used to convert the DS-SeNet-TransUNet network model trained in the pytorch framework into tensorflow format and then deploy it on the cloud server.

[0025] The client is used for users to send requests and obtain response results sent by the server. The request uses JSON as the data interaction format and uses JavaScript technology to upload and preview images.

[0026] The server is used to receive user requests and respond to the results to the client. The server receives image data in Base64 format, uses the trained DS-SeNet-TransUNet network to perform recognition and feature segmentation, and returns the results to the client.

[0027] The present invention provides a method and a movable platform for detecting underwater cracks in bridges based on deep learning, optimizes an underwater image enhancement method, improves and optimizes a neural network for underwater crack detection classification and semantic segmentation to obtain a trained DS-SeNet-TransUNet network, improves the accuracy of underwater image crack detection and measurement, realizes a movable platform for underwater image crack detection, and can deploy the platform on various mobile electronic devices to perform real-time recognition and feature segmentation of underwater cracks in bridges, automatically recognize crack locations, perform semantic segmentation to extract features, and finally measure cracks. Compared with manual detection, this method can avoid detection personnel from going into the water in person, thus ensuring personal safety, and can greatly improve detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 is a flow chart of the method of the present invention;

[0030] Figure 2 It is the mold style of the present invention;

[0031] Figure 3 It is a test block after curing of the present invention;

[0032] Figure 4 It is a structural diagram of the underwater sealed cabin of the present invention;

[0033] Figure 5 This is a structural diagram of the underwater sealed cabin of the present invention;

[0034] Figure 6 This is a structural diagram of a fixing clamp in an underwater sealed cabin of the present invention;

[0035] Figure 7 It is a chessboard calibration plate of the present invention;

[0036] Figure 8 It is an underwater image taken by the present invention based on the chessboard calibration plate;

[0037] Fig. 9 is the underwater image after calibration of the present invention;

[0038] Fig.10 It is a histogram of pixel distribution of underwater image of the present invention;

[0039] Fig.11 is the underwater image after interpolation of the present invention;

[0040] Fig.12 It is the DCGAN training flow chart of the present invention;

[0041] Fig.13 It is the structure diagram of YOLOv5s of the present invention;

[0042] FIG. 14( a ) is an underwater image taken by the present invention;

[0043] FIG14( b ) is a crack image after identification and positioning by the present invention;

[0044] FIG14( c ) is a crack image after semantic segmentation of the present invention;

[0045] FIG. 15( a ) is a crack location diagram of the present invention;

[0046] FIG15( b ) is a diagram showing the result of image processing according to the present invention;

[0047] FIG15( c ) is a result diagram after semantic segmentation of the present invention;

[0048] FIG. 15( d ) is the crack quantification information of the present invention;

[0049] Fig.16 It is a structural diagram of the movable platform of the present invention;

[0050] Fig.17 It is a specific effect diagram of the system interface of the present invention;

[0051] Description of Figure Numbers:

[0052] 1. Underwater light; 2. Light bracket; 3. Sealed front cover; 4. Camera; 5. Camera bracket; 6. Sealed tail cover; 7. Line sealing tube; 8. Sealing rubber ring. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Figure 1 It is a flow chart of the method of the present invention, such as Figure 1 As shown, the method of this embodiment may include:

[0055] Step 1: Preparation of crack test blocks, which includes, under a standard laboratory environment, placing a PE protective film in a mold, filling the mold with concrete to make a test block, and curing the test block.

[0056] Step 2: Underwater camera calibration and sealed cabin production. The underwater camera calibration is used to obtain a real coordinate system based on the underwater camera coordinate system and the image coordinate system, wherein the image coordinate system is established according to the captured underwater image, and the parameters of the underwater camera are recalibrated based on the real coordinate system. The parameters include internal parameters and external parameters, wherein the internal parameters include camera focal length, principal point coordinates and distortion parameters, wherein the external parameters include camera position and camera attitude. The underwater camera is a structured light camera. The sealed cabin production is used to produce an underwater sealed cabin, and the underwater camera is embedded in the underwater sealed cabin and the waterproofness is verified.

[0057] Step 3: underwater image acquisition and enhancement. The underwater image acquisition includes placing the cured test blocks in an underwater environment in turn, and obtaining a crack image set of the test blocks in the underwater environment according to the calibrated underwater camera in the underwater sealed cabin. The underwater image enhancement includes enhancing the contrast and clarity of the crack images in the crack image set based on the MSRCR image enhancement algorithm and the limited contrast adaptive histogram equalization method, and marking the enhanced crack image set.

[0058] Step 4: underwater image expansion, wherein the underwater image expansion includes constructing a deep convolutional generative adversarial network, optimizing the deep convolutional generative adversarial network, training the optimized deep convolutional generative adversarial network based on the annotated crack image set and obtaining a virtual underwater crack image set containing cracks, annotating the virtual underwater crack image set,

[0059] Step 5: underwater data set preparation, wherein the underwater data set preparation includes merging the annotated virtual underwater fracture image set into the annotated fracture image set and expressing them as the underwater data set.

[0060] Step 6, improve the yolov5 network, the improved yolov5 network includes adding an AF-FPN structure on the basis of the yolov5 network, the AF-FPN structure is to add a feature enhancement module and an adaptive attention module on the basis of the pyramid network, the feature enhancement module is used to enhance the feature pyramid, and the adaptive attention module is used to reduce the loss of context information in feature channels and high-level feature maps; according to the improved yolov5 network, the cracks of the underwater images in the underwater data set are detected and located, and the target area where the cracks are located is obtained,

[0061] Step 7: Improve the TransUNet network. The improved TransUNet network is to replace all standard convolutional layers in TransUNet with depthwise separable convolutional layers to form a DS-TransUNet network. The SeNet attention mechanism is introduced into the DS-TransUNet network to form a DS-SeNet-TransUNet network. The DS-SeNet-TransUNet network is trained based on the underwater dataset after crack detection and positioning.

[0062] Step 8: Based on the trained DS-SeNet-TransUNet network, identify and verify the target area where the crack is located and obtain the quantitative information of the crack, which includes the length, width, and tortuosity of the crack.

[0063] Step 9: Obtain the underwater image of the bridge to be detected, and detect the underwater image of the bridge to be detected based on the trained DS-SeNet-TransUNet network.

[0064] Based on the above scheme, the present invention provides a method and a movable platform for underwater crack detection of bridges based on deep learning, optimizes the underwater image enhancement method, improves and optimizes the neural network for underwater crack detection classification and semantic segmentation to obtain a trained DS-SeNet-TransUNet network, improves the underwater image crack detection and measurement accuracy, realizes a movable platform for underwater image crack detection, and can deploy the platform on various mobile electronic devices to perform real-time recognition and feature segmentation of underwater cracks in bridges, automatically identify the crack location and perform semantic segmentation to extract features and finally measure the cracks. Compared with manual detection, this method can avoid the detection personnel from going into the water in person, which brings protection to personal safety and can greatly improve the detection efficiency.

[0065] Specifically, this embodiment provides the details of the crack detection method, including:

[0066] Step 1: Preparation of crack test blocks. The preparation of crack test blocks includes inserting a PE protective film into a mold under a standard laboratory environment. The mold is as follows: Figure 2 As shown, concrete is poured to make test blocks, and the test blocks are cured. Figure 3 As shown,

[0067] Step 2: Underwater camera calibration and sealed cabin production. The underwater camera calibration is used to obtain a real coordinate system based on the underwater camera coordinate system and the image coordinate system, wherein the image coordinate system is established according to the captured underwater image, and the parameters of the underwater camera are recalibrated based on the real coordinate system. The parameters include internal parameters and external parameters, wherein the internal parameters include camera focal length, principal point coordinates and distortion parameters, wherein the external parameters include camera position and camera attitude. The underwater camera is a structured light camera. The sealed cabin production is used to produce an underwater sealed cabin, embed the underwater camera into the underwater sealed cabin and perform waterproof verification.

[0068] Specifically, in order to facilitate image collection, a self-made sealed cabin camera method is used to capture underwater images under the premise of ensuring safety. A structured light camera is selected as the underwater image acquisition device, and the structured light camera is embedded in the manufactured underwater sealed cabin, such as Figure 4 As shown, 1, underwater light, 2, light bracket, 3, sealed front cover, 4, camera, 5, camera bracket, 6, sealed tail cover, 7, line sealing tube, 8, sealing rubber ring, its appearance is as follows Figure 5 As shown, and water resistance verified.

[0069] In order to meet the requirements of compressive strength, impact resistance and light transmittance, the sealed cabin is made of acrylic board. In order to maintain good sealing, the structural design minimizes the components, and takes simplicity and practicality as the main design core elements. The main structure is divided into a sealed front cover and a sealed rear cover. The internal design clip is used to fix the camera, such as Figure 6 As shown, the interface is designed in the sealed tail cover, and the line sealing tube is used to strengthen the sealing. A light bracket is set in the front to fix the underwater searchlight to prevent the environment from being too dark to identify objects. After the sealed cabin is fixed with bolts, sealant is poured and applied to each interface.

[0070] In order to overcome the influence of underwater water quality, light refraction and other factors to ensure better underwater image quality, the camera parameters must be recalibrated. The real coordinate system is converted into the captured pixel coordinate system through the camera coordinate system and the image coordinate system. The conversion formula is as follows:

[0071]

[0072]

[0073] (Xw, Yw, Zw) represents the coordinates of a point in the world coordinate system in three-dimensional space, f x =f / dx,f y =f / dy, where f is the focal length of the camera. c is the Z-axis coordinate of the point in the camera coordinate system, u and v represent the x-axis and y-axis coordinates of the point. The leftmost side shows the coordinates in the camera coordinate system in matrix form; the physical coordinates of the imaging point are (u0, v0), where dx and dy are the physical width and height corresponding to each pixel of the imaging element, the unit is generally mm, both are intrinsic parameters of the camera, and f is the focal length of the camera. R and T are 3×3 rotation matrices and 3×1 translation vectors, respectively.

[0074] The specific calibration process is as follows:

[0075] 1. Select a checkerboard calibration plate, such as Figure 7 shown.

[0076] 2. Prepare the camera: adjust the camera underwater to ensure that the camera position and posture are fixed and remain stable.

[0077] 3. Capture calibration images: Use a camera to capture the checkerboard calibration plate, keep the camera position and posture unchanged, and capture images at multiple angles and depths, such as Figure 8 As shown in the figure, due to refraction and other reasons, the image taken in normal water will be deformed, so by shooting a calibration plate with known points, the deformation is calculated and the image point error is inferred.

[0078] 4. Extract corner points: Import the captured calibration image into the computer and use image processing software to extract the corner points of the calibration plate. The corner point positions of each black and white image library are all corner points. This part is completed by computer.

[0079] 5. Calculate internal parameters: Use the corner point coordinate data and the camera calibration algorithm to calculate the internal parameter matrix, including the camera's focal length, principal point coordinates, and distortion parameters.

[0080] Left camera intrinsic parameter matrix:

[0081]

[0082] Distortion parameter matrix:

[0083]

[0084] Right camera intrinsic parameter matrix:

[0085]

[0086] Distortion parameter matrix:

[0087]

[0088] The left and right camera rotation matrices are as follows:

[0089]

[0090] Translation matrix:

[0091] T=[-59.7716 -0.0460 -0.3189]

[0092] Color camera calibration, the internal parameters of the color camera are as follows:

[0093]

[0094] Distortion parameter matrix:

[0095]

[0096] 6. Calculate external parameters: Use the camera calibration algorithm to calculate the camera's external parameters, including the camera's position and posture.

[0097] 7. Test calibration effect: Use test images to verify the calibration effect and ensure the accuracy and stability of the calibration results. Fig. 9 shown.

[0098] Step 3: underwater image acquisition and enhancement. The underwater image acquisition includes placing the cured test blocks in an underwater environment in turn, and obtaining a crack image set of the test blocks in the underwater environment according to the calibrated underwater camera in the underwater sealed cabin. The underwater image enhancement includes enhancing the contrast and clarity of the crack images in the crack image set based on the MSRCR image enhancement algorithm and the limited contrast adaptive histogram equalization method, and marking the enhanced crack image set.

[0099] Specifically, the test blocks are placed in an underwater environment in turn, and the calibrated camera is used to obtain crack images underwater to obtain the data set. The underwater crack images have the characteristics of low contrast, blue-green color, blur, etc., which are not conducive to the network to extract the detailed features of the image, resulting in low detection accuracy of the trained model. To solve the above problems, this paper adopts image enhancement algorithm to improve the quality of underwater images and improve the detection accuracy of the training model.

[0100] The images of the crack image set are enhanced. A contrast-limited adaptive histogram equalization (CLAHE) method is introduced based on the MSRCR image enhancement algorithm to improve the contrast and clarity of underwater images.

[0101] The specific process is as follows: the grayscale area of ​​the acquired underwater image is evenly distributed, and the result is presented as Fig.10 The bar graph on the right; set a critical value and then calculate the histogram in each area; segment the image area exceeding the critical value and evenly distribute it to the entire underwater image; perform grayscale averaging on each area of ​​the segmented image; finally, refer to the various areas around a certain pixel point and interpolate the point. The differenced image is as follows: Fig.11 shown.

[0102] Step 4: Underwater image expansion, the underwater image expansion includes constructing a deep convolutional generative adversarial network and optimizing the deep convolutional generative adversarial network. The optimization of the deep convolutional generative adversarial network includes:

[0103] S1. Replace all pooling layers in the deep convolutional generative adversarial network with micro-step convolution layers.

[0104] S2, delete the fully connected layer in the deep convolutional generative adversarial network,

[0105] S3. The activation function of the last network layer in the deep convolutional generative adversarial network is the tanh function, and the other network layers use the ReLU function as the activation function.

[0106] Based on the crack image set, the optimized deep convolutional generative adversarial network is trained to obtain a virtual underwater crack image set containing cracks, and the virtual underwater crack image set is annotated.

[0107] Specifically, the deep convolutional generative adversarial network (DCGAN) was built and optimized based on the original convolutional generative network GAN to expand the crack image set samples. The optimization contents are as follows:

[0108] (1) Use micro-step convolution to replace the pooling layer in the original network and complete the upsampling operation, so that the width and height of the feature layer continue to increase.

[0109] (2) Add batch normalization to the network layer to avoid network collapse and improve network stability. At the same time, it makes the gradient descent more stable and avoids the probability of gradient vanishing.

[0110] (3) Delete the fully connected layer in the network to deepen the model layer.

[0111] (4) Except for the last network layer, which uses the tanh function as the activation function, the remaining network layers all use the ReLU function as the activation function.

[0112] The generation and discrimination learning of DCGAN generative adversarial network is a binary extreme value problem, and its formula is as follows:

[0113]

[0114] The objective function of DCGAN consists of the objective function of the generative network and the generative function of the discriminative network.

[0115] In the above formula, x represents the real underwater crack sample input to the network;

[0116] z represents the random noise input into the network;

[0117] D(x) represents the probability that the discriminant network considers x to be a real crack;

[0118] G(z) represents the virtual underwater crack sample after random noise is generated by inputting into the generation network;

[0119] P data (x) represents the distribution of real underwater cracks in the entire sample;

[0120] P z (z) represents the distribution of virtual underwater cracks generated by noise;

[0121] D(G(z)) represents the probability that the discriminant network considers the virtual sample to be a real sample.

[0122] The specific training process is as follows:

[0123] (1) Noise is input into the generative network, and the generative network combines the noise with the learned crack sample information. The combination is to superimpose the input noise vector with the crack sample information vector to generate a virtual underwater crack image.

[0124] (2) The generated virtual image and the real underwater crack image are input into the discriminant network for judgment. The discriminant network outputs a probability value and back-propagates it to the generative network.

[0125] (3) The generative network and the discriminative network compare the probability values ​​with the authenticity of the underwater crack samples and continuously optimize them. The generative network will learn more features of real underwater cracks, thereby producing more realistic underwater crack photos. At the same time, the discriminative network is also constantly improving its judgment ability.

[0126] (4) After continuous optimization, the generating network and the discriminating network reach convergence, and the network training is completed. The training flow chart is as follows: Fig.12 shown.

[0127] Step 5: underwater data set preparation, wherein the underwater data set preparation includes adding the annotated virtual underwater crack image set to the annotated crack image set and expressing them as the underwater data set.

[0128] Step 6: Improve the yolov5 network. The improved yolov5 network includes adding an AF-FPN structure on the basis of the yolov5 network. The AF-FPN structure is to add a feature enhancement module and an adaptive attention module on the basis of the pyramid network. The feature enhancement module is used to enhance the feature pyramid. The adaptive attention module is used to reduce the loss of contextual information in feature channels and high-level feature maps. According to the improved yolov5 network, the cracks of the underwater images in the underwater data set are detected and located, and the target area where the cracks are located is obtained.

[0129] Specifically, an underwater crack detection network based on YOLOv5s and TransUNet was built. The two networks were improved and combined to obtain a network system that combines the dual functions of image classification and semantic segmentation. The specific improvements are as follows:

[0130] The AF-FPN structure is introduced based on YOLOv5s to improve its recognition accuracy of underwater cracks and balance without affecting the recognition speed. The structure is as follows Fig.13As shown. The AF-FPN structure adds a feature enhancement module FEM and an adaptive attention module AAM to the pyramid network to reduce the degree of information loss in image data and improve reasoning ability. By performing multiple convolutions on the input image to generate feature maps, M6 and M5 are summed and propagated from top to bottom to fuse with other features at lower levels, ultimately achieving optimization of network performance and speed. The improved yoloV5 network structure is shown below. The network accuracy has increased by 10%, the detection rate has also increased to 130 frames per second, and the accuracy of crack recognition has increased significantly.

[0131] Combine YOLOv5s with DS-SeNet-TransUNet, use YOLOv5s for underwater crack detection and positioning, then save the detected target area, and finally input the saved content into the DS-SeNet-TransUNet model for segmentation and recognition.

[0132] Step 7: Improve the TransUNet network. The improved TransUNet network is to replace all standard convolutional layers in TransUNet with depth-separable convolutional layers to form a DS-TransUNet network. The SeNet attention mechanism is introduced into the DS-TransUNet network to form a DS-SeNet-TransUNet network. The SeNet attention mechanism includes a compression stage, an excitation stage, and a reweighting stage.

[0133] The compression stage is used to convert the shape of the input image from CxHxW to Cx1x1, and then perform a global average pooling operation on each channel to output the global features of each channel, that is, a vector of length C, where C is the channel and also represents the number of convolution kernels. H and W represent the height and width of the input image.

[0134] The excitation stage is used to map the global features of each channel through two fully connected layers to obtain a vector of length C, wherein the first fully connected layer maps the global features of the channel to an intermediate dimension and performs nonlinear transformation through the ReLU activation function; the second fully connected layer maps the features of the intermediate dimension back to the number of channels and obtains the weight coefficient of the channel through the Sigmoid function.

[0135] The reweighting stage is used to multiply the channel weight coefficient by each channel in the input image, and then output the reweighted channel.

[0136] The DS-SeNet-TransUNet network is trained. The training of the DS-SeNet-TransUNet network includes training the DS-SeNet-TransUNet network according to the cosine annealing learning rate adjustment method. The principle of the cosine annealing formula is as follows:

[0137]

[0138] Among them, η t represents the current learning rate, η min Represents the minimum learning rate, η max Represents the maximum learning rate, T cur represents the current round, T max Represents the maximum number of rounds.

[0139] Specifically, TransUNet is a lightweight image segmentation neural network based on Transformer. The network structure adopts the Transformer Encoder-Decoder architecture and self-attention mechanism, and uses the pre-trained Transformer language model to improve the efficiency and accuracy of the network. In TransUNet, the input image is first sent to the Transformer Encoder, from which a series of features are extracted, which are crack features in the image. These features are then sent to the Transformer Decoder for decoding to generate an output segmentation image of the same size as the input image. Improvement measures: In TransUNet, all standard convolutional layers are replaced with depthwise separable convolutional layers to form a DS-TransUNet network. When the same image data is input, the number of calculation parameters of ordinary convolution is 108, while the number of total calculation parameters after adopting depthwise separable convolution is 39, which greatly reduces the number of parameters and improves the calculation efficiency. The SeNet attention mechanism is introduced in the improved DS-TransUNet network. The final neural network (DS-SeNet-TransUNet) was then formed, which not only has a huge advantage in speed but also has a slight improvement in recognition accuracy for underwater crack data sets.

[0140] Step 8: Based on the trained DS-SeNet-TransUNet network, the target area where the crack is located is identified and verified, and the quantitative information of the crack is obtained. The quantitative information includes the length, width, and tortuosity of the crack. Specifically, the trained network can not only quickly identify the target to prevent missed detection, but also display the details of the crack to the inspector in more detail through semantic segmentation, and quantify the crack information. The combined application effect diagram of crack identification is shown in Figures 14(a), (b), and (c). Figure 14(a) is an underwater image taken, Figure 14(b) is a crack image after identification and positioning, and Figure 14(c) is a crack image after semantic segmentation.

[0141] Step 9: Obtain the underwater image of the bridge to be detected, and detect the underwater image of the bridge to be detected based on the trained DS-SeNet-TransUNet network.

[0142] The trained DS-SeNet-TransUNet network can also be embedded in an underwater cable robot device, which has a lighting and ranging device. After enhancing the underwater image acquired by the underwater cable robot device, crack detection is performed according to the trained DS-SeNet-TransUNet network. Specifically, the algorithm is embedded in the underwater cable robot device, which has a lighting and ranging device. When performing actual crack detection, the image detected to contain cracks is enhanced, and the skeleton is extracted after binarization of the processed image. The skeleton is the crack information, and pixel statistics are automatically calculated. The curve fitting method, i.e., the least squares fitting, is used to calculate the length and width of the crack. The processing results of the underwater cable robot device are shown in Figures 15(a), (b), (c), and (d), respectively.

[0143] A mobile platform for underwater crack detection, including a client, a server, and a model deployment module.

[0144] The model deployment module is used to deploy the trained DS-SeNet-TransUNet network model. TensorFlow Serving is used as the model deployment system. Onnx is used to convert the DS-SeNet-TransUNet network model trained in the pytorch framework into tensorflow format and then deploy it on the cloud server.

[0145] The client is used for users to send requests and obtain response results sent by the server. The request uses JSON as the data interaction format and uses JavaScript technology to upload and preview images.

[0146] The server is used to receive user requests and respond to the results to the client. The server receives image data in Base64 format, uses the trained DS-SeNet-TransUNet network to perform recognition and feature segmentation, and returns the results to the client.

[0147] Specifically, the user end implements image upload and preview functions, supports local album selection, and can be packaged into Android and iOS applications for download and installation. The server receives image data in Base64 format, uses the trained model for recognition and feature segmentation, and returns the results. The MySQL database stores the image data uploaded by the user. The server adds and queries data by connecting to the database and returns it to the user end in JSON format. At the same time, the image processing and crack measurement functions are embedded in the platform to make the platform more complete. The specific structure is as follows: Fig.16 shown.

[0148] The system uses front-end technologies (HTML, CSS, etc.) to design the user interface and JavaScript technology to implement image uploading, data rendering, and data interaction. Pycharm is used for design and training to implement underwater crack image recognition and segmentation models. The Flask framework is used to provide an image recognition interface and the backend application is deployed on a cloud server to meet the application requirements on mobile lightweight devices. The specific effects are as follows: Fig.17 shown.

[0149] Overall beneficial effects:

[0150] The present invention provides a method and a movable platform for detecting underwater cracks in bridges based on deep learning, optimizes an underwater image enhancement method, improves and optimizes a neural network for underwater crack detection classification and semantic segmentation to obtain a trained DS-SeNet-TransUNet network, improves the underwater image crack detection and measurement accuracy, realizes the construction of a movable platform for underwater image crack detection, can deploy the platform on various mobile electronic devices, perform real-time recognition and feature segmentation of underwater cracks in bridges, automatically recognizes the crack location, performs semantic segmentation to extract features, and finally measures the cracks. Compared with manual detection, this method can avoid detection personnel from going into the water in person, thus ensuring personal safety, and can greatly improve detection efficiency.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting underwater cracks in bridges based on deep learning, characterized in that: include, Step 1: Preparation of crack test blocks, which includes, under a standard laboratory environment, placing a PE protective film in a mold, filling the mold with concrete to make a test block, and curing the test block. Step 2: Underwater camera calibration and sealed cabin production. The underwater camera calibration is used to obtain a real coordinate system based on the underwater camera coordinate system and the image coordinate system, wherein the image coordinate system is established according to the captured underwater image, and the parameters of the underwater camera are recalibrated based on the real coordinate system. The parameters include internal parameters and external parameters, wherein the internal parameters include camera focal length, principal point coordinates and distortion parameters, wherein the external parameters include camera position and camera attitude. The underwater camera is a structured light camera. The sealed cabin production is used to produce an underwater sealed cabin, and the underwater camera is embedded in the underwater sealed cabin and the waterproofness is verified. Step 3: underwater image acquisition and enhancement. The underwater image acquisition includes placing the cured test blocks in an underwater environment in turn, and obtaining a crack image set of the test blocks in the underwater environment according to the calibrated underwater camera in the underwater sealed cabin. The underwater image enhancement includes enhancing the contrast and clarity of the crack images in the crack image set based on the MSRCR image enhancement algorithm and the limited contrast adaptive histogram equalization method, and marking the enhanced crack image set. Step 4: underwater image expansion, wherein the underwater image expansion includes constructing a deep convolutional generative adversarial network, optimizing the deep convolutional generative adversarial network, training the optimized deep convolutional generative adversarial network based on the annotated crack image set and obtaining a virtual underwater crack image set containing cracks, annotating the virtual underwater crack image set, Step 5: underwater data set preparation, wherein the underwater data set preparation includes merging the annotated virtual underwater fracture image set into the annotated fracture image set and expressing them as the underwater data set. Step 6, improve the yolov5 network, the improved yolov5 network includes adding an AF-FPN structure on the basis of the yolov5 network, the AF-FPN structure is to add a feature enhancement module and an adaptive attention module on the basis of the pyramid network, the feature enhancement module is used to enhance the feature pyramid, and the adaptive attention module is used to reduce the loss of context information in feature channels and high-level feature maps; according to the improved yolov5 network, the cracks of the underwater images in the underwater data set are detected and located, and the target area where the cracks are located is obtained, Step 7: Improve the TransUNet network. The improved TransUNet network is to replace all standard convolutional layers in TransUNet with depthwise separable convolutional layers to form a DS-TransUNet network. The SeNet attention mechanism is introduced into the DS-TransUNet network to form a DS-SeNet-TransUNet network. The DS-SeNet-TransUNet network is trained based on the underwater dataset after crack detection and positioning. Step 8: Based on the trained DS-SeNet-TransUNet network, identify and verify the target area where the crack is located and obtain the quantitative information of the crack, which includes the length, width, and tortuosity of the crack. Step 9: Obtain the underwater image of the bridge to be detected, and detect the underwater image of the bridge to be detected based on the trained DS-SeNet-TransUNet network.

2. The method for detecting underwater cracks in bridges based on deep learning according to claim 1, characterized in that: The optimization of the deep convolutional generative adversarial network includes S1. Replace all pooling layers in the deep convolutional generative adversarial network with micro-step convolution layers. S2, delete the fully connected layer in the deep convolutional generative adversarial network, S3. The activation function of the last network layer in the deep convolutional generative adversarial network uses the tanh function, and the other network layers use the ReLU function as the activation function.

3. The method for detecting underwater cracks in bridges based on deep learning according to claim 1, characterized in that: The training of the DS-SeNet-TransUNet network includes training the DS-SeNet-TransUNet network according to a cosine annealing learning rate adjustment method.

4. The method for detecting underwater cracks in bridges based on deep learning according to claim 1, characterized in that: The trained DS-SeNet-TransUNet network can also be embedded in an underwater cable robot device, which has a lighting and ranging device. After enhancing the underwater image acquired by the underwater cable robot device, crack detection is performed based on the trained DS-SeNet-TransUNet network.

5. A movable platform for underwater crack detection using the underwater crack detection method for bridges according to claim 1, characterized in that: Including client, server, and model deployment modules, The model deployment module is used to deploy the trained DS-SeNet-TransUNet network model. TensorFlowServing is used as the model deployment system. Onnx is used to convert the DS-SeNet-TransUNet network model trained in the pytorch framework into tensorflow format and then deploy it on the cloud server. The client is used for users to send requests and obtain response results sent by the server. The request uses JSON as the data interaction format and uses JavaScript technology to upload and preview images. The server is used to receive user requests and respond to the results to the client. The server receives image data in Base64 format, uses the trained DS-SeNet-TransUNet network to perform recognition and feature segmentation, and returns the results to the client.

Images