Underwater Pier Rapid Detection Method and System Based on Sonar Camera and Deep Learning

Abstract

The present invention discloses a method and system for rapid underwater pier detection based on sonar cameras and deep learning, which relates to the technical field of underwater bridge detection. The present invention includes: based on an improved YOLOv7 model, receiving sonar images containing multiple piers and their surface defects to achieve rapid underwater detection of piers and defects; combining the bounding box coordinate information obtained by the improved YOLOv7 model with sonar distance parameters to obtain the actual distance information data of the defects relative to the piers, realizing underwater positioning of the defects; quickly moving the underwater camera close to the pier defect position for image acquisition according to the obtained distance information data; inputting the collected pier defect images into the improved DeepLabv3+ model to output the recognition results of the defects. Compared with the traditional method of manually probing each pier one by one, the present invention can achieve rapid and accurate detection of underwater pier defects, not only greatly improving the detection accuracy, but also enhancing the detection efficiency and saving human resources.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater bridge detection, and specifically to a method and system for rapid underwater bridge pier detection based on sonar camera and deep learning. Background Technique

[0002] The number of bridges in China exceeds 1 million, and their safe operation and maintenance are basic needs. Currently, the detection of bridges mostly focuses on the structures exposed in the air, ignoring the hidden underwater structures. The existing detection of underwater bridge pier defects mainly relies on divers carrying more than a hundred catties of equipment to explore the underwater bridge piers of the bridge, and then the detection personnel take pictures and observe the defects to judge the type and severity of the defects. This method depends on the subjectivity of the detection personnel and has a low accuracy. In addition, manual detection requires divers to explore the defects of each bridge pier in a fast-flowing river, with low intelligence, long detection time, high cost and great risk. We propose a method and system for rapid underwater bridge pier detection based on sonar and deep learning. Summary of the Invention

[0003] The purpose of the present invention is to provide a method and system for rapid underwater bridge pier detection based on sonar and deep learning, which can achieve rapid positioning and accurate identification of underwater bridge pier defects.

[0004] To achieve the above purpose, the present invention provides the following technical solutions: A method for rapid underwater bridge pier detection based on sonar and deep learning includes the following steps:

[0005] Based on an improved YOLOv7 model, receive sonar images containing multiple bridge piers and their surface defects, and achieve rapid underwater detection of bridge piers and defects;

[0006] Combine the bounding box coordinate information obtained by the improved YOLOv7 model with sonar distance parameters to obtain the actual distance information data of the defects relative to the bridge piers;

[0007] According to the obtained distance information data, collect images of the bridge pier defect positions;

[0008] Input the collected bridge pier defect images into the improved DeepLabv3+ model, and output the recognition results of the defects.

[0009] Furthermore, the sonar is specifically set as a mechanical scanning sonar, and a two-dimensional image of the underwater bridge pier and its surface defects is obtained by using the sonar scanning method.

[0010] Furthermore, the improved YOLOv7 model introduces Swin-Transformer (Swin-T) in the backbone and introduces a spatial-channel attention mechanism (CBAM) in the neck.

[0011] Further, in the improved YOLOv7 model, when the feature map is input into the backbone, it no longer passes through the convolutional layer, but through the patch partition layer and two pairs of Swin-T modules, and the two pairs of Swin-T modules are connected by the patch merging layer.

[0012] Further, in the improved YOLOv7, CBAM is connected to the three RepConv layers and one upsampling layer in the head layer of YOLOv7, so as to further integrate the features passed into the head.

[0013] Further, the bounding box coordinate information obtained by the improved YOLOv7 model is combined with the sonar distance parameter to obtain the actual distance information data of the defect relative to the pier, specifically as follows:

[0014] After obtaining the category of each target and the coordinate information of the bounding box, select the category with a target confidence greater than 0.6 and its coordinate information, convert the exported YOLO coordinate format to the easily computable PASCAL_VOC coordinate format, convert the bounding box coordinates output by the model from the ratio relative to the image to the actual pixel coordinates, and combine the scanning range parameter during sonar operation and the pixel distance in the image to obtain the relative position of the actual underwater pier and the defect.

[0015] Further, an industrial camera is used to collect images of the pier defect location, and the model of the industrial camera is set to Baumer TXG20-I7.

[0016] Further, in the improved DeepLabv3+ model, in the original DeepLabv3+ semantic segmentation model, after cascading and fusing the shallow feature map and the upsampled feature map, a branch of continuous convolution and adaptive average pooling is introduced.

[0017] According to one aspect of the present invention, the present invention provides an underwater pier rapid detection system based on sonar camera and deep learning, including:

[0018] Receiving module: used to receive the sonar image input containing multiple piers and their surface defects based on the improved YOLOv7 model, and realize the rapid underwater detection of piers and defects;

[0019] Distance information acquisition module: used to combine the bounding box coordinate information obtained by the improved YOLOv7 model with the sonar distance parameter to obtain the actual distance information data of the defect relative to the pier;

[0020] Image acquisition module: used to collect images of the pier defect location according to the obtained distance information data;

[0021] Defect Output Module: It is used to input the collected pier defect images into the improved DeepLabv3+ model and output the recognition results of the defects.

[0022] Furthermore, the receiving module includes:

[0023] Acquisition Unit: It is used to acquire multiple piers and their surface defect images, specifically set as a mechanical scanning sonar;

[0024] Analysis Unit: It is used to analyze the image information data elements;

[0025] Transmission Unit: It is used to send the analyzed image information to the improved YOLOv7 model.

[0026] The present invention at least has the following beneficial effects:

[0027] 1. The present invention proposes a rapid detection model for underwater piers and defects of bridges, which improves the detection accuracy of piers and defects in sonar images.

[0028] 2. The present invention proposes a refined recognition model for underwater pier defects of bridges, which can well recognize various defects of actual underwater piers, facilitating the improvement of the practicality of the device.

[0029] 3. In the present invention, sonar images are used to assist in defect positioning, which can greatly improve the detection efficiency compared with the traditional manual probing of each pier. Description of the Drawings

[0030] Figure 1 is the flow chart of the method described in the present invention;

[0031] Figure 2 is the diagram of the improved YOLOv7 model described in the present invention;

[0032] Figure 3 is the diagram of the improved DeepLabv3+ model described in the present invention;

[0033] Figure 4 is the diagram of the detection and positioning results of underwater pier defects by the improved YOLOv7 described in the present invention.

[0034] Figure 5 is the diagram of the recognition results of underwater pier defects by the improved DeepLabv3+ described in the present invention. Detailed Embodiments

[0035] Next, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present disclosure.

[0036] Please refer to Figures 1-5 , the present invention provides a technical solution: a method for quickly detecting underwater bridge piers based on a sonar camera and deep learning, including the following steps:

[0037] S1. Obtain a sonar image containing multiple bridge piers and defects and input it into an improved YOLOv7 model to achieve fast underwater detection of bridge piers and defects;

[0038] The sonar is specifically set as a mechanical scanning sonar, and a two-dimensional image of the underwater bridge pier and its surface defects is obtained by using the sonar scanning method;

[0039] S2. Combine the bounding box coordinate information obtained by the improved YOLOv7 model with the sonar distance parameter to obtain the actual distance of the defect relative to the bridge pier, specifically as follows:

[0040] After obtaining the category and coordinate information of each target, select the category with a target confidence greater than 0.6 and its coordinate information, convert the exported YOLO coordinate format to the easily computable PASCAL_VOC coordinate format, convert the bounding box coordinates output by the model from the ratio relative to the image to the actual pixel coordinates, and combine the scanning range parameter during sonar operation and the pixel distance in the image to obtain the relative position of the actual underwater bridge pier and the defect;

[0041] Regarding the technical solution of this embodiment, as Figure 2 shown, the improved YOLOv7 object detection model is mainly used for the fast detection and positioning of underwater bridge piers and defects in sonar images; it mainly includes the YOLOv7 model, Swin-Transformer, and spatial-channel attention mechanism (CBAM), and the latter two are used to enhance the network's capture of the distribution relationship between targets and highlight the region of interest;

[0042] Improve the YOLOv7 architecture by combining Swin-T and CBAM: When the feature map enters the backbone, it no longer passes through the convolutional layer, but through the patch partition layer and two pairs of Swin-T modules, which are connected by the patch merging layer; Swin-T introduces the ability to further extract the low-level features of the original sonar image in the first few layers of the backbone, promoting the network to learn the relationship between the pixels of the measured target; in addition, connect CBAM to the three RepConv layers and one upsampling layer in the Head to further integrate the features passing into the Head; the improvement of YOLOv7 makes full use of the characteristics of Swin-T and CBAM and retains the key modules of the original YOLOv7 model, which can further improve the detection and localization accuracy of underwater bridge piers and defects.

[0043] S3. According to the distance information, move the camera closer to the defect location for image acquisition;

[0044] The camera is specifically an industrial camera, and the model is set to Baumer TXG20-I7;

[0045] S4. Input the collected bridge pier defect images into the improved DeepLabv3+ model and output the recognition results of the defects;

[0046] Regarding the technical solution of this embodiment, as Figure 3 shown, the improved DeepLabv3+ semantic segmentation model is mainly used for the recognition and segmentation of underwater bridge pier defects in optical images: in the encoder part, select Xception with fewer parameters and stronger feature extraction ability as the backbone for DeepLabv3+ to extract defect features, and transfer the extracted deep defect feature information to the Atrous Spatial Pyramid Pooling (ASPP) module;

[0047] In the decoder part, DeepLabv3+ uses 1×1 convolution to adjust the channels of the shallow feature map output by the backbone network, and at the same time uses bilinear interpolation to upsample the multi-scale high-level features output by the encoder, then cascade and fuse the two feature maps, and use 3×3 convolution to optimize the features;

[0048] To more effectively utilize the encoder to extract image semantic information and better restore the details and resolution of the image, the improvement of the DeepLabv3+ model mainly focuses on the decoder part. After cascading and fusing the shallow feature map and the upsampled feature map, introduce branches of continuous convolution and adaptive average pooling to further extract and transform the features; and weight the image features output by the convolution with the image weights generated by the adaptive average pooling to help integrate the defect feature information of different scales and shapes and improve the model's ability to restore details.

[0049] As Figure 4 This is an exemplary embodiment of the detection and positioning results of underwater bridge piers and defects in sonar images using the method proposed in this invention. The position information detected by combining the target detection algorithm and sonar distance information can accurately reflect the position of the defects underwater, thereby guiding manual or underwater ROVs equipped with underwater cameras for defect identification.

[0050] As Figure 5 This is an exemplary embodiment of the recognition and segmentation results of underwater bridge pier defects in optical images using the method proposed in this invention. The proposed improved model has strong accuracy, readability, and robustness when recognizing either single or multi-type defects, and has good application prospects in actual underwater bridge pier defect segmentation and recognition tasks.

[0051] According to one aspect of this invention, this invention provides a rapid underwater bridge pier detection system based on a sonar camera and deep learning, including:

[0052] A receiving module: configured to receive the input of a sonar image containing multiple bridge piers and their surface defects based on an improved YOLOv7 model, and achieve rapid underwater detection of bridge piers and defects;

[0053] A distance information acquisition module: configured to combine the bounding box coordinate information obtained by the improved YOLOv7 model with sonar distance parameters to obtain the actual distance information data of the defects relative to the bridge piers;

[0054] An image acquisition module: configured to collect images of the positions of bridge pier defects according to the obtained distance information data;

[0055] A defect output module: configured to input the collected bridge pier defect images into an improved DeepLabv3+ model and output the recognition results of the defects.

[0056] Furthermore, the receiving module includes:

[0057] An acquisition unit: configured to acquire images of multiple bridge piers and their surface defects, specifically set as a mechanical scanning sonar;

[0058] An analysis unit: configured to analyze the image information data elements;

[0059] A sending unit: configured to send the analyzed image information to the improved YOLOv7 model.

[0060] Specifically, the above receiving module, distance information acquisition module, image acquisition module, and defect output module can be embedded in a computer processing system. The computer, based on the above-provided rapid underwater bridge pier detection method using sonar and deep learning, calls the above modules to complete the task of equivalent modeling; the above receiving module, distance information acquisition module, image acquisition module, and defect output module can perform operations according to the specific steps given by the above-mentioned rapid underwater bridge pier detection method using sonar and deep learning.

[0061] It should be noted that the division of each module of the above system is only a logical function division. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by a processing element; they can also all be implemented in hardware form; it is also possible that some modules are implemented in the form of software called by a processing element, and some modules are implemented in hardware form. For example, the image noise removal module can be a separately established processing element, or can be integrated in a certain chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and the function of the above signal processing module is called and executed by a certain processing element of the above device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together or can be independently implemented. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through the hardware integrated logic circuit or software-form instructions in the processor element. [[ID=…]]

[0062] For example, the above modules can be one or more integrated circuits configured to implement the above method, such as: one or more application specific integrated circuits (ASICs), or, one or more microprocessors (DSPs), or, one or more field programmable gate arrays (FPGAs), etc. Again, when a certain module above is implemented in the form of a program code scheduled by a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program code. Again, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0063] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device.

[0064] For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. When an element is referred to as being "assembled on", "mounted on", "fixed to" or "disposed on" another element, it can be directly on the other element or there may also be an intermediate element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be an intermediate element at the same time. The terms "vertical", "horizontal", "upper", "lower", "left", "right" and similar expressions used herein are for illustrative purposes only and do not represent the only implementation.

[0065] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

[0066] In the description of this specification, the description referring to terms such as "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

Claims

1. A rapid detection method for underwater bridge piers based on sonar cameras and deep learning, characterized by: The following steps are involved: Based on the improved YOLOv7 model, it receives sonar images containing multiple bridge piers and their surface defects, enabling rapid underwater detection of bridge piers and defects. The bounding box coordinate information obtained by the improved YOLOv7 model is combined with the sonar distance parameters to obtain the actual distance information data of the defect relative to the bridge pier, realizing the underwater location of the defect. Based on the distance information data obtained, the underwater camera is quickly moved close to the pier defect location to collect images; The collected pier defect images are input into the improved DeepLabv3+ model to output the defect recognition results; The improved YOLOv7 model introduces Swin-Transformer (Swin-T) in the backbone and spatial-channel attention mechanism (CBAM) in the neck; In the improved YOLOv7 model, when the feature map is input into the backbone, it no longer passes through the convolution layer, but passes through the patch partitioning layer and two pairs of Swin-T modules. The two pairs of Swin-T modules are connected by a patch merging layer. The improved YOLOv7 connects CBAM with three RepConv layers and one upsampling layer in the head layer of YOLOv7 to further integrate the features passed into the head; The improved DeepLabv3+ model introduces branches of continuous convolution and adaptive average pooling after concatenating shallow feature maps with upsampled feature maps in the original DeepLabv3+ semantic segmentation model.

2. The method for rapid underwater bridge pier detection based on sonar camera and deep learning according to claim 1 is characterized by: The sonar is specifically configured as a mechanical scanning sonar, which uses a sonar scanning method to obtain two-dimensional images of underwater bridge piers and surface defects thereof.

3. The method for rapid underwater bridge pier detection based on sonar camera and deep learning according to claim 2 is characterized in that: The bounding box coordinate information obtained by the improved YOLOv7 model is combined with the sonar distance parameters to obtain the actual distance information data of the defect relative to the bridge pier, as shown below: After obtaining the category and bounding box coordinate information of each target, the category and coordinate information of the target with a confidence level greater than 0.6 are selected. The exported YOLO coordinate format is converted into the easy-to-calculate PASCAL_VOC coordinate format. The bounding box coordinates output by the model are converted from the scale relative to the image to actual pixel coordinates. The relative position of the actual underwater bridge pier and the defect is obtained by combining the scanning range parameters during sonar operation and the pixel distance in the image.

4. The method for rapid underwater bridge pier detection based on sonar camera and deep learning according to claim 3 is characterized by: An underwater industrial camera is used to capture images of the defective locations of the bridge piers. The model of the industrial camera is Baumer TXG20-I7.

5. A rapid underwater bridge pier detection system based on sonar camera and deep learning, used to implement the rapid underwater bridge pier detection method based on sonar camera and deep learning according to any one of claims 1 to 4, characterized in that: include: Receiving module: Based on the improved YOLOv7 model, it receives sonar image input containing multiple bridge piers and their surface defects, enabling rapid underwater detection of bridge piers and defects. The improved YOLOv7 model introduces Swin-Transformer (Swin-T) in the backbone and spatial-channel attention mechanism (CBAM) in the neck; In the improved YOLOv7 model, when the feature map is input into the backbone, it no longer passes through the convolution layer, but passes through the patch partitioning layer and two pairs of Swin-T modules. The two pairs of Swin-T modules are connected by a patch merging layer. The improved YOLOv7 connects CBAM with three RepConv layers and one upsampling layer in the head layer of YOLOv7 to further integrate the features passed into the head; Distance information acquisition module: used to combine the bounding box coordinate information obtained by the improved YOLOv7 model with the sonar distance parameters to obtain the actual distance information data of the defect relative to the bridge pier; Image acquisition module: used to acquire images of the defective positions of piers based on the obtained distance information data; Defect output module: used to input the collected pier defect images into the improved DeepLabv3+ model and output the defect recognition results; The improved DeepLabv3+ model introduces branches of continuous convolution and adaptive average pooling after concatenating shallow feature maps with upsampled feature maps in the original DeepLabv3+ semantic segmentation model.

6. The underwater bridge pier rapid detection system based on sonar camera and deep learning according to claim 5 is characterized in that: The receiving module includes: Acquisition unit: used to acquire images of multiple bridge piers and their surface defects, specifically configured as a mechanical scanning sonar; Parsing unit: used to parse image information data elements; Sending unit: used to send the parsed image information to the improved YOLOv7 model.

Images