Multimodal pier damage detection method and system based on image sonar adjustment

Abstract

The application provides a kind of multi-modal pier damage detection method and system based on image sonar adjustment, it is related to image processing method and system, system includes: holder, for carrying image sonar;And receive step adjustment signal, according to preset step angle adjustment sonar detection angle;Image sonar, for collecting sonar image;Quality evaluation module, for evaluating and obtaining at least one image index of sonar image;Adjustment strategy module, for gradually adjusting sonar detection angle in angle adjustment range with set step angle;And obtain all sonar images in angle adjustment range and corresponding image index;And select the sonar image of image index or image index weighted sum maximum;Image processing module, for processing sonar image breakage identification module, for pier breakage identification to the sonar image after processing, complete the damage detection of pier.

Description

TECHNICAL FIELD

[0001] The present application relates to an image processing method and system, in particular to a multi-modal pier damage detection method and system based on image sonar adjustment. BACKGROUND

[0002] In bridge structure health detection, the integrity of the pier, as a key load-bearing structure, is directly related to the safe operation of the bridge. The existing detection methods mainly rely on manual diving or underwater robots carrying optical cameras for visual detection. However, in turbid water or poor lighting conditions, the quality of optical images is severely degraded, making it difficult to accurately identify pier damage.

[0003] In recent years, image sonar technology has been gradually applied to underwater structure detection due to its advantages in underwater long-distance imaging and penetration. However, image sonar has limitations such as low resolution and obvious geometric distortion, which restrict its application in high-precision detection. Moreover, existing detection systems lack effective fusion and calibration mechanisms for multi-modal data (such as sonar images, optical images, and laser scales) in complex underwater environments, making it difficult to ensure the accuracy and consistency of detection results. SUMMARY

[0004] To solve the technical problems existing in the prior art, the present application provides a multi-modal pier damage detection method and system based on image sonar adjustment, which improves the resolution of sonar images collected by image sonar, reduces geometric distortion, and increases detection accuracy and environmental adaptability.

[0005] To achieve the above-mentioned application purposes, the present application provides a multi-modal pier damage detection method based on image sonar adjustment, comprising:

[0006] Step S1, collecting a sonar image of a pier;

[0007] Step S2, evaluating and obtaining at least one image index of the sonar image;

[0008] Step S3, comparing the size of the image index with the corresponding image index threshold value;

[0009] When any image index is less than the corresponding image index threshold value, the sonar detection angle is adjusted step by step at a set step angle, and steps S1-S2 are executed after each adjustment of the sonar detection angle until all sonar images within the angle adjustment range and the corresponding image indexes are obtained;

[0010] Step S4, comparing the size of each sonar image or the weighted sum of the image indexes, and selecting the sonar image with the largest image index or the weighted sum of the image indexes;

[0011] Step S5, processing the sonar image;

[0012] The processing includes at least one of denoising the sonar image, information complementation and fusion enhancement of the sonar image by an optical image, and correction of geometric distortion of the sonar image by distance data;

[0013] Step S6, pier damage identification is performed on the processed sonar image, and pier damage detection is completed.

[0014] The application also provides a multi-modal pier damage detection system based on image sonar adjustment, comprising:

[0015] A gimbal is used to carry the image sonar, and receives a step adjustment signal to adjust the sonar detection angle according to a preset step angle;

[0016] The gimbal serves as a motion carrier of multiple sensors, and realizes flexible pointing of the sonar and the camera through multi-degree-of-freedom rotation; the laser ruler independently provides a distance reference, and the three form a multi-modal detection capability through space calibration and time synchronization.

[0017] The image sonar is used to collect a sonar image and send it to a quality assessment module;

[0018] The quality assessment module is used to assess and obtain at least one image index of the sonar image, and send it to an adjustment strategy module;

[0019] The adjustment strategy module is used to compare the size of the image index and a corresponding image index threshold value;

[0020] When any image index is less than the corresponding image index threshold value, a step adjustment signal is sent to the gimbal to gradually adjust the sonar detection angle in a preset step angle within an angle adjustment range; and after each adjustment of the sonar detection angle, a sonar image is obtained through the image sonar, and a corresponding image index is obtained through the quality assessment module, until all sonar images and corresponding image indexes within the angle adjustment range are obtained;

[0021] And the size of the image index or the weighted sum of the image indexes of each sonar image is compared, the sonar image with the largest image index or the weighted sum of the image indexes is selected, and is sent to an image processing module;

[0022] The image processing module is used to process the sonar image and send it to a damage identification module;

[0023] The processing includes at least one of denoising the sonar image, information complementation and fusion enhancement of the sonar image by an optical image, and correction of geometric distortion of the sonar image by distance data;

[0024] The damage identification module is configured to identify damage of the piers based on the processed sonar image, thereby completing damage detection of the piers.

[0025] According to one of the technical solutions of the present application, the adjustment strategy module is further configured to send the optimal detection angle to the pan-tilt head, wherein the optimal detection angle is the sonar detection angle corresponding to the sonar image with the maximum image index or the weighted sum of image indexes.

[0026] The pan-tilt head is further configured to receive the optimal detection angle and adjust to the optimal detection angle.

[0027] According to one of the technical solutions of the present application, the adjustment strategy module is further configured to compare the image index with the corresponding image index threshold.

[0028] When all the image indexes are greater than the corresponding image index threshold, the sonar image is sent to the image processing module.

[0029] According to one of the technical solutions of the present application, the image index comprises at least one of gradient entropy and edge sharpness.

[0030] According to one of the technical solutions of the present application, the image processing module comprises:

[0031] The noise reduction unit is configured to perform multi-layer decomposition on the sonar image by wavelet transform, extract low-frequency approximate coefficients and high-frequency detail coefficients after each layer of decomposition, perform soft threshold denoising processing on all high-frequency detail coefficients, retain the low-frequency approximate coefficients, and reconstruct the image by inverse wavelet transform using the low-frequency approximate coefficients and the high-frequency detail coefficients after denoising processing, thereby obtaining the denoised sonar image.

[0032] According to one of the technical solutions of the present application, the system further comprises an optical camera.

[0033] The optical camera is configured to collect optical images of the same detection area at the same time as the sonar images are collected.

[0034] The image processing module further comprises a fusion enhancement unit, and the fusion enhancement unit is configured to,

[0035] extract feature maps of the sonar image and the optical image by the convolutional neural network;

[0036] obtain a sonar image key vector from the feature map of the sonar image, obtain an optical image query vector and an optical image value vector from the feature map of the optical image, and calculate an attention weight using the sonar image key vector, the optical image query vector, and a scaling factor;

[0037] weight the optical image value vector by the attention weight, and then fuse the weighted optical image value vector with the feature map of the sonar image to obtain a fused sonar image.

[0038] According to one of the technical solutions of the application, further comprising;

[0039] A laser scale is used to collect distance data of the same detection area while collecting the sonar image.

[0040] The image processing module further comprises a geometric correction unit, which is configured to,

[0041] Register the distance data with the pixel coordinates of the sonar image.

[0042] Recalculate the real coordinates of each pixel point in the sonar image in the distance data, convert the pixel coordinates of the sonar image into real space coordinates, and obtain a geometrically corrected sonar image.

[0043] According to one of the technical solutions of the application, the damage identification module comprises:

[0044] A consistency verification unit is configured to detect damage cracks in the fused sonar image and the corrected sonar image respectively, and extract the center lines of the damage cracks.

[0045] Recalculate the Hausdorff spatial distance of the two center lines, and obtain a consistency score based on the size relationship between the Hausdorff spatial distance and a preset spatial distance threshold.

[0046] An accuracy prompting unit is configured to calculate a comprehensive confidence based on the consistency score and an image quality index.

[0047] And when the comprehensive confidence is greater than or equal to a preset confidence threshold, it is determined that the pier is damaged.

[0048] According to one of the technical solutions of the application, further comprising:

[0049] A synchronization module is configured to synchronize the time of the gimbal, the image sonar, the optical camera, and the laser scale.

[0050] The application provides a multi-modal pier damage detection method and system based on image sonar adjustment, which has the following advantages compared with the prior art:

[0051] The resolution of the sonar image collected by the image sonar is improved, the geometric distortion is reduced, the detection accuracy and environmental adaptability are improved, and through key technologies such as image sonar dynamic angle adjustment, cross-modal data fusion enhancement, and laser scale auxiliary geometric correction, high-precision and high-robustness detection of pier damage is realized, and the detection performance and automation level of the system in complex underwater environments are improved. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description only represent some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without any creative effort.

[0053] Figure 1 A flow chart schematically showing a multi-modal pier damage detection method based on image sonar adjustment according to an embodiment of the present application;

[0054] Figure 2 A structural diagram schematically showing a multi-modal pier damage detection system based on image sonar adjustment according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] The description of the embodiments of the present application should be combined with the corresponding drawings, which should be regarded as a part of the complete description. In the drawings, the shapes or thicknesses of the embodiments can be exaggerated and simplified for illustration or convenience. Furthermore, parts of the structures in the drawings will be described separately, and it should be noted that the elements not shown or not described in the drawings are in the form known by those skilled in the art.

[0056] The description of the embodiments herein, any reference to directions and positions, is only for the convenience of description, and cannot be understood as any limitation on the scope of protection of the present application. The following description of the preferred embodiments will involve combinations of features, which can exist independently or in combination, and the present application is not particularly limited to the preferred embodiments. The scope of the present application is defined by the claims.

[0057] As shown in Figures 1-2 A multi-modal pier damage detection method based on image sonar adjustment according to an embodiment of the present application comprises:

[0058] Step S1, collecting sonar images of the pier;

[0059] Step S2, evaluating and obtaining at least one image index of the sonar images;

[0060] Step S3, comparing the size of the image index with that of the corresponding image index threshold value;

[0061] When any image index is less than the corresponding image index threshold value, the sonar detection angle is adjusted step by step at a set step angle, and steps S1-S2 are performed after each adjustment of the sonar detection angle until all sonar images and corresponding image indexes in the angle adjustment range are obtained;

[0062] Step S4, comparing the size of the image index or the image index weighted sum of each sonar image, selecting the sonar image with the largest image index or image index weighted sum;

[0063] Step S5, processing the sonar image;

[0064] The processing includes at least one of noise reduction of the sonar image, information complementation and fusion enhancement of the sonar image by the optical image, and correction of geometric distortion of the sonar image by distance data;

[0065] Step S6, pier damage identification is performed on the processed sonar image, and the damage detection of the pier is completed.

[0066] In the multi-modal pier damage detection system based on image sonar adjustment, comprising:

[0067] A gimbal is used to carry the image sonar, and receives a step adjustment signal to adjust the sonar detection angle according to a preset step angle;

[0068] The image sonar is used to collect sonar images and send them to a quality assessment module;

[0069] The quality assessment module is used to assess and obtain at least one image index of the sonar image, and send it to an adjustment strategy module;

[0070] The adjustment strategy module is used to compare the size of the image index and the corresponding image index threshold value;

[0071] When any image index is less than the corresponding image index threshold value, a step adjustment signal is sent to the gimbal to gradually adjust the sonar detection angle in the angle adjustment range at a set step angle; and after each adjustment of the sonar detection angle, the sonar image is obtained by the image sonar and the corresponding image index is obtained by the quality assessment module, until all sonar images in the angle adjustment range and the corresponding image indexes are obtained;

[0072] and the size of the image index or the image index weighted sum of each sonar image is compared, the sonar image with the largest image index or image index weighted sum is selected, and sent to an image processing module;

[0073] The image processing module is used to process the sonar image and send it to a damage identification module;

[0074] The processing includes at least one of noise reduction of the sonar image, information complementation and fusion enhancement of the sonar image by the optical image, and correction of geometric distortion of the sonar image by distance data;

[0075] The damage identification module is used to perform pier damage identification on the processed sonar image, and complete the damage detection of the pier.

[0076] In this embodiment, mainly relates to image sonar enhancement module, image processing module and damage identification module. The image sonar enhancement module is used to improve the imaging quality of image sonar in complex underwater environment, mainly including adaptive angle adjustment unit.

[0077] The adaptive angle adjustment unit includes a gimbal, a quality evaluation module and an adjustment strategy module.

[0078] In the adaptive angle adjustment unit, the image sonar is connected through the gimbal, the quality evaluation module analyzes the gradient entropy and edge sharpness of the sonar image, and then the adjustment strategy module dynamically adjusts the gimbal pitch angle (sonar detection angle) to form an optimal angle between the sound wave incidence direction and the crack direction on the pier surface, thereby enhancing the reflection signal strength of the crack area and improving the imaging clarity of the crack area.

[0079] The specific working principle of the adaptive angle adjustment unit is as follows:

[0080] The quality evaluation module performs image quality evaluation:

[0081] The system collects the current sonar image in real time , calculates its image indicators, i.e. gradient entropy and edge sharpness , wherein:

[0082]

[0083] wherein is the probability of the i gray level in the gradient histogram of the sonar image, N is the number of gradient histogram levels.

[0084] The edge sharpness uses the Laplacian operator to calculate the sharpness of the sonar image:

[0085]

[0086] wherein, M is the total number of pixels in the sonar image.

[0087] The adjustment strategy module executes the following angle adjustment strategy: if or , wherein , are preset threshold values, then the gimbal pitch angle adjustment is started, the angle is adjusted (step angle) each time, and the sonar image is reacquired, the and after each adjustment are recorded, and the angle that makes maximum is selected as the optimal detection angle, wherein is a weighting coefficient. If the image index is only one index, the above angle adjustment strategy can be performed by comparing the size of the index alone.

[0088] In the process of image acquisition, a dynamic window mechanism can also be used to adjust the size of the evaluation window according to the local edge density of the image The size of the evaluation window is dynamically adjusted:

[0089]

[0090] wherein is the window area, is an edge detection threshold. If > a small window (such as ) is used; otherwise, a large window (such as ) is used.

[0091] The image processing module performs corresponding processing on the enhanced sonar image, and then performs damage identification through the damage identification module.

[0092] In the multi-modal pier damage detection system based on image sonar adjustment, the adjustment strategy module is also used to send the sonar detection angle corresponding to the image index or the maximum weighted sum of the image index to the gimbal as the optimal detection angle;

[0093] The gimbal is also used to receive the optimal detection angle and adjust to the optimal detection angle.

[0094] In this embodiment, the control and coordination subsystem is mainly involved.

[0095] The control and coordination subsystem is responsible for realizing closed-loop control of the gimbal angle adjustment.

[0096] The adaptive adjustment strategy based on image quality feedback is embedded in the control logic, which can automatically trigger the gimbal angle adjustment process when the sonar image clarity is lower than the threshold (the image quality evaluation module feedbacks that the image clarity decreases), find the optimal detection angle, and improve the overall detection efficiency and automation level. And it can be evaluated again after adjustment to form a closed-loop feedback to ensure that the detection angle is always in the optimal state.

[0097] In the multi-modal pier damage detection system based on image sonar adjustment, the adjustment strategy module is also used to compare the size of the image index and the corresponding image index threshold;

[0098] When all image indexes are greater than the corresponding image index threshold, the sonar image is sent to the image processing module.

[0099] In the multi-modal pier damage detection system based on image sonar adjustment, the image index includes at least one of gradient entropy and edge clarity.

[0100] In the multi-modal pier damage detection system based on image sonar adjustment, the image processing module comprises:

[0101] The noise reduction unit is configured to perform multi-layer decomposition on the sonar image by using wavelet transform, extract low-frequency approximation coefficients and high-frequency detail coefficients after each layer of decomposition, perform soft threshold denoising processing on all high-frequency detail coefficients, and retain the low-frequency approximation coefficients; and reconstruct the image by using the low-frequency approximation coefficients and the high-frequency detail coefficients after the denoising processing through inverse wavelet transform to obtain the denoised sonar image.

[0102] In the embodiment, the image processing module comprises a noise reduction unit (multi-scale feature extraction unit). The multi-scale feature extraction unit performs multi-scale decomposition on the sonar image by using wavelet transform or a CNN convolutional neural network, separates high-frequency noise and low-frequency structural features (crack features), realizes effective enhancement and denoising of crack information, and improves the signal-to-noise ratio of the image.

[0103] If wavelet transform is used, the process is as follows:

[0104] The sonar image is decomposed by three layers by using wavelet transform, and low-frequency approximation coefficients and high-frequency detail coefficients (horizontal, vertical, and diagonal directions) are extracted. The high-frequency coefficients are subjected to soft threshold denoising processing, and the low-frequency structural information is retained. The reconstructed image is:

[0105]

[0106] In the multi-modal pier damage detection system based on image sonar adjustment, the optical camera further comprises an optical camera:

[0107] The optical camera is configured to collect an optical image of the same detection area at the same time as the sonar image is collected.

[0108] The image processing module further comprises a fusion enhancement unit, and the fusion enhancement unit is configured to:

[0109] extract feature maps of the sonar image and the optical image by using a convolutional neural network;

[0110] obtain a sonar image key vector from the feature map of the sonar image, obtain an optical image query vector and an optical image value vector from the feature map of the optical image, and calculate an attention weight by using the sonar image key vector, the optical image query vector, and a scaling factor;

[0111] weight the optical image value vector by using the attention weight, and then fuse the weighted optical image value vector with the feature map of the sonar image to obtain a fused sonar image.

[0112] In this embodiment, the image processing module is also used to realize multi-modal data fusion, realize information complementation and fusion enhancement between the image sonar and the optical image, and contains a fusion enhancement unit (realizing a cross-modal attention mechanism). The cross-modal attention mechanism is based on a Transformer architecture, establishes a pixel-level mapping relationship between the sonar image and the optical image, uses the high-resolution edge information in the optical image to guide the local enhancement of the sonar image, enhances the fuzzy area of the sonar image, and improves the clarity of the crack edge.

[0113] The process of the fusion enhancement unit realizing the cross-modal attention mechanism is as follows:

[0114] 1. The sonar image and the optical image are respectively extracted by using a CNN to obtain feature maps

[0115] 、

[0116] Attention weight calculation: the attention weight is calculated by dot product :

[0117]

[0118] wherein are the query and key vectors of the optical image and the sonar image respectively, is a scaling factor.

[0119] 2. Weighted fusion of the optical image feature to the sonar image:

[0120]

[0121] wherein V is the value vector of the optical image. The final output is a fusion image .

[0122] In the multi-modal pier damage detection system based on image sonar adjustment, it also includes;

[0123] A laser ruler is used to collect distance data of the same detection area while collecting the sonar image.

[0124] The image processing module further includes a geometric correction unit, which is used to

[0125] register the distance data and the pixel coordinates of the sonar image;

[0126] Then calculate the corresponding coordinates of each pixel point in the image in the sonar image, transform the pixel coordinates of the sonar image into spatial coordinates, and obtain the geometrically corrected sonar image.

[0127] In this embodiment, the image processing module further comprises a geometric correction unit for implementing geometric correction containing laser ruler guidance.

[0128] The geometric correction unit dynamically corrects geometric distortion in the sonar image caused by sound wave propagation delay based on the distance data collected by the laser ruler, improves the accuracy of crack size measurement, and improves the measurement accuracy of crack length and width.

[0129] The geometric correction unit implements the geometric correction process as follows:

[0130] 1. The laser ruler collects the distance points on the surface of the pier Align the coordinates of the sonar image through the ICP algorithm.

[0131] 2. For each pixel point in the sonar image x, y ), calculate the real coordinates in the corresponding point ), construct the mapping function T , and transform the image coordinates into spatial coordinates:

[0132]

[0133] Get the corrected image Satisfy the spatial consistency.

[0134] In the multi-modal pier damage detection system based on image sonar adjustment, the damage recognition module comprises:

[0135] A consistency verification unit is configured to detect damage cracks in the fused sonar image and the corrected sonar image respectively, and extract the center lines of the damage cracks respectively.

[0136] Then calculate the Hausdorff spatial distance of the two center lines, and based on the size relationship between the Hausdorff spatial distance and the preset spatial distance threshold, obtain a consistency score;

[0137] An accuracy prompt unit is configured to calculate a comprehensive confidence based on the consistency score and the image quality index.

[0138] And when the comprehensive confidence is greater than or equal to a preset confidence threshold, it is determined that the pier is damaged.

[0139] In this embodiment, the main part is the damage recognition and accuracy evaluation module.

[0140] The damage recognition and accuracy evaluation module is configured to recognize pier damage based on the fused image data, and evaluate the reliability of the detection result.

[0141] The consistency verification unit realizes bimodal consistency verification, calculates a consistency score as a confidence basis by comparing the positions, trends and other characteristics of the cracks in the sonar image and the optical image. Through the bimodal consistency verification mechanism, the accuracy and reliability of the damage recognition are improved, and the system is suitable for complex underwater pier detection scenes such as turbid water area and poor lighting.

[0142] The accuracy prompt unit judges the reliability of the current detection result according to the image quality index and the bimodal consistency result. If the crack position deviation exceeds the set tolerance threshold (such as 5%), it is automatically marked as “low reliability” and triggers the manual review mechanism to ensure the reliability of the detection result. The process of the consistency verification unit realizing bimodal consistency verification is as follows:

[0143] 1. Detect cracks in the fused image and the geometric correction image , and extract the crack center lines and .

[0144] 2. Calculate the Hausdorff spatial distance of the two center lines :

[0145]

[0146] If < , the consistency score is high, otherwise it is low. is the consistency score threshold, which is determined according to actual needs.

[0147] The process of the accuracy prompt unit realizing dynamic accuracy prompt is as follows:

[0148] 1. Calculate the comprehensive confidence according to the consistency score and the image quality index C :

[0149]

[0150] wherein is a weight factor.

[0151] 2. If (preset confidence threshold), it is marked as “low reliability” and the manual review is prompted.

[0152] In the multi-modal pier damage detection system based on image sonar adjustment, it further comprises:

[0153] A synchronization module is used to synchronize the time of the pan-tilt, image sonar, optical camera and laser ruler.

[0154] In this embodiment, the control and coordination subsystem is also responsible for coordinating the synchronous acquisition and data alignment between the multi-sensor such as image sonar, optical camera, laser ruler, etc., ensuring the time consistency of image sonar, optical camera, laser ruler and the angle adjustment of the holder through timestamp synchronization and spatial coordinate registration technology, and ensuring the precise time and spatial consistency of multi-modal data during acquisition.

[0155] Implementation data:

[0156] In a certain underwater pier detection task, the system successfully identified a crack with a width of less than 10 cm in turbid water, and controlled the crack length measurement error to within ±1.2% through laser ruler correction. Dynamic angle adjustment improves image clarity by 32%, cross-modal fusion enhances crack contrast by 45%, and overall detection efficiency improves by 40%.

[0157] According to one aspect of the present application, a computer readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement a multi-modal pier damage detection method based on image sonar adjustment as in the above technical solution.

[0158] The computer readable storage medium can include any medium capable of storing or transmitting information. Examples of computer readable storage media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, and the like. Code segments can be downloaded via a computer network such as the Internet, an intranet, etc.

[0159] The multi-modal pier damage detection method and system based on image sonar adjustment of the present application, the system comprises: a holder for carrying an image sonar; and a receiving step adjustment signal, adjusting the sonar detection angle according to the preset step angle; an image sonar for acquiring sonar images; a quality assessment module for evaluating and obtaining at least one image index of the sonar image; an adjustment strategy module for adjusting the sonar detection angle step by step within the angle adjustment range at a set step angle; obtaining all sonar images within the angle adjustment range and corresponding image indexes; and selecting the sonar image with the maximum image index or image index weighted sum; an image processing module for processing the sonar image; a damage recognition module for recognizing the damage of the pier by processing the sonar image, and completing the damage detection of the pier.

[0160] Moreover, it should be noted that the present application can be provided as a method, an apparatus, or a computer program product. Therefore, the present application embodiments can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application embodiments can take the form of a computer program product on one or more computer-usable storage media (including disks, diskettes, tapes, optical, silicon, solid storage, etc.) embodying computer-readable instructions.

[0161] The present application embodiments are described with reference to the flowchart illustrations and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the present application embodiments. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing terminal devices to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal devices, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0162] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal devices to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions means which implement the function specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. These computer program instructions can also be loaded onto a computer or other programmable data processing terminal devices, such that a series of operational steps are performed on the computer or other programmable terminal devices to create a computer implemented process so that the instructions executed on the computer or other programmable terminal devices provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0163] It should also be noted that the terms "comprising", "including", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or terminal device. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0164] Finally, it should be noted that the above is the preferred embodiment of the present application, it should be pointed out that although the preferred embodiment of the present application has been described, for those skilled in the art, once the basic creative concept of the present application is known, without departing from the principles of the present application, improvements and refinements can also be made, these improvements and refinements should also be considered as the protection scope of the present application. Therefore, the appended claims are intended to be interpreted as including the preferred embodiment and all changes and modifications falling within the scope of the embodiments of the present application.

Claims

1. A multi-modal pier damage detection system based on image sonar adjustment, for performing a multi-modal pier damage detection method, comprising: Step S1, collecting a sonar image of a pier; Step S2, evaluating and obtaining at least one image index of the sonar image; Step S3, comparing the size of the image index with that of a corresponding image index threshold value; When any image index is less than the corresponding image index threshold value, then gradually adjust the sonar detection angle by a preset step angle within an angle adjustment range, and perform steps S1-S2 after each adjustment of the sonar detection angle until all sonar images within the angle adjustment range and corresponding image indexes are obtained; Step S4, comparing the size of each sonar image or the sum of image indexes, and selecting the sonar image with the largest image index or the sum of image indexes; Step S5, processing the sonar image; The processing includes at least one of denoising the sonar image, information complementation and fusion enhancement of the sonar image by an optical image, and correction of geometric distortion of the sonar image by distance data of a laser rangefinder; Step S6, pier damage identification is performed on the processed sonar image to complete the damage detection of the pier; The multi-modal pier damage detection system comprises: a gimbal for carrying an image sonar, and receiving a step adjustment signal to adjust the sonar detection angle according to a preset step angle; the image sonar for collecting a sonar image and sending it to a quality evaluation module; the quality evaluation module for evaluating and obtaining at least one image index of the sonar image and sending it to an adjustment strategy module; the adjustment strategy module for comparing the size of the image index with that of a corresponding image index threshold value; When any image index is less than the corresponding image index threshold value, then send a step adjustment signal to the gimbal to gradually adjust the sonar detection angle by a preset step angle within an angle adjustment range; and after each adjustment of the sonar detection angle, obtain a sonar image by the image sonar and corresponding image indexes by the quality evaluation module until all sonar images within the angle adjustment range and corresponding image indexes are obtained; and for comparing the size of each sonar image or the sum of image indexes, and selecting the sonar image with the largest image index or the sum of image indexes, and sending it to an image processing module; the image processing module for processing the sonar image and sending it to a damage identification module; The processing includes at least one of denoising the sonar image, information complementation and fusion enhancement of the sonar image by an optical image, and correction of geometric distortion of the sonar image by distance data of a laser rangefinder; the damage identification module for performing pier damage identification on the processed sonar image to complete the damage detection of the pier; the adjustment strategy module is also used for sending the sonar detection angle corresponding to the sonar image with the largest image index or the sum of image indexes as an optimal detection angle to the gimbal; the gimbal is also used for receiving the optimal detection angle and adjusting to the optimal detection angle.

2. The image sonar regulation based multi-modal pier damage detection system of claim 1, wherein, The adjusting strategy module is further configured to compare the image indicators with corresponding image indicator thresholds; When all the image indicators are greater than the corresponding image indicator thresholds, the sonar image is sent to an image processing module.

3. The multi-modal pier damage detection system based on image sonar regulation of claim 2, wherein, The image indicators include at least one of gradient entropy and edge sharpness.

4. The multi-modal pier damage detection system based on image sonar regulation of claim 3, wherein, The image processing module includes: A denoising unit is configured to perform multi-layer decomposition on the sonar image by using wavelet transform, extract low-frequency approximation coefficients and high-frequency detail coefficients after each layer of decomposition, perform soft threshold denoising processing on all the high-frequency detail coefficients, retain the low-frequency approximation coefficients, and reconstruct the image by using the low-frequency approximation coefficients and the high-frequency detail coefficients after denoising processing through inverse wavelet transform to obtain a denoised sonar image.

5. The multi-modal pier damage detection system based on image sonar regulation of claim 4, wherein, Further comprising an optical camera: The optical camera is configured to collect an optical image of the same detection area at the same time as the sonar image is collected. The image processing module further includes a fusion enhancement unit, which is configured to, extract feature maps of the sonar image and the optical image through a convolutional neural network; then obtain a sonar image key vector from the feature map of the sonar image, an optical image query vector and an optical image value vector from the feature map of the optical image, and calculate an attention weight by using the sonar image key vector, the optical image query vector and a scaling factor; then weight the optical image value vector by using the attention weight, and then fuse the weighted optical image value vector with the feature map of the sonar image to obtain a fused sonar image.

6. The multi-modal pier damage detection system based on image sonar regulation of claim 4 or 5, wherein, Further comprising; a laser scale configured to collect distance data of the same detection area at the same time as the sonar image is collected; The image processing module further includes a geometric correction unit, which is configured to, register the crack distance data with pixel coordinates of the sonar image; then calculate real coordinates corresponding to each pixel point in the sonar image in the distance data, convert the pixel coordinates of the sonar image into real space coordinates, and obtain a geometrically corrected sonar image.

7. The image sonar regulation based multi-modal pier damage detection system of claim 6, wherein, The damage identification module includes: a consistency verification unit configured to detect damage cracks in the fused sonar image and the corrected sonar image respectively, and extract center lines of the damage cracks respectively; then calculate a Hausdorff spatial distance of the two center lines, and obtain a consistency score based on a size relationship between the Hausdorff spatial distance and a preset spatial distance threshold; an accuracy prompting unit configured to calculate a comprehensive confidence based on the consistency score and an image quality indicator; and when the comprehensive confidence is greater than or equal to a preset confidence threshold, it is determined that the pier is damaged.

8. The image sonar regulation based multi-modal pier damage detection system of claim 6, wherein, Further comprising: a synchronization module configured to synchronize the time of the gimbal, the image sonar, the optical camera and the laser scale.

Images