Dam body defect composite sensing method based on sound-light thickness guidance

Abstract

The invention provides a dam body defect composite sensing method based on sound-light thickness guidance, and the method comprises the steps: firstly controlling an underwater robot to synchronously collect sonar and optical images of a dam body region, and guaranteeing the time-space alignment of data; performing targeted preprocessing on the bimodal image and constructing unified input; the method is characterized in that macroscopic contour features and optical detail texture features of sonar are respectively extracted through a double-branch network, and an environment adaptive dynamic weight mechanism based on a real-time detection distance and an image signal-to-noise ratio is innovatively introduced to carry out feature fusion; carrying out calibration, denoising and multi-scale optimization on the fusion features; and finally, outputting defect types and positions by utilizing a decoupling detection head. According to the method, through the cooperative normal form of sonar coarse scanning and optical fine detection and dynamic weight adjustment, the large-area scanning efficiency and the fine defect recognition precision are effectively considered, the adaptability and reliability in the complex underwater environment are improved, and an efficient solution is provided for full-automatic unmanned detection of the dam body.

Description

Technical Field

[0001] This invention belongs to the field of water conservancy engineering safety inspection technology, specifically involving a composite perception method for dam defects based on coarse and fine acoustic-optical guidance. It is particularly suitable for efficient and high-precision automated defect detection scenarios for large-area underwater dam structures mounted on underwater robots. Background Technology

[0002] As a crucial national water conservancy infrastructure, the structural safety of dams directly impacts the safety of life and property downstream and the stability of the ecological environment. The dam body, especially the underwater section, is subjected to long-term hydraulic erosion, seepage pressure, and material aging, making it prone to defects such as cracks, erosion, and voids. Timely detection and accurate assessment of these defects are key to preventing major safety accidents such as dam failures. Currently, underwater dam inspection primarily relies on sensors mounted on remotely operated vehicles (ROVs / AUVs).

[0003] Existing underwater robot-based detection technologies are mainly divided into two categories: long-range perception based on sonar and short-range perception based on optical vision. Sonar detection has the advantages of wide coverage and being unaffected by water turbidity, but its image resolution is low, its ability to identify fine cracks is poor, and it is easily affected by aquatic plants, debris, etc., resulting in a high false alarm rate. Optical vision detection can provide high-resolution image details and accurately identify features such as crack texture and width, but it is extremely susceptible to suspended matter in the water, uneven lighting, and turbidity. It also has a short detection range and low efficiency, making it difficult to meet the needs of rapid surveys of large-area dams.

[0004] To combine the advantages of both, existing technologies have proposed detection schemes based on acoustic-optical fusion. For example, patent document CN118154993B discloses a dual-modal detection method based on acoustic-optical image fusion, which uses a unified backbone network to extract dual-modal features and calculates fixed fusion weights based on image feature similarity. However, this method performs indiscriminate fine processing on all regions, resulting in large computational redundancy and a lack of adaptability to the detection scene. It cannot dynamically adjust the perception strategy according to external environmental factors such as distance and water quality, making it difficult to balance efficiency and accuracy in complex underwater environments and failing to meet the requirements of real-time detection in engineering applications.

[0005] Furthermore, other multimodal sensing schemes (such as patent documents with publication numbers CN120852689A and CN119270281B) either focus on marine ranch environment reconstruction and SLAM mapping, or on multi-sensor precise positioning and trajectory optimization. Their sensing frameworks, fusion mechanisms, and objectives are fundamentally different from the present invention, which is specifically designed for efficient detection of dam defects. None of them have solved the core technical problem of "how to achieve the optimal balance between efficiency, accuracy, and robustness in large-area dam defect detection through the coordinated guidance and dynamic adaptation of acoustic and optical modes".

[0006] Therefore, there is an urgent need in this field for a novel composite sensing method for dam defects that can intelligently coordinate the advantages of sonar's "wide coverage" and optics' "high precision" and can adapt to changes in complex underwater environments. Summary of the Invention

[0007] To address the needs of existing technologies, this invention provides a composite sensing method for dam defects based on coarse and fine acoustic-optical guidance. The aim is to construct a collaborative sensing paradigm of "coarse sonar scanning guidance and precise optical inspection" and introduce an environment-adaptive dynamic weight fusion mechanism to achieve efficient coverage, accurate identification, and robust detection of defects in a wide range of underwater dams.

[0008] A composite sensing method for dam defects based on acoustic-optical coarse-grained guidance includes the following steps:

[0009] Step 1: Control the underwater robot equipped with sonar and optical sensors to move along a preset path, synchronously trigger the sonar and optical sensors to collect the original sonar and optical images of the dam area, and ensure the data is aligned in time and space through sensor calibration and time synchronization.

[0010] Step 2: Perform mean filtering and grayscale stretching on the original sonar image to obtain preprocessed sonar image data; perform contrast enhancement and noise reduction on the original optical image to obtain preprocessed optical image data; pair the preprocessed sonar image data and optical image data and unify their sizes to construct a multimodal input tensor;

[0011] Step 3: Dual-branch feature extraction and environment-adaptive dynamic weight fusion:

[0012] Step 3.1: Extract sonar data from the multimodal input tensor and input it into the sonar branch network to extract a coarse-grained feature map representing the macroscopic contour. ;

[0013] Step 3.2: Extract optical data from the multimodal input tensor and input it into the optical branch network to extract fine-grained feature maps representing detailed textures. ;

[0014] Step 3.3: Acquire the current detection range and image signal-to-noise ratio in real time, and determine the sonar branch weights through the dynamic weight calculation module. With optical branch weight and according to the formula Weighted fusion is performed to obtain intermediate fusion features. ;

[0015] Step 4: For the intermediate fusion features Channel mapping and modal consistency calibration are performed, and noise suppression is carried out based on a preset interference feature library to obtain denoised features; the denoised features are input into a feature pyramid network for multi-scale fusion to generate the final fused feature map;

[0016] Step 5: Input the fused feature map into the decoupled detection head, and output the defect category probability and bounding box coordinates through parallel classification and regression branches respectively; after confidence threshold filtering and non-maximum suppression processing, the final defect detection result is output.

[0017] Furthermore, the spatiotemporal alignment in step 1 includes spatial coordinate alignment and time synchronization;

[0018] The spatial coordinate alignment is achieved through sensor calibration, and a transformation matrix from the sonar and optical coordinate systems to the unified world coordinate system is established using a dual-mode adapter calibration board.

[0019] The time synchronization is ensured by hardware synchronization triggering and timestamps are recorded during acquisition. When the timestamp deviation between sonar and optical data exceeds a set threshold, linear interpolation is used for data compensation.

[0020] Furthermore, the sonar branch network in step 3.1 is a lightweight residual network based on MobileOne blocks; the optical branch network in step 3.2 is a network structure with MobileOne blocks as front-end units and CSPDarknet as the backbone network.

[0021] Furthermore, in step 3.3, the dynamic weight calculation module determines the weights using the following formula:

[0022]

[0023] in: This is the Softmax activation function, used to map parameters to weights in the 0-1 range; and These are the learnable parameters obtained through network training; This represents the real-time image signal-to-noise ratio.

[0024] Furthermore, the weights determined by the dynamic weight calculation module satisfy the following conditions:

[0025] When the detection distance At that time, sonar branch weight Automatically increases to 55%-70%;

[0026] When the detection distance And image signal-to-noise ratio At that time, optical branch weight Automatically increases to 60%-75%;

[0027] When the image signal-to-noise ratio At that time, sonar branch weight It automatically rises to 57%-65%.

[0028] Furthermore, the calibration process in step 4 specifically involves:

[0029] For the intermediate fusion features Channel mapping and decomposition are performed to obtain the calibrated sonar modal features and optical modal features;

[0030] A modal consistency loss function is introduced to constrain the alignment of the calibrated sonar modal features with the optical modal features in a unified space;

[0031] Calculate the Euclidean distance between the calibrated feature and the template in the preset interference feature library. If the distance is less than the set threshold, it is determined to be an interference feature and is removed.

[0032] Furthermore, the feature pyramid network in step 4 adopts a path aggregation network structure, which integrates multi-scale feature information through the coordinated action of three paths: bottom-up, top-down, and horizontal connection.

[0033] Furthermore, the decoupling detection head in step 5 includes:

[0034] The classification branch, consisting of convolutional layers, batch normalization layers, activation functions, and a sigmoid output layer, is used to output the probability of the "defect / non-defect" category.

[0035] The regression branch, consisting of convolutional layers, batch normalization layers, activation functions, and linear output layers, is used to output the center coordinates, width, and height of the defect bounding box.

[0036] The classification loss uses the binary cross-entropy loss function, and the regression loss uses the GIoU loss function.

[0037] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the above-described method for composite sensing of dam defects based on coarse and fine acoustic-optical guidance.

[0038] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method for composite sensing of dam defects based on coarse and fine acoustic-optical guidance.

[0039] Compared with the prior art, the present invention has the following significant advantages:

[0040] By adopting the guided perception paradigm of "coarse scanning with sonar to locate suspected areas and fine optical inspection for precise identification", redundant fine optical calculations in defect-free areas are effectively avoided. This significantly improves the efficiency of scanning large-area dam bodies while ensuring the accuracy of identifying minute defects, fundamentally solving the industry problem of balancing efficiency and accuracy.

[0041] An innovative dynamic weight allocation mechanism driven by external environmental parameters (distance D, signal-to-noise ratio S) is proposed, replacing the traditional fixed or internal feature similarity-based weight scheme. This enables the system to automatically adjust the contribution of the acoustic and optical modes according to real-time conditions such as water turbidity and detection distance, maintaining reliable detection capabilities even under harsh conditions such as long distance and high turbidity, and exhibiting strong environmental adaptability.

[0042] By using feature space calibration and modal consistency constraints, the problem of fusion distortion caused by the heterogeneity of acoustic and optical features is effectively solved. Combined with the active suppression technology of the preset interference feature library, the false alarm rate caused by common underwater interferences such as aquatic plants and water flow artifacts can be significantly reduced, thereby improving the effectiveness of fusion features and the reliability of detection results.

[0043] The overall solution adopts a lightweight network design and modular processing flow, which can be efficiently deployed on the edge computing unit carried by the underwater robot. Combined with fully automatic detection path planning, it can realize unmanned underwater operation, significantly reduce detection costs, cycle and personnel safety risks, and provide an efficient and reliable technical tool for regular safety inspections and emergency testing of dams. Attached Figure Description

[0044] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0045] The present invention will now be described in detail with reference to the accompanying drawings. Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The directional terms such as left, center, right, top, and bottom in the embodiments of the present invention are only relative concepts or referenced to the normal use state of the product, and should not be considered restrictive.

[0046] A composite sensing method for dam defects based on coarse-grained acoustic-optical guidance, such as Figure 1 As shown, it includes the following steps:

[0047] Step 1: Control the underwater robot equipped with sonar and optical sensors to move along a preset path, synchronously trigger the sonar and optical sensors to collect the original sonar and optical images of the dam area, and ensure the data is aligned in time and space through sensor calibration and time synchronization.

[0048] Spatiotemporal alignment includes spatial coordinate alignment and time synchronization;

[0049] The spatial coordinate alignment is achieved through sensor calibration, using a dual-modal adapter calibration board to establish a transformation matrix from the sonar and optical coordinate systems to a unified world coordinate system. Specifically, the process involves establishing a world coordinate system with the underwater fixed reference point of the dam body as the origin. ,in, As the origin of the world coordinate system, The coordinate axes are arranged horizontally from left to right along the dam body. The coordinate axes are the vertical axes along the dam body from bottom to top; A coordinate axis perpendicular to the dam surface was used. A 50cm×50cm dual-mode calibration plate was fabricated using a black waterproof acrylic sheet as the substrate. A 10×10 white circular array was printed on the surface (to meet the sonar's requirements for grayscale contrast detection). A 1mm wide black crosshair was printed at the center of each white circle as a precise optical positioning target point, meeting the optical sensor's requirements for texture detail localization. An underwater robot was controlled to place the calibration plate within the joint detection range of the two sensors, simultaneously acquiring sonar images of the calibration plate. With optical images Calculate the center coordinates of each circle. As a sonar target, the coordinates of the cross intersection are extracted using a corner detection algorithm. As optical targets, the world coordinates of the targets are then correlated. Based on the physical dimensions of the calibration plate, the true coordinates of each target in the world coordinate system are determined. A known correspondence sample of pixel coordinates and world coordinates was established; by substituting the real coordinates into the spatial transformation formula, a result based on the rotation matrix was obtained. Translation vector The reprojection error is obtained by comparing the predicted optical pixel coordinates with the actual extracted optical target pixel coordinates and calculating the sum of the squares of the Euclidean distances between the two. The calculation formula is as follows:

[0050]

[0051] in, It is the forward projection function from world coordinates to optical pixel coordinates. The larger the value, the more likely it is to be the current rotation matrix. Translation vector The lower the accuracy of the parameter transformation, the better; later, the sum value is adjusted iteratively using the Levenberg-Marquardt algorithm, recalculating with each iteration until the pixel value is reached. If the accuracy requirement is met, the optimal rotation matrix can be obtained by solving this problem. Translation vector These are the optimal transformation parameters. With these parameters, precise bidirectional transformation can be achieved: arbitrary pixel coordinates of sonar or optics can be converted into world coordinates through the inverse projection process from pixel coordinates to world coordinates, and world coordinates can be projected into pixel coordinates of the dual-modality sonar, ultimately achieving spatial alignment between sonar and optical data. Thus, through a calibration board and optimization algorithm specifically designed for underwater acoustic-optical dual-modality, the limitations of poor adaptability and insufficient accuracy of generalized solutions are overcome, providing a high-precision spatial homogeneous data foundation for subsequent feature-level fusion.

[0052] The time synchronization is ensured by hardware synchronization triggering, and a timestamp (sonar timestamp) is recorded during data acquisition. Optical timestamp When the timestamp deviation between sonar and optical data exceeds a set threshold, linear interpolation is used for data compensation; the edge computing unit calculates the time difference in real time. If ΔT > 0.5ms, then time compensation is performed on the data using linear interpolation.

[0053] like Extracting optical data to The fragments are aligned with sonar data;

[0054] like Extracting sonar data to The fragments are aligned with the optical data;

[0055] By using a redundant mechanism of hardware triggering and software compensation, microsecond-level time synchronization is achieved, ensuring that the dam body status information at the same moment strictly corresponds, and avoiding spatiotemporal misalignment of subsequent fusion features due to sensor response delay.

[0056] The core principle of Step 1 is based on the difference in modal characteristics between sonar grayscale-sensitive long-range detection and optical texture-sensitive short-range imaging. By using a dedicated deployment design with coaxial and coplanar alignment and optimal spacing, spatial deviation is reduced at the hardware level. A redundant mechanism of hardware hard synchronization triggering and software timestamp calibration is used to achieve microsecond-level time synchronization. A dual-modal adaptation calibration board combining a white circular array and black cross markers is used to extract precise target points. A unified world coordinate system is established with the fixed reference point of the dam body as the origin. The transformation matrix between the sonar detection coordinate system and the optical imaging coordinate system is solved by least squares method combined with iterative optimization. This ensures that the same dam body location at different detection distances is spatially homogeneous and temporally synchronized in the dual-modal data. This overcomes the limitations of existing generalized spatiotemporal alignment schemes, which have poor adaptability and insufficient accuracy. It provides a high-precision data foundation for subsequent dynamic weight fusion and feature calibration.

[0057] Step 2: Perform mean filtering and grayscale stretching on the original sonar image to obtain preprocessed sonar image data; perform contrast enhancement and noise reduction on the original optical image to obtain preprocessed optical image data; pair the preprocessed sonar image data and optical image data and unify their sizes to construct a multimodal input tensor;

[0058] Sonar image preprocessing:

[0059] A 3×3 convolution kernel is used to perform mean filtering across the image, replacing the center pixel with the mean of neighboring pixels to suppress speckle noise and water flow noise; subsequently, the pixel grayscale values ​​are linearly mapped to [the image using a grayscale stretching formula]. To enhance contour contrast, the grayscale stretching formula is as follows:

[0060]

[0061] in, and These are the coordinates (row and column positions) of a pixel in the image. Represents the pixel position in the original sonar image The original grayscale value at that location; It is a specific numerical value, and its range depends on the output bit depth of the sonar sensor (e.g., 0-255 for 8 bits, 0-4095 for 12 bits, etc.). This value directly reflects the echo intensity received by the sonar sensor at that location, that is, the original signal strength.

[0062] This indicates the position of the same pixel in the image after grayscale stretching preprocessing. The new grayscale value at the location; this value is calculated using the above linear transformation formula, and its range is standardized to 0 to 255 (the standard 8-bit grayscale image range). The purpose is to enhance the contrast of the image, stretch the grayscale distribution that may have been concentrated in a narrow range to the entire usable range, so that the macroscopic contours, edges and other structures are more visually clear, which is convenient for subsequent feature extraction.

[0063] This represents the minimum gray value of all pixels in the entire original sonar image S;

[0064] This represents the maximum grayscale value of all pixels in the entire original sonar image S;

[0065] This indicates that the original grayscale values ​​are normalized to the [0, 1] interval;

[0066] This means mapping the normalized value to a standard 8-bit grayscale range. ;

[0067] Optical image preprocessing: First, the contrast enhancement algorithm of Limiting Contrast Adaptive Histogram Equalization (CLAHE) is used to enhance the contrast, with 8×8 pixels as blocks and a contrast threshold of 2.0 to enhance the details of the crack edges; then, a 5×5 Gaussian kernel with a standard deviation of 0.8 is used for filtering to suppress ambient light interference and random noise caused by water turbidity.

[0068] Data pairing and standardization: The preprocessed sonar grayscale image and optical image are uniformly scaled to 640×640 pixels, and the three-channel RGB optical image is converted into a single-channel grayscale image, which is then paired with the corresponding sonar image to generate acousto-optic fusion data pairs with consistent dimensions. This data is then used to construct a four-dimensional tensor input for the subsequent network. The function and benefits of this step are: by using a dedicated preprocessing scheme for the different noise characteristics of sonar and optical images, the modality-specific interference of the underwater environment is effectively reduced, and heterogeneous data is converted into fusion data pairs with a unified format and standard size, providing a suitable input foundation for the efficient parallel processing of the subsequent network.

[0069] Step 2 completes the standardized preprocessing of the bimodal data, but at this point the data is still pixel-level low-level visual information and cannot be directly used for crack detection. Therefore, starting from this step, the present invention enters the core stage of bimodal feature collaborative optimization and precise fusion. Subsequent steps will no longer rely on a simple splicing mode with fixed weights, but will automatically extract and calibrate complementary semantic features related to crack detection from the standardized bimodal data pairs through a constructed dynamic adaptation fusion architecture, and finally establish a precise correspondence between the bimodal fusion features and crack location and size parameters.

[0070] Step 3: Dual-branch feature extraction and environment-adaptive dynamic weight fusion. The purpose is to extract coarse-grained contour features of sonar and fine-grained detail features of optics respectively, and dynamically adjust the weight ratio of dual-branch features according to the detection scene to achieve the adaptation of "coarse detection at long distance - fine detection at close distance".

[0071] Step 3.1: Extract sonar data from the multimodal input tensor and input it into the sonar branch network to extract a coarse-grained feature map representing the macroscopic contour. Its dimensions are 256×64×64; among them, the sonar branch network is a lightweight residual network based on MobileOne blocks. It solves the gradient vanishing problem in deep networks through residual connections, and is equipped with 3 layers of deep convolution operations (convolution kernel size 3×3, stride 1, padding=1). It focuses on capturing coarse-grained features such as the macroscopic contour of the dam area and the approximate range of suspected defects, and effectively filters speckle noise in sonar images. Its function is to provide core feature support for long-distance coarse detection.

[0072] Step 3.2: Extract optical data from the multimodal input tensor and input it into the optical branch network to extract fine-grained feature maps representing detailed textures. Its dimensions are 256×64×64; the optical branch network uses the MobileOne block as the front unit and is responsible for the initial feature extraction and dimensionality compression. It is followed by the CSPDarknet network as the backbone. Through the progressive operation of its C2f module, it accurately captures fine-grained features such as the edge contour, texture density, and fine size of the crack; its function is to enhance the detail expression capability of optical images and make up for the lack of resolution of sonar images.

[0073] Step 3.3: Real-time acquisition of current detection distance via edge computing unit. (Output from laser rangefinder sensor) and image signal-to-noise ratio (Calculated by analyzing the logarithm of the ratio of the mean pixel value of the image signal to the standard deviation of the noise pixels), based on these two environmental parameters, the sonar branch weights are determined through a dynamic weight calculation module. With optical branch weight and according to the formula Weighted fusion is performed to obtain an intermediate fused feature with a size of 256×64×64. ;

[0074] The dynamic weight calculation module determines the weights using the following formula:

[0075]

[0076] in: This is the Softmax activation function, used to map parameters to weights in the 0-1 range; and β is a learnable parameter obtained through network training. It is usually trained to be negative to reflect that the higher the signal-to-noise ratio, the lower the weight of the sonar. For the sonar branch weights, The two are complementary, representing the optical branch weights, ensuring a dynamic balance in the contribution of dual-modal features;

[0077] The function and benefits of this dynamic weight allocation mechanism are: it overcomes the limitations of traditional fixed-weight or single-parameter driven fusion schemes; and it detects distance... (Determining modal dominance) and image signal-to-noise ratio (Reflecting environmental interference) Driven by two parameters and combined with learnable coefficients and Softmax smooth mapping, the weights are continuously and adaptively adjusted according to complex underwater scenarios.

[0078] The weights determined by the dynamic weight calculation module satisfy the following conditions:

[0079] When the detection distance Time (long-distance scene), sonar branch weight Automatically increases to 55%-70%, sonar coarse-grained features dominate the fusion process, quickly locks the range of suspected defects, and avoids feature interference caused by blurred long-distance optical imaging;

[0080] When the detection distance And image signal-to-noise ratio Time (close-up, clear scene), Decrease The proportion decreased. Increase, make The influence of (negative values) weakens, and the weight of optical branches decreases. Automatically increases to 60%-75%, with fine-grained optical features dominating the fusion process, accurately capturing crack details and compensating for the lack of close-range resolution in sonar;

[0081] When the image signal-to-noise ratio (High turbidity interference scene) Lower, make (Negative) absolute value increases. The positive effect of sonar branch weights becomes more apparent. It automatically rises to 57%-65%, giving full play to the anti-interference advantages of sonar and resisting the influence of water turbidity on optical imaging;

[0082] This mechanism enables a smooth transition from "long-distance coarse inspection" to "close-range fine inspection" and scene adaptation, achieving a dynamic optimal balance between detection efficiency and accuracy;

[0083] Step 4: For the intermediate fusion features Channel mapping and modal consistency calibration are performed, and noise suppression is carried out based on a preset interference feature library to obtain denoised features. These denoised features are then input into a feature pyramid network for multi-scale fusion to generate the final fused feature map. Specifically:

[0084] Step 4.1: Modal feature calibration and interference suppression:

[0085] Feature calibration: 1×1 convolution is used to calibrate intermediate features. The number of channels is mapped to 512 dimensions to eliminate the dimensional barrier between coarse-grained sonar features and fine-grained optical features, enabling both to have a computational basis in the same high-dimensional space; at the same time, a modal consistency loss function is introduced. Constraints are imposed, among which... and For the calibrated bimodal features (i.e., sonar modal features and optical modal features), cosine similarity is used to constrain the bimodal features to align in the same space, by minimizing... It forces the orientation of dual-modal features to be consistent in a unified space, and quantization ensures alignment accuracy, thus solving the alignment deviation problem caused by the lack of quantization constraints in traditional simple registration.

[0086] Interference suppression: Based on a pre-set underwater interference feature library (including templates for water flow artifacts, aquatic plants, etc.), the calibrated features are calculated. Euclidean distance from the interfering template T ,in, Features The dimensional elements, For the interference template T, the first Dimensional element; set distance threshold ,like If it is determined to be a interference feature, then directly from the feature... Remove from the middle; if Features deemed valid are retained and integrated into denoised features. ;

[0087] The function and benefits of this step are as follows: by unifying the feature space and actively suppressing noise, the heterogeneity problem caused by the different physical principles of bimodal data is effectively solved, and the purity and effectiveness of the fused features in representing real defects are improved.

[0088] Step 4.2: Multi-scale fusion of Feature Pyramid Network (PAFPN);

[0089] The feature pyramid network adopts a path aggregation network structure, which integrates multi-scale feature information through the collaborative operation of three paths: bottom-up, top-down, and lateral connections; specifically:

[0090] Bottom-up approach: After two 3×3 convolution downsampling operations, two levels of low-resolution, high-semantic features (128×32×32 and 64×16×16) are generated to improve the ability to represent the overall outline and category attributes of defects.

[0091] Top-down approach: Upsample low-resolution features to 64×64×64 using transposed convolution to restore spatial resolution and match the original feature scale;

[0092] Lateral connectivity: connecting coarse-grained features of sonar at the same scale With optical fine-grained features Complementary elements are achieved through addition, thus integrating detailed information.

[0093] Finally, the fused feature map is output after a 1×1 convolution with a size of 512×64×64. Its beneficial effects are: through multi-scale feature aggregation, the final feature map has both strong semantic information (which is beneficial for classification) and accurate spatial information (which is beneficial for localization), realizing all-round perception of dam defects from full coverage to precise details.

[0094] Step 5: Input the fused feature map into the decoupled detection head, and output the defect category probability and bounding box coordinates through parallel classification and regression branches respectively; after confidence thresholding and non-maximum suppression processing, the final defect detection result is output; specifically:

[0095] Step 5.1: Decoupling branch processing;

[0096] The classification branch, consisting of a progressive structure of 3×3 convolutional layers, batch normalization (BN) layers, ReLU activation functions, and 1×1 convolutional and sigmoid output layers, focuses on extracting semantic features to output the "defect / non-defect" category probability; the classification loss uses the binary cross-entropy loss function.

[0097]

[0098] in, The true label is N, and the number of samples is N. The model represents the first The probability value of a sample being predicted as "positive" (i.e., having defects / cracks);

[0099] The regression branch, consisting of a 3×3 convolutional layer, a batch normalization (BN) layer, a LeakyReLU activation function, and a 1×1 convolutional and linear output layer, focuses on extracting spatial features and directly outputs the center x-coordinate, center y-coordinate, bounding box width w, and bounding box height h of the crack bounding box, providing accurate spatial parameters for crack localization. The regression loss uses the GIoU loss function. The beneficial effect of the decoupled structure is that it separates the classification and localization tasks, avoids mutual interference between tasks, and can simultaneously optimize detection accuracy and localization accuracy.

[0100] Step 5.2: Post-processing optimization and result output;

[0101] First, a confidence threshold of 0.5 is set to filter the predicted boxes, retaining high-quality predicted boxes with crack probability P≥0.5 and removing low-quality false detection boxes.

[0102] Then, non-maximum suppression (NMS) was used, with the IOU threshold set to 0.3. After sorting the predicted boxes from high to low confidence, the predicted box with the highest confidence was selected as the baseline box. The IOU value between the baseline box and other predicted boxes was calculated. Duplicate predicted boxes with IOU ≥ 0.3 were removed, and non-duplicate predicted boxes with IOU < 0.3 were retained. The process was iterated until the filtering was completed.

[0103] Finally, the candidate predicted box set is converted into a triplet format of "bounding box coordinates (x, y, w, h) + category + confidence (P)" as the final defect detection result output, which can be synchronized to the host computer for visualization annotation and generation of detection report;

[0104] The function and benefits of the post-processing step are as follows: through confidence screening and NMS two-level filtering, while ensuring a high recall rate, the false alarm rate and redundancy of the results are significantly reduced, and clean and accurate detection results are output.

[0105] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the above-described method for composite sensing of dam defects based on coarse and fine acoustic-optical guidance.

[0106] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method for composite sensing of dam defects based on coarse and fine acoustic-optical guidance.

[0107] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A composite sensing method for dam defects based on acoustic-optical coarse-fine guidance, characterized in that, Includes the following steps: Step 1: Control the underwater robot equipped with sonar and optical sensors to move along a preset path, synchronously trigger the sonar and optical sensors to collect the original sonar and optical images of the dam area, and ensure the data is aligned in time and space through sensor calibration and time synchronization. Step 2: Perform mean filtering and grayscale stretching on the original sonar image to obtain preprocessed sonar image data; The original optical image is subjected to contrast enhancement and noise reduction to obtain preprocessed optical image data; The preprocessed sonar image data and optical image data are paired and their sizes are unified to construct a multimodal input tensor; Step 3: Dual-branch feature extraction and environment-adaptive dynamic weight fusion: Step 3.1: Extract sonar data from the multimodal input tensor and input it into the sonar branch network to extract a coarse-grained feature map representing the macroscopic contour. ; Step 3.2: Extract optical data from the multimodal input tensor and input it into the optical branch network to extract fine-grained feature maps representing detailed textures. ; Step 3.3: Acquire the current detection range and image signal-to-noise ratio in real time, and determine the sonar branch weights through the dynamic weight calculation module. With optical branch weight and according to the formula Weighted fusion is performed to obtain intermediate fusion features. ; Step 4: For the intermediate fusion features Channel mapping and modal consistency calibration are performed, and noise suppression is carried out based on a preset interference feature library to obtain denoised features; the denoised features are input into a feature pyramid network for multi-scale fusion to generate the final fused feature map; Step 5: Input the fused feature map into the decoupled detection head, and output the defect category probability and bounding box coordinates through parallel classification and regression branches respectively; after confidence threshold filtering and non-maximum suppression processing, the final defect detection result is output.

2. The method for composite sensing of dam defects based on coarse and fine acoustic-optical guidance according to claim 1, characterized in that: The spatiotemporal alignment in step 1 includes spatial coordinate alignment and time synchronization; The spatial coordinate alignment is achieved through sensor calibration, and a transformation matrix from the sonar and optical coordinate systems to the unified world coordinate system is established using a dual-mode adapter calibration board. The time synchronization is ensured by hardware synchronization triggering and timestamps are recorded during acquisition. When the timestamp deviation between sonar and optical data exceeds a set threshold, linear interpolation is used for data compensation.

3. The method for composite sensing of dam defects based on coarse and fine acoustic-optical guidance according to claim 1, characterized in that: The sonar branch network in step 3.1 is a lightweight residual network based on MobileOne blocks; the optical branch network in step 3.2 is a network structure with MobileOne blocks as front-end units and CSPDarknet as the backbone network.

4. The method for composite sensing of dam defects based on coarse and fine acoustic-optical guidance according to claim 1, characterized in that: In step 3.3, the dynamic weight calculation module determines the weights using the following formula: in: This is the Softmax activation function, used to map parameters to weights in the 0-1 range; and These are the learnable parameters obtained through network training; This represents the real-time image signal-to-noise ratio.

5. The method for composite sensing of dam defects based on coarse and fine acoustic-optical guidance according to claim 4, characterized in that: The weights determined by the dynamic weight calculation module satisfy the following conditions: When the detection distance At that time, sonar branch weight Automatically increases to 55%-70%; When the detection distance And image signal-to-noise ratio At that time, optical branch weight Automatically increases to 60%-75%; When the image signal-to-noise ratio At that time, sonar branch weight It automatically rises to 57%-65%.

6. The method for composite sensing of dam defects based on coarse and fine acoustic-optical guidance according to claim 1, characterized in that: The calibration process in step 4 is as follows: For the intermediate fusion features Channel mapping and decomposition are performed to obtain the calibrated sonar modal features and optical modal features; A modal consistency loss function is introduced to constrain the alignment of the calibrated sonar modal features with the optical modal features in a unified space; Calculate the Euclidean distance between the calibrated feature and the template in the preset interference feature library. If the distance is less than the set threshold, it is determined to be an interference feature and is removed.

7. The method for composite sensing of dam defects based on coarse and fine acoustic-optical guidance according to claim 1, characterized in that: The feature pyramid network in step 4 adopts a path aggregation network structure, which integrates multi-scale feature information through the coordinated action of three paths: bottom-up, top-down, and lateral connection.

8. The method for composite sensing of dam defects based on coarse and fine acoustic-optical guidance according to claim 1, characterized in that: The decoupling detection head in step 5 includes: The classification branch, consisting of convolutional layers, batch normalization layers, activation functions, and a sigmoid output layer, is used to output the probability of the "defect / non-defect" category. The regression branch, consisting of convolutional layers, batch normalization layers, activation functions, and linear output layers, is used to output the center coordinates, width, and height of the defect bounding box. The classification loss uses the binary cross-entropy loss function, and the regression loss uses the GIoU loss function.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the composite sensing method for dam defects based on coarse and fine acoustic-optical guidance as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the composite sensing method for dam defects based on coarse and fine guidance as described in any one of claims 1 to 8.

Images