Underwater building defect detection method based on adaptive deep learning model training

Abstract

The invention relates to the technical field of underwater image recognition, in particular to an underwater building defect detection method based on adaptive deep learning model training, which comprises the following steps: acquiring underwater image data as a first training set, and generating a second training set containing defect prototype characterization and environment feature vectors by using a generative adversarial network; constructing a self-adaptive detection model, wherein the self-adaptive detection model comprises a trunk feature extractor, a multi-scale feature pyramid, a self-adaptive attention module and a defect knowledge embedding module; performing course sorting on the training data by using the environment feature vector and the defect prototype representation; optimizing by using standard detection loss, prototype comparison loss and auxiliary loss; adversarial training is introduced, alignment and fine adjustment are continuously carried out on defect prototype embedded vectors, and finally a trained adaptive detection model is obtained; and defect detection of the underwater building is realized by using the trained adaptive detection model, and finally accurate and efficient detection of underwater defects is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater image recognition, and in particular to a method for detecting underwater building defects based on adaptive deep learning model training. Background Art

[0002] Underwater infrastructure is vital to the economy and society, and its structural integrity monitoring is in urgent need. Traditional underwater inspection methods rely on divers, which have inherent limitations such as high cost, high risk, subjective results, and physiological constraints. Early automated image processing technologies were also limited by noise, varying lighting conditions, and complex defect morphologies in underwater environments.

[0003] Deep learning has shown great potential in recent years, but the unique characteristics of underwater environments, such as dynamic changes, data scarcity, and poor image quality, including color distortion and blur, pose significant challenges to the direct application of standard models. Underwater optical imaging is affected by factors such as light absorption and scattering, water turbidity, uneven illumination, and biofouling, resulting in significant degradation of image quality.

[0004] Current deep learning technology applications in underwater defect detection lack a model training method that can dynamically adjust and effectively learn under the conditions of variable underwater image quality and a continued shortage of high-quality annotated data. At the same time, when faced with evolving defect characteristics or new operating environments, it is difficult to perform rapid and incremental knowledge updates and model optimization without catastrophic forgetting. This greatly restricts its potential for long-term, reliable deployment and application on resource-constrained autonomous underwater platforms.

[0005] To this end, an underwater building defect detection method based on adaptive deep learning model training is proposed. Summary of the Invention

[0006] The present invention aims to provide an underwater structure defect detection method based on adaptive deep learning model training. This method achieves accurate underwater structure defect detection through iterative defect carving, adversarial generative network data enhancement, adaptive model construction, and multi-strategy training optimization. The method includes obtaining underwater image data as a first training set, using a generative adversarial network to generate a second training set containing defect prototype representations and environmental feature vectors; constructing an adaptive detection model, including a backbone feature extractor, a multi-scale feature pyramid, an adaptive attention module, and a defect knowledge embedding module; using the environmental feature vectors and defect prototype representations to sort the training data; optimizing the training data using standard detection loss, prototype comparison loss, and auxiliary loss; introducing adversarial training to continuously align and fine-tune the defect prototype embedding vectors, ultimately obtaining a trained adaptive detection model; and using the trained adaptive detection model to detect underwater structure defects.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] The underwater structure defect detection method based on adaptive deep learning model training includes:

[0009] Multimodal underwater image data is obtained as a first training set, and a generative adversarial network is used to perform data enhancement and feature extraction on the first training set to obtain a second training set containing corresponding defect prototype representations and environmental feature vectors;

[0010] Build an adaptive detection model, which includes a backbone feature extractor, a multi-scale feature pyramid, an adaptive attention module, and a defect knowledge embedding module that uses defect prototype representation and performs dynamic optimization.

[0011] The training data is sorted using environmental feature vectors and defect prototype representations. Optimization is performed using a combination of standard detection loss, prototype contrast loss, and auxiliary loss. The auxiliary loss optimizes the objective by minimizing the difference between features in a predefined metric space. Adversarial training is introduced at the feature layer of the adaptive detection model, continuously aligning and fine-tuning the defect prototype embedding vectors in the defect knowledge embedding module to obtain a trained adaptive detection model.

[0012] The trained adaptive detection model is deployed in the equipment to realize defect detection of underwater buildings.

[0013] Preferably, the environment encoder in the generative adversarial network is designed to receive and process multimodal underwater image data synchronously acquired and aligned from at least two types of sensors, wherein the sensors include an imaging sonar device and at least one optical imaging device; the environment encoder processes each modal data through an independent feature extraction branch, including acoustic features and visual features, and deeply interacts and fuses the features extracted by each independent branch through a multimodal feature fusion module to output a fixed-dimensional environment feature vector, wherein the environment feature vector contains acoustic characteristics and visual environment information.

[0014] Preferably, the defect prototype decoder in the generative adversarial network converts the defect prototype representation into a defect prototype latent vector, which is fused with the environmental feature vector through a cross-attention mechanism. The fused vector is upsampled through multiple layers of transposed convolution to gradually improve the spatial resolution. During the upsampling process, the weights of the convolution layer and the scaling parameters and translation parameters of the batch normalization layer are dynamically generated and applied by a small auxiliary network based on the environmental feature vector to ensure that the generated defect instances meet the imaging characteristics of the target underwater environment in terms of morphology, edges, acoustic reflection characteristics and texture.

[0015] Preferably, the adaptive attention module in the adaptive detection module has a built-in fusion gating unit, which receives the global feature vector obtained by the global average pooling operation and the semantic feature vector extracted from the multi-scale feature pyramid, concatenates the global feature vector and the semantic feature vector, and inputs them into a gating network including a fully connected layer and a nonlinear activation function. The output layer activation value of the gating network is normalized by the Softmax function to form a set of dynamic weighting coefficients with the same number as the levels of the multi-scale feature pyramid, and the dynamic weighting coefficients are applied to the feature maps of the corresponding levels respectively.

[0016] Preferably, the defect knowledge embedding module in the adaptive detection model uses defect prototype representation to initialize a set of vectors whose dimensions match the image feature embedding dimensions as defect prototype embedding vectors; during forward propagation of the network, for the input image region features, the defect knowledge embedding module calculates the similarity scores between each defect prototype embedding vector through a learnable linear transformation, uses the similarity scores to perform weighted summation on the defect prototype embedding vectors, and fuses the results with the original image region features. At the same time, during the backpropagation process, the defect prototype embedding vector is updated based on the gradient of the loss function.

[0017] Preferably, the course sorting of training data using environmental feature vectors and defect prototype representations includes: dividing all underwater image training samples according to the weighted distance measurement between their corresponding environmental feature vectors and a predefined set of reference environmental feature vectors of different discrete levels to form environmental difficulty buckets; within each environmental difficulty bucket, sorting the samples from easy to identify to difficult to identify based on the comprehensive defect recognizability evaluation results of the training samples; the comprehensive defect recognizability evaluation includes, for the original annotated training samples, considering the initial classification confidence of the defect prototype representation associated with the training samples and the frequency with which the training samples were correctly identified in the initial training stage; for the enhanced samples generated by the generative adversarial network, determining them based on the known recognizability characteristics of the defect prototype used when they were generated.

[0018] Preferably, the prototype contrast loss is specifically implemented as follows: the image feature embedding extracted from the adaptive detection model and representing each defect area to be evaluated is used as an anchor point; within the current training data batch, for the anchor point, a positive sample feature embedding is selected, which belongs to the same defect category as the anchor point and whose own feature embedding is closest to the defect prototype representation corresponding to the defect category; in addition, a negative sample feature embedding is selected, which belongs to a different defect category from the anchor point and whose own feature embedding is closest to the prototype representation of the defect category to which it belongs; the prototype contrast loss minimizes the L2 distance between the anchor point and the positive sample feature embedding, and maximizes the L2 distance between the anchor point and the negative sample feature embedding, ensuring that the difference between the L2 distance between the anchor point and the negative sample feature embedding and the L2 distance between the anchor point and the positive sample feature embedding is greater than a preset boundary margin.

[0019] Preferably, the introduction of adversarial training in the feature layer of the adaptive detection model includes: constructing a discriminator network composed of fully connected layers; the discriminator network receives the feature vectors output by the convolutional layer blocks in the adaptive detection model and formed by global average pooling as input; the training goal of the discriminator network is to output a binary classification prediction probability to indicate that the input features it receives are input images derived from simple and complex underwater environment imaging characteristics, where the simplicity and complexity of the environment are divided according to the definition of environmental difficulty buckets in course learning; the loss function of the adaptive detection model includes the adversarial loss transmitted from the discriminator network through the gradient reversal layer; the adversarial loss guides the adaptive detection model to optimize the feature extraction process, and through the optimization, when the adaptive detection model processes images derived from different underwater environment imaging characteristics, the intermediate layer feature vectors generated are closer in statistical distribution, thereby enabling the adaptive detection model to extract essential defect features that are insensitive to environmental changes.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1. A generative adversarial network (GAN) is used to perform data augmentation and feature extraction on multimodal underwater image data. The environment encoder deeply fuses acoustic and visual features, outputting an environment feature vector containing both acoustic and visual environmental information. This enhances perception of complex underwater environments, enabling a more comprehensive representation of the underwater environment and providing effective guidance for subsequent networks.

[0022] 2. The present invention constructs an adaptive detection model, which includes a backbone feature extractor, a multi-scale feature pyramid (FPN), an adaptive attention module and a defect knowledge embedding module. Among them, the adaptive attention module has a built-in fusion gating unit, which dynamically weights and fuses global and local saliency features according to the global context, optimizes the multi-scale feature fusion method, and enhances the model's adaptability to complex underwater environments and the recognition accuracy of different defects. At the same time, the defect knowledge embedding module effectively integrates the defect prior knowledge extracted from the generative adversarial network into the detection network, and enables the image features to interact with the defect prior knowledge through the attention mechanism, thereby improving the recognition accuracy of specific defect types and enhancing the sensitivity and discrimination of defect features, especially when distinguishing visually similar or rare defects.

[0023] 3. The training data is sorted using environmental feature vectors and defect prototype representations, allowing the model to learn from easy to difficult tasks. This improves learning stability and the model's final performance. Optimization is performed using a combined loss function consisting of a standard detection loss, a prototype contrast loss to guide feature alignment with prototypes, and an auxiliary loss for environmental feature consistency. The prototype contrast loss and its adversarial dynamic margin adjustment mechanism exert adaptive learning pressure, driving the model to learn highly discriminative and robust feature representations for different defect types, effectively reducing confusion between different defect categories and further improving classification accuracy.

[0024] 4. Introducing adversarial training at the feature layer of the adaptive detection model enables the model to learn to extract essential defect features that are insensitive to environmental changes, significantly enhancing the model's environmental robustness and practical application performance. Simultaneously, during training, the defect prototype embedding vectors in the defect knowledge embedding module are continuously aligned and fine-tuned, allowing them to dynamically evolve to more accurately represent various defects, improving prototype comparison loss efficiency and the accuracy of final defect identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flow chart of a method for underwater building defect detection based on adaptive deep learning model training provided by an embodiment of the present invention;

[0026] Figure 2 A flowchart of sorting training sample courses provided by an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of the total training loss provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0029] See also Figures 1 to 3 , the present invention provides an underwater building defect detection method based on adaptive deep learning model training, the technical solution is as follows:

[0030] Example 1:

[0031] In order to improve the detection accuracy of underwater building defect detection, the underwater building defect detection method based on adaptive deep learning model training provided by the present invention is introduced. The specific flow chart is as follows: Figure 1 shown.

[0032] Underwater image data is obtained as the first training set, and a generative adversarial network is used to perform data enhancement and feature extraction on the first training set to obtain a second training set containing corresponding defect prototype representations and environmental feature vectors.

[0033] The underwater images obtained mainly consist of two parts

[0034] Imaging Sonar: Sonar can operate effectively in turbid water and low visibility environments, providing critical structure outline and medium- and long-range detection information.

[0035] Low-light-level camera: used to improve imaging capabilities under low-light conditions, obtain optical images of the target area, and provide supplementary information for the fine features of defects.

[0036] During the acquisition process, sonar images and low-light-level camera images are synchronously acquired and spatially aligned. The low-light-level camera and sonar equipment are integrated on the same underwater robot (ROV) platform. Using the ROV's positioning and attitude information, combined with the known relative position and field of view parameters of the sensors, preliminary spatial registration of the sonar and optical images is performed to obtain multimodal image data of the same target area. Image acquisition environments are diverse, ranging from relatively clear to highly turbid waters. The target surface may be covered by marine organisms or sediment. Acoustic equipment is susceptible to various noise interference when acquiring signals underwater; optical imaging also faces problems such as light scattering, absorption, and color distortion.

[0037] The initial dataset, or the first training set, consisted of raw underwater sonar images and optical images from a low-light-level camera system. These images were manually screened and annotated with defects, including defect categories and bounding boxes. The sonar data was format converted and geometrically corrected; the optical images were dehazed and color corrected; and all images were resized and initially denoised.

[0038] The environment encoder in the generative adversarial network receives underwater sonar images and low-light-level optical images as input. First, a convolutional neural network (CNN) layer group extracts acoustic features from the sonar image and visual features from the optical image, respectively. Subsequently, a cross-modal attention module is introduced. The sonar feature map serves as the query, and the optical feature map serves as the key and value. The similarity between the query and the key is calculated to assign weights to the optical feature map values, resulting in an optical feature representation modulated by the sonar information. Finally, these two modulated feature maps are concatenated with the original feature map and processed through a convolutional layer to form a more informative and discriminative multimodal environment feature vector. This fused multimodal feature vector is then processed through a self-attention mechanism to aggregate global environmental context. Finally, the environment encoder outputs a fixed-dimensional environment feature vector. By integrating information from both sonar and optical modalities, this environment feature vector provides a more comprehensive representation of the underwater environment. Specifically, the environmental feature vector incorporates acoustic environment characteristics, including the overall acoustic reflection intensity distribution statistics, turbidity-related texture details reflected by high-frequency components derived from Fast Fourier Transform analysis, and signal inhomogeneity indicators calculated by analyzing the skewness and kurtosis of the signal intensity histogram. It also incorporates optical environment characteristics, including underwater light attenuation, color distortion, image contrast, and scattering effect intensity assessed from color channel histograms. This approach provides the environmental encoder with a more robust and detailed understanding of the underwater environment.

[0039] By intelligently combining the penetration of sonar and the detail of optics, perception of complex underwater environments is significantly enhanced. A cross-modal attention mechanism dynamically adapts to environmental changes, helping to enhance the recognition of ambiguous defects. More accurate environmental feature vectors provide more effective guidance for subsequent networks, enabling defect sample generation and knowledge embedding. This contributes to improving the overall accuracy and robustness of defect detection, addressing the challenge of insufficient information from a single sensor.

[0040] In a generative adversarial network (GAN), the defect prototype decoder is responsible for generating defects based on the characteristics of the underwater environment. Its workflow begins by receiving two inputs: a defect prototype latent vector, representing common defect categories for different underwater structures, converted by the encoder; and an environment feature vector, output by the environment encoder, representing the comprehensive characteristics of the current underwater environment. The decoder first fuses these two input vectors using a cross-attention mechanism. This attention mechanism enables effective interaction between the feature information of the defect prototype and the environment, allowing defect generation to simultaneously consider the inherent morphology of the defect and the influence of the current underwater environment. The fused vectors after cross-attention are then input into a network consisting of multiple layers of transposed convolutions. This network gradually increases the spatial resolution of the feature map through layer-by-layer upsampling, ultimately aiming to directly generate a fused image region with simulated defects. A key technical aspect of this process lies in dynamic parameter generation. During the upsampling process, the weights of the transposed convolutional layers within the decoder network, as well as the scaling and translation parameters of the subsequent batch normalization layer, are not fixed. Instead, they are calculated and dynamically applied in real time by a small auxiliary neural network based on the current environment feature vector. This auxiliary network learns a mapping from the environment feature vector to the decoder layer parameters.

[0041] This dynamic parameter adjustment mechanism ensures that the resulting defect instances, represented by their fused images, faithfully conform to the comprehensive imaging characteristics of the target underwater environment. This includes the expected acoustic reflection characteristics of the defect area, edge clarity or blurriness, color representation, and texture features. In this way, the generative adversarial network can generate highly realistic and environmentally adaptive defect data, providing high-quality samples for subsequent defect detection model training.

[0042] Build an adaptive detection model, which includes a backbone feature extractor, a multi-scale feature pyramid, an adaptive attention module, and a defect knowledge embedding module that uses defect prototype representation and performs dynamic optimization.

[0043] The backbone feature extractor is responsible for extracting hierarchical deep features from the input fused image. In this embodiment, EfficientNet-B3 was selected as the backbone network. This backbone network has been pre-trained on a large image dataset, which helps the model learn universal image feature representations and accelerates convergence on underwater defect detection tasks. The backbone network outputs feature maps at different stages, with varying spatial resolutions and semantic information strengths, laying the foundation for subsequent multi-scale feature analysis. Feature maps at shallower layers retain more spatial detail, while feature maps at deeper layers contain richer semantic information.

[0044] To effectively detect underwater building defects of varying sizes, the adaptive detection model constructs a multi-scale feature pyramid (FPN). By combining feature maps output by the backbone network at different depths, FPN constructs a series of feature layers with rich semantic information and multi-scale information. Specifically, FPN uses a top-down approach, amplifying high-level semantic features through upsampling and laterally connecting them with feature maps from lower layers with higher spatial resolution but weaker semantic information. These features are then aligned through 1x1 convolutions and then concatenated. The resulting multi-level feature pyramid, with each level corresponding to a different receptive field and scale, enables the model to simultaneously focus on large objects and small defects.

[0045] The core function of the adaptive attention module is to dynamically weight and fuse global and local saliency features based on global context, thereby enhancing the model's ability to perceive important features. This module incorporates a built-in fusion gating unit. Specifically, the fusion gating unit first receives a global context feature vector output from the deep feature layers of the backbone network, following a global average pooling operation. It also receives a feature vector extracted from the feature map at the top level of the multi-scale feature pyramid. These two feature vectors are concatenated along the channel dimension and then input into a gating network consisting of several fully connected layers and nonlinear activation functions. The activation values ​​of the output layer of the gating network are normalized using a softmax function to form a set of dynamic weighting coefficients. This set of coefficients has the same number of levels as the multi-scale feature pyramid. Finally, these dynamically generated weighting coefficients are applied to the local saliency feature maps at the corresponding level of the multi-scale feature pyramid. By broadcasting the coefficients to each spatial location in the feature map and performing element-wise multiplication, this achieves adaptive weighted fusion of features at different scales. This enables the model to dynamically adjust its attention to features at different scales based on image content.

[0046] The fusion gating unit in the adaptive attention module optimizes the fusion method of multi-scale features through its intelligent dynamic weighting mechanism, enhances the model's adaptability to complex underwater environments and the recognition accuracy of different defects, thus providing a more effective and robust solution for underwater building defect detection.

[0047] The defect knowledge embedding module aims to effectively integrate defect prior knowledge extracted from the generative adversarial network into the detection network. Using an attention mechanism, it enables image features to interact with prior defect knowledge, thereby improving the accuracy of defect type recognition. Specifically, a set of learnable embedding vectors, representing common defect categories of different underwater structures, extracted from the generative adversarial network, is initialized as defect prototype embedding vectors. The dimensions of these embedding vectors match the embedding dimensions of the feature-level image features output by the FPN in the adaptive detection model. During the network's forward propagation, the defect knowledge embedding module calculates the similarity between each defect prototype embedding vector and the image region features extracted from a specific layer or region of interest (ROI) of the FPN. This similarity is then calculated using a learnable linear transformation to generate a similarity score. The learnable linear transformation calculates similarity as follows: First, the module performs a linear transformation on the input image region features. This transformation is performed by a first set of learnable parameters, namely weights and biases, which are adjusted and optimized during the overall model training process via the backpropagation algorithm. Its function is to map the image region features into a new transformed image feature representation. At the same time, the module independently performs a similar linear transformation on each defect prototype embedding. This transformation is controlled by a second set of independent learnable parameters: weights and biases. The goal is to map each prototype embedding into a new transformed prototype feature representation. The learnable parameters used in these two linear transformations are learned by the model based on training data, aiming to project the original features into a shared feature space that is more conducive to subsequent comparison. After obtaining the transformed image features and the transformed prototype features, the module calculates similarity by taking the dot product between them. This dot product result is quantized into a similarity score. These similarity scores are normalized using a Softmax function and used as attention weights. These attention weights are then weighted and summed across the corresponding defect prototype embeddings to produce a context vector that incorporates prior defect knowledge. Finally, this context vector is fused with the original image region features, concatenated, and processed through a fully connected layer. During backpropagation, based on the gradient of the overall loss function, not only are the network parameters for image feature extraction updated, but these learnable defect prototype embeddings are also continuously aligned and fine-tuned to more accurately represent the essential characteristics of each defect.

[0048] By using defect prototypes extracted from the generative adversarial network as learnable prior knowledge and leveraging the attention mechanism to interact with image features, the model can more accurately match known defect patterns. This enhances sensitivity and discrimination to defect features, particularly when distinguishing visually similar or rare defects. It also improves the model's generalization and robustness in the face of sample imbalance, as the model can leverage prototype knowledge even with a small number of training samples. Furthermore, good initialization helps accelerate model convergence, improves training efficiency, and provides a degree of explainability for model decisions.

[0049] The training data is sorted using environmental feature vectors and defect prototype representations. A combined loss function, including standard detection loss, prototype contrast loss to guide feature-prototype alignment, and auxiliary loss, is used for optimization. Adversarial training is introduced at the feature layer of the adaptive detection model to continuously align and fine-tune the defect prototype embedding vectors in the defect knowledge embedding module, resulting in a trained adaptive detection model.

[0050] In order to improve the training efficiency and model performance, before using the second training set to train the adaptive detection model, the courses are first sorted. The specific course sorting flow chart is as follows: Figure 2 As shown in Figure 2, the sorting process fully utilizes the environmental feature vectors and defect prototype representations associated with each image in the second training set. The specific implementation is divided into two main stages. First, all training samples, including original images and enhanced images generated by the generative adversarial network, are preliminarily grouped according to their corresponding environmental feature vectors. This is achieved by calculating the weighted distance between the environmental feature vector of each sample and a set of predefined reference environmental feature vectors. These reference vectors represent typical environmental conditions at different discrete levels, ranging from extremely clear underwater environments to extremely turbid underwater environments. The reference environmental vectors are obtained through a data-driven clustering method. Specifically, a representative and diverse set of underwater images (a calibration dataset specially prepared for this purpose) is first selected and processed using the environmental encoder in the generative adversarial network to extract their corresponding environmental feature vectors. Subsequently, the K-Means clustering algorithm is applied to these extracted environmental feature vectors, where the number of clusters K is set to the same as the number of desired discrete levels of environmental difficulty. The centers of the K clusters generated by cluster analysis are then established as the predefined reference environmental feature vectors. This approach ensures that the reference vector is objectively generated based on the data and utilizes the same feature space as the model's subsequent learning process, providing a robust and consistent basis for environmental difficulty assessment during course learning. Finally, based on the distance metric, the samples are divided into several environmental difficulty buckets, including clear environment buckets, moderately turbid environment buckets, and highly turbid environment buckets.

[0051] Within each environmental difficulty bucket, samples are then carefully ranked from easily identifiable defects to difficult defects. This ranking is based on a combination of the following: for each sample, the initial classification confidence corresponding to its associated defect prototype representation is considered; for original annotated data, the historical frequency of correct identification during the initial training phase is considered; and for enhanced images generated by a generative adversarial network, the known recognizability characteristics of the prototype used to generate the defect are considered. Together, these factors form the criteria for determining sample learning priority within that environmental difficulty level.

[0052] To determine the ease of recognition of each defect prototype, this embodiment utilizes an adaptive multi-scale model to evaluate the learning performance of real, labeled defect data from the second training set during the initial training phase. Specifically, by analyzing the adaptive detection model's performance metrics for different defect categories during initial training on these real defect samples, such as recognition accuracy, learning speed, and confidence, a quantitative ease of recognition score is assigned to each defect category. Prototypes corresponding to defect categories that the model can quickly and accurately recognize in the early stages receive higher ease of recognition scores, while prototypes that fail to do so receive lower scores. When the generative adversarial network generates defect samples for data augmentation based on the prototypes, these synthesized samples inherit the ease of recognition scores of their parent prototypes. This score is then used to carefully sort these synthesized samples from easy to difficult within the same environment difficulty bucket during the second phase of the learning process. This approach is not only data-driven, but also fully leverages existing real-world labeled data and core detection models. It ensures a high correlation between ease of recognition assessment and the model's own learning characteristics, making the entire training process more intelligent and efficient, and forming a closed-loop logic that learns difficulty from data and then guides subsequent learning based on the learned difficulty.

[0053] This two-stage ranking strategy forms the final training course sequence. This approach allows the model to initially be exposed to samples from simple environments where defects are easily identifiable. As training progresses, it gradually transitions to samples from complex environments where defects are difficult to identify. This simulates the human learning process of moving from easy to difficult, helping to improve learning stability and the model's ultimate performance.

[0054] The training of the adaptive detection model is optimized using a carefully designed combined loss function, which includes the standard detection loss, the innovative prototype comparison loss, and the auxiliary loss of environmental feature consistency. The specific total loss diagram is shown in the figure below. Figure 3 shown.

[0055] A standard detection loss is used to supervise the model's defect classification and localization. Based on the specific detection head design, a Focal Loss is used for defect category classification, and a GIoU Loss is used for defect bounding box regression. Additionally, a prototype contrast loss is used to ensure that the image feature embeddings of the defect region extracted by the adaptive detection model align with the corresponding defect prototype and distinguish them from prototypes of other categories. The specific implementation is as follows: the image feature embedding extracted from the adaptive detection model and representing each defect area to be evaluated is defined as an anchor point; within the current training data batch, for the anchor point, a positive sample feature embedding is selected, the positive sample feature embedding belongs to the same defect category as the anchor point, and the L2 distance between the feature embedding of the positive sample feature embedding and the defect prototype representation corresponding to the defect category to which the anchor point belongs is minimized; a negative sample feature embedding is selected, the negative sample feature embedding belongs to a different defect category than the anchor point, and the L2 distance between the feature embedding of the negative sample feature embedding and the feature embedding of the anchor point is minimized; the goal of the prototype contrast loss is to minimize the L2 distance between the anchor point and the positive sample feature embedding, while maximizing the L2 distance between the anchor point and the negative sample feature embedding, ensuring that the difference between the two is greater than a preset margin. This mechanism forces the model to learn highly discriminative features between different defect types and is guided by the learned prototypes.

[0056] The preset boundary margin is specifically a dynamically adjusted boundary margin. The dynamic boundary margin adjustment method is to introduce an adversarial learning mechanism, and set this margin value by constructing a special margin adjuster network to perform a dynamic game with the main feature learning network. In this framework, the margin adjuster is responsible for actively generating a boundary margin that is most challenging but still achievable for the main network, aiming to encourage the main network to learn better feature representations; while the main feature learning network strives to optimize feature embedding under the conditions of this dynamically changing margin. The collaborative training of these two networks makes the setting of the boundary margin itself an adaptive learning process and a dynamic adversarial process, rather than a passively adjusted hyperparameter, thereby continuously providing moderate learning pressure to the main network to drive it to learn more discriminative and robust features for complex situations.

[0057] Furthermore, the calculation of the auxiliary loss aims to achieve cross-environmental representation stability. This approach involves processing paired samples—specifically, samples containing the same defect type from different real-world observation environments, as well as multiple variations of the same defect generated by a generative adversarial network to simulate different environmental conditions—to extract intermediate-level features from each pair in the adaptive detection model. This loss is then optimized by minimizing the difference between these paired features in a predefined metric space. Specifically, the L2 distance is used to measure and minimize the difference between feature pairs, aiming to minimize the angular difference between them. This direct penalty for feature differences drives the model to learn more essential defect representations that are less sensitive to environmental changes, striving to ensure that these features remain as consistent as possible across different environments. Furthermore, when environmental factors inevitably cause systematic changes in features, this loss mechanism helps ensure that these changes follow a stable pattern that the model can learn and effectively compensate for, ultimately ensuring reliable extraction of core defect information. This directly enhances the model's robustness to variations in water clarity, illumination, and other environmental factors.

[0058] The total loss function is the weighted sum of various losses, which are mainly composed of standard detection loss, prototype comparison loss, and environmental feature consistency auxiliary loss. The weight of each loss is a hyperparameter that needs to be adjusted.

[0059] With the precise supervision of standard detection loss, efficient classification and accurate positioning of underwater defects are achieved. Adaptive learning pressure is applied through an innovative prototype comparison loss and its adversarial dynamic margin adjustment mechanism, driving the model to learn highly discriminative and robust feature representations for different defect types. Auxiliary loss based on environmental feature consistency ensures that features extracted from similar defect samples under different environmental conditions remain stable, significantly enhancing the model's robustness in the face of complex underwater environmental factors. Overall, this design aims to comprehensively improve the accuracy, robustness, and generalization capabilities of the detection model to meet the challenges of real-world underwater monitoring.

[0060] Furthermore, in order to further enhance the feature robustness of the adaptive detection model to environmental changes, adversarial training is introduced in the feature layer of the model to ensure stable performance under variable underwater conditions. Specifically, a discriminator network is constructed. The discriminator is usually composed of multiple fully connected layers. It receives the convolutional layer block in the adaptive detection model, such as an output layer of a feature pyramid network or a deep layer of the backbone network, and outputs and globally average pooled feature vectors as input. The training goal of the discriminator is to output a binary classification prediction probability to indicate whether the input feature it receives is derived from an image with simple underwater environment visual or acoustic characteristics, or from an image with complex underwater environment visual or acoustic characteristics. The simplicity and complexity of the environment can be divided according to the definition of environment bucketing in the previous course.

[0061] In the adaptive detection model's loss function, an adversarial loss is passed from the discriminator via a gradient reversal layer. The gradient reversal layer reverses the sign of the gradient from the discriminator during backpropagation. This adversarial loss forces the adaptive detection model to generate intermediate-layer feature vectors when processing images with complex underwater environments, appearing to the discriminator to belong to the same distribution as feature vectors generated when processing images with simple underwater environments. This tricks the discriminator into being unable to distinguish the environmental type from which the features originate. This forces the adaptive detection model to learn to extract more essential defect features that are insensitive to environmental changes, significantly improving its performance in real-world applications. During training, the learnable defect prototype embeddings in the defect knowledge embedding module are continuously aligned and fine-tuned based on the gradient of the overall loss function. These prototypes are not just fixed prior knowledge but can dynamically evolve based on training data and task feedback. This enables a more accurate and detailed representation of the essential characteristics of various defects, thereby better serving the prototype contrast loss and the ultimate defect recognition task.

[0062] Through the synergy of the aforementioned course learning, combined loss function optimization, innovative feature-level adversarial training, and continuous optimization of the defect prototype embedding vector, the adaptive detection model was fully trained. After training, the result was an adaptive detection model with high detection accuracy and strong environmental robustness for underwater structure defects, which can be used for subsequent deployment and actual inspection tasks.

[0063] By introducing adversarial training at the feature level, the adaptive detection model learns essential defect characteristics that are unaffected by changes in the underwater environment, significantly enhancing the model's environmental robustness and practical application performance. Furthermore, the defect prototype embedding vectors are continuously aligned and fine-tuned during training, allowing them to dynamically evolve to more accurately represent various defects, thereby improving the efficiency of prototype comparison loss and the accuracy of final defect identification. These two factors, combined with curriculum learning and combination loss, ultimately ensure that the adaptive detection model achieves high detection accuracy and strong environmental adaptability, making it capable of handling complex deployment tasks.

[0064] The trained adaptive detection model is deployed in the equipment to realize defect detection of underwater buildings.

[0065] To achieve an efficient adaptive detection model during the training phase, dynamic pruning and structural adaptation techniques are introduced during training. The core of this technique is to periodically adjust the network's sparse connectivity structure in real time during model training iterations. Specifically, at regular intervals, the system identifies and removes connections that contribute less to the current defect detection task based on pre-set importance metrics, including the size of neural connection weights and their gradient information, thereby achieving network sparsification. Furthermore, to maintain the model's learning capacity and explore more optimal sparsity structures, this technique allows some previously removed connections to be reactivated based on their subsequent potential importance, as determined by gradient trends. Furthermore, the system allows the network to grow new connections in locations that contribute most to improved model performance. This entire dynamic pruning and connection regeneration process is constrained by a preset target network sparsity. In this way, while the adaptive detection model learns to complete defect detection tasks, its own network topology is continuously adaptively optimized. Compared with traditional post-training pruning methods, this technology of dynamically adjusting and optimizing sparse structures during training gives network models greater flexibility and adaptive adjustment opportunities, thus having the potential to achieve high model compression rates while maintaining a small loss in detection accuracy. Ultimately, an efficient and high-performance sparse adaptive detection model is obtained, laying the foundation for subsequent deployment and efficient inference on resource-constrained platforms.

[0066] During the actual inspection phase, the adaptive inspection model, optimized and trained using dynamic pruning and structural adaptation techniques, is first deployed on an underwater robot or corresponding analysis platform. During operation, the robot's onboard imaging sonar and low-light-level camera simultaneously capture images of the target underwater structure. The acquired bimodal image data undergoes multimodal fusion processing consistent with the training phase and is then fed as unified input into the deployed adaptive inspection model. Leveraging its powerful feature extraction, multi-scale analysis, and defect recognition capabilities learned during training, the model rapidly analyzes and processes the fused data, ultimately outputting the detected defect type, precise spatial location, and associated confidence information.

[0067] This method fuses acoustic and optical multimodal image information and applies a cross-modal attention mechanism to provide the model with comprehensive scene information and subtle defect details far beyond those of a single sensor, laying a solid data foundation for subsequent accurate identification. The core adaptive multi-scale detection model integrates a dynamic attention module and a defect knowledge embedding module. The dynamic attention module guides the model to adaptively focus on the image feature regions and scales most critical for defect identification, effectively eliminating irrelevant information interference. The defect knowledge embedding module utilizes defect types extracted and continuously optimized from a generative adversarial network to significantly enhance the model's accuracy and ability to distinguish defect types, especially fine-grained defects. At the training level, a standard detection loss is directly optimized to improve classification accuracy and localization precision. The innovative prototype contrast loss and its accompanying adversarial dynamic margin adjustment mechanism strongly drive the model to learn highly discriminative defect feature representations, effectively reducing confusion between different defect categories and further improving classification accuracy. The combined application of these mechanisms ensures that the model not only accurately identifies and precisely locates underwater structure defects, but also maintains high-precision detection performance in complex and changing underwater environments.

[0068] Example 2:

[0069] In this embodiment, underwater image data is collected from three different types of sensors, which are integrated into the same underwater robot (ROV / AUV) platform to ensure temporal synchronization and spatial alignment during the acquisition process:

[0070] Imaging sonar equipment: provides the ability to detect the outline of building structures and larger defects at medium and long distances and in turbid water bodies.

[0071] Low-light-level camera: used to obtain optical images of target areas under insufficient lighting conditions, providing visual information for fine features of defects.

[0072] Muddy water clearing camera equipment: This type of equipment is designed to improve image quality, enhance contrast, and suppress scattering in turbid water bodies. It can obtain structural details and defect characteristics that are difficult to show with traditional optical imaging methods.

[0073] To fully leverage the complementary information from these three modal data, this embodiment employs an enhanced multimodal feature fusion mechanism before inputting the data into the environment encoder of the generative adversarial network. First, initial feature maps are extracted from sonar images, low-light optical images, and images captured by the muddy water clearing camera, using separate convolutional neural network (CNN) branches. The network structure of each branch is optimized for the data characteristics of the corresponding modality. Next, a multimodal fusion module is introduced. This module can be designed as an attention-based structure, such as an extended cross-modal attention network or a multi-head attention layer, capable of simultaneously receiving feature maps from three independent branches. This module learns the complex correlations and complementarities between features from different modalities, dynamically assigns weights to features from different modalities, and performs deep fusion. For example, the enhanced detail features from the muddy water clearing image can be used to enhance blurred areas in the low-light image, while the structural contour information provided by the sonar can provide global context for understanding the optical image. The fused multimodal features are then processed through several convolutional layers to ultimately form a fixed-dimensional environment feature vector that is richer in information and more comprehensive in its representation of the underwater environment. This vector will be used as the output of the environment encoder in the generative adversarial network.

[0074] Through this trimodal fusion strategy, the generated environmental feature vector can more accurately capture the complexity of the underwater environment, including the enhanced information provided by the muddy water clearing equipment under harsh visual conditions, laying a solid foundation for subsequent defect sample generation and model training.

[0075] The environment encoder in the generative adversarial network receives input from three sensor types: sonar, low-light cameras, and muddy water clearing cameras. Data is extracted independently and then deeply fused using a multimodal attention mechanism to form a unified environmental representation. By integrating enhanced information from muddy water clearing cameras under harsh visual conditions, as well as complementary data from sonar and low-light images, the encoder's output, an environmental feature vector, more accurately and meticulously captures the complexities and nuances of extreme underwater environments.

[0076] The defect prototype decoder generates specific defect instances based on the latent representation of the defect prototype and the enhanced multimodal environmental feature vector obtained from the environmental encoder. The defect prototype decoder uses a cross-attention mechanism to effectively fuse the inherent morphological information of the defect with the influence of the current underwater environment. The key internal transposed convolution layer weights and batch normalization layer parameters are dynamically generated and applied by an auxiliary network based on the input environmental feature vector. Because the environmental feature vector contains richer and more accurate trimodal environmental information, the defect samples generated by the decoder can more realistically simulate and highly adapt to the target underwater environment in terms of morphology, edge clarity, optical and acoustic reflection characteristics, and texture details.

[0077] In this way, the generative adversarial network can generate higher-quality, more environmentally adaptable, and more realistic defect data, especially in simulated turbid water environments. This data, along with its corresponding defect prototype representation and refined environmental feature vectors, constitutes the second training set for training the adaptive detection model, providing high-quality input for the model to learn stronger environmental robustness and defect recognition capabilities.

[0078] The adaptive detection model is trained using a second training set generated through trimodal data fusion and a generative adversarial network. Key steps in the training process include: sorting training data based on curriculum learning based on environment and prototype perception; optimizing using a combined loss function consisting of a standard detection loss, a prototype contrast loss (whose margin uses an adversarial dynamic adjustment mechanism), and an auxiliary loss for environmental feature consistency; introducing feature-level adversarial training at the feature layer of the adaptive detection model to enhance environmental robustness; continuously aligning and fine-tuning the defect prototype embedding vectors in the defect knowledge embedding module; and integrating dynamic pruning and structural adaptation techniques during the training phase to obtain an efficient model. Through this training process, the resulting adaptive detection model further enhances its adaptability and defect recognition capabilities in complex underwater environments, especially turbid water bodies, because its training data is derived from richer sensor information.

[0079] Ultimately, this adaptive detection model, trained on trimodal fusion data and optimized through dynamic pruning, is deployed in actual underwater inspection tasks. The deployment platform can be the embedded processing unit of an underwater vehicle (ROV / AUV). During the inspection phase, the sonar, low-light camera, and muddy water clearing camera onboard the underwater vehicle simultaneously collect image data of the target underwater structure in real time. These three modal real-time image data are first processed according to the trimodal data fusion strategy to form a unified multimodal input feature. Subsequently, the deployed lightweight adaptive detection model receives and efficiently processes this fused multimodal image data. Leveraging its superior capabilities acquired through training on augmented data, the model performs in-depth analysis of the input fused image, accurately identifying various underwater structure defects and outputting information such as the defect category, location, and confidence level. To demonstrate the superiority of our method, we compared it with YOLOV11 and RT-DETRV2 on the same test set. The results are shown in Table 1. As can be seen from Table 1, the detection accuracy of the method of the present invention is much better than that of the YOLOV11 and RT-DETRV2 detection models, indicating that the method of the present invention has obvious advantages in underwater building defect detection methods.

[0080] Table 1 Comparison of detection effects between existing methods and the method of the present invention

[0081] parameter YOLOV11 RT-DETRV2 Method of the present invention Image size 512*512 512*512 512*512 Precision P 88.2 92.1 95.9 Recall R 83.8 84.5 87.5 mAP 87.4 91.2 94.3 F1 score 0.859 0.881 0.915

[0082] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An underwater structure defect detection method based on adaptive deep learning model training is characterized by: include: Multimodal underwater image data is obtained as a first training set, and a generative adversarial network is used to perform data enhancement and feature extraction on the first training set to obtain a second training set containing corresponding defect prototype representations and environmental feature vectors; Build an adaptive detection model, which includes a backbone feature extractor, a multi-scale feature pyramid, an adaptive attention module, and a defect knowledge embedding module that uses defect prototype representation and performs dynamic optimization. The training data is sorted using environmental feature vectors and defect prototype representations. Optimization is performed using a combination of standard detection loss, prototype contrast loss, and auxiliary loss. The auxiliary loss optimizes the objective by minimizing the difference between features in a predefined metric space. Adversarial training is introduced at the feature layer of the adaptive detection model, continuously aligning and fine-tuning the defect prototype embedding vectors in the defect knowledge embedding module to obtain a trained adaptive detection model. The trained adaptive detection model is deployed to the equipment to realize defect detection of underwater buildings.

2. The underwater structure defect detection method based on adaptive deep learning model training according to claim 1 is characterized in that: The environment encoder in the generative adversarial network is designed to receive and process multimodal underwater image data synchronously acquired and aligned from at least two types of sensors, wherein the sensors include an imaging sonar device and at least one optical imaging device; the environment encoder processes each modal data, including acoustic features and visual features, through an independent feature extraction branch, and deeply interacts and fuses the features extracted by each independent branch through a multimodal feature fusion module to output a fixed-dimensional environment feature vector, which contains acoustic characteristics and visual environment information.

3. The underwater structure defect detection method based on adaptive deep learning model training according to claim 2 is characterized in that: The defect prototype decoder in the generative adversarial network converts the defect prototype representation into a defect prototype latent vector, which is fused with the environmental feature vector through a cross-attention mechanism. The fused vector is upsampled through multiple layers of transposed convolution to gradually improve the spatial resolution. During the upsampling process, the weights of the convolution layer and the scaling parameters and translation parameters of the batch normalization layer are dynamically generated and applied by a small auxiliary network based on the environmental feature vector, ensuring that the generated defect instances meet the imaging characteristics of the target underwater environment in terms of morphology, edges, acoustic reflection characteristics and texture.

4. The underwater structure defect detection method based on adaptive deep learning model training according to claim 1 is characterized in that: The adaptive attention module in the adaptive detection module has a built-in fusion gating unit, which receives a global feature vector obtained through a global average pooling operation and a semantic feature vector extracted from a multi-scale feature pyramid, concatenates the global feature vector and the semantic feature vector, and inputs the concatenated data into a gating network comprising a fully connected layer and a nonlinear activation function. The activation value of the output layer of the gating network is normalized by a Softmax function to form a set of dynamic weighting coefficients with the same number as the levels of the multi-scale feature pyramid. The dynamic weighting coefficients are applied to the feature maps of the corresponding levels respectively.

5. The underwater structure defect detection method based on adaptive deep learning model training according to claim 1 is characterized in that: The defect knowledge embedding module in the adaptive detection model uses defect prototype representation to initialize a set of vectors whose dimensions match the image feature embedding dimensions as defect prototype embedding vectors. During the network forward propagation, the defect knowledge embedding module calculates the similarity scores between each defect prototype embedding vector through a learnable linear transformation for the input image region features, uses the similarity scores to perform a weighted summation on the defect prototype embedding vectors, and fuses the result with the original image region features. At the same time, during the backpropagation process, the defect prototype embedding vectors are updated based on the gradient of the loss function.

6. The underwater structure defect detection method based on adaptive deep learning model training according to claim 1 is characterized in that: The course sorting of training data using environmental feature vectors and defect prototype representations includes: dividing all underwater image training samples according to the weighted distance measurement between their corresponding environmental feature vectors and a pre-defined set of reference environmental feature vectors of different discrete levels to form environmental difficulty buckets; within each environmental difficulty bucket, sorting the samples from easy to identify to difficult to identify based on the comprehensive defect recognizability evaluation results of the training samples; the comprehensive defect recognizability evaluation includes, for the originally labeled training samples, considering the initial classification confidence of the defect prototype representation associated with the training samples and the frequency with which the training samples were correctly identified in the initial training stage; for the enhanced samples generated by the generative adversarial network, determining them based on the known recognizability characteristics of the defect prototypes used when they were generated.

7. The underwater structure defect detection method based on adaptive deep learning model training according to claim 1 is characterized in that: The prototype contrast loss is specifically implemented as follows: the image feature embedding extracted from the adaptive detection model and representing each defect area to be evaluated is defined as an anchor point; within the current training data batch, for the anchor point, a positive sample feature embedding is selected, the positive sample feature embedding belongs to the same defect category as the anchor point, and the L2 distance between the feature embedding of the positive sample feature embedding and the defect prototype representation corresponding to the defect category to which the anchor point belongs is minimized; a negative sample feature embedding is selected, the negative sample feature embedding belongs to a different defect category than the anchor point, and the L2 distance between the feature embedding of the negative sample feature embedding and the feature embedding of the anchor point is minimized; the goal of the prototype contrast loss is to minimize the L2 distance between the anchor point and the positive sample feature embedding, while maximizing the L2 distance between the anchor point and the negative sample feature embedding, to ensure that the difference between the two is greater than a preset margin.

8. The underwater structure defect detection method based on adaptive deep learning model training according to claim 1 is characterized in that: The introduction of adversarial training in the feature layer of the adaptive detection model includes: constructing a discriminator network composed of fully connected layers, the discriminator network receiving as input the feature vectors output by the convolutional layer blocks in the adaptive detection model and formed by global average pooling; the training goal of the discriminator network is to output a binary classification prediction probability to indicate that the input features it receives are input images derived from simple and complex underwater environment imaging characteristics, where the simplicity and complexity of the environment are divided according to the definition of environmental difficulty buckets in course learning; the loss function of the adaptive detection model includes the adversarial loss transmitted from the discriminator network through the gradient reversal layer; the adversarial loss guides the adaptive detection model to optimize the feature extraction process. Through the optimization, when the adaptive detection model processes images derived from different underwater environment imaging characteristics, the intermediate layer feature vectors generated are closer in statistical distribution, thereby enabling the adaptive detection model to extract essential defect features that are insensitive to environmental changes.

Images