Underwater building defect detection method based on adaptive deep learning model training

Abstract

The present application relates to the technical field of underwater image recognition, in particular to a method for detecting defects of underwater buildings based on adaptive deep learning model training, comprising: obtaining underwater image data as a first training set, and generating a second training set containing defect prototype representation and environmental feature vectors by using a generative adversarial network; constructing an adaptive detection model, including a backbone feature extractor, a multi-scale feature pyramid, an adaptive attention module and a defect knowledge embedding module; using the environmental feature vectors and the defect prototype representation to sort the training data by courses; optimizing by using standard detection loss, prototype contrast loss and auxiliary loss; introducing adversarial training to continuously align and fine-tune the defect prototype embedding vector, and finally obtaining the trained adaptive detection model; and using the trained adaptive detection model to realize defect detection of underwater buildings, thereby realizing accurate and efficient detection of underwater defects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of underwater image recognition, in particular to an underwater building defect detection method based on adaptive deep learning model training. BACKGROUND

[0002] Underwater infrastructure is crucial to the economy and society, and its structural integrity monitoring is urgently needed. Traditional underwater detection methods rely on divers, which have inherent limitations such as high cost, high risk, subjective results, and physiological limitations. Early automated image processing techniques are also limited in performance due to noise, lighting changes, and the complexity of defect morphology in underwater environments.

[0003] In recent years, deep learning has shown great potential, but the dynamic changes in underwater environments, data scarcity, and poor image quality, as well as unique characteristics such as color distortion and blurring, pose a serious challenge to the direct application of standard models. Underwater optical imaging is affected by factors such as light absorption and scattering, water turbidity, uneven lighting, and biological attachment, resulting in a significant decline in image quality.

[0004] Current deep learning techniques in underwater defect detection applications lack a model training method that can dynamically adjust and effectively learn under conditions of variable underwater image quality and a persistent lack of high-quality labeled data. At the same time, when faced with evolving defect features or new operating environments, it is difficult to quickly and incrementally update knowledge and optimize models without catastrophic forgetting, which greatly limits its potential for long-term, reliable deployment and application on resource-constrained autonomous underwater platforms.

[0005] Therefore, an underwater building defect detection method based on adaptive deep learning model training is proposed. SUMMARY

[0006] The purpose of the present application is to provide an underwater building defect detection method based on adaptive deep learning model training, which realizes accurate detection of underwater building defects through iterative defect carving generative adversarial network data augmentation, adaptive model construction, and multi-strategy training optimization. The method includes obtaining underwater image data as a first training set, using a generative adversarial network to generate a second training set containing defect prototype representations and environmental feature vectors; constructing an adaptive detection model, including a backbone feature extractor, a multi-scale feature pyramid, an adaptive attention module, and a defect knowledge embedding module; using the environmental feature vectors and defect prototype representations to sort the training data by course; using standard detection loss, prototype contrast loss, and auxiliary loss for optimization; introducing adversarial training to continuously align and fine-tune the defect prototype embedding vectors, and finally obtaining a trained adaptive detection model; and using the trained adaptive detection model to detect defects in underwater buildings.

[0007] To achieve the above object, the present application provides the following technical solutions:

[0008] The underwater building defect detection method based on adaptive deep learning model training comprises:

[0009] The multi-modal underwater image data is acquired as a first training set, the first training set is subjected to data enhancement and feature extraction by using a generative adversarial network, and a second training set containing corresponding defect prototype representation and environmental feature vectors is obtained;

[0010] An adaptive detection model is constructed, the adaptive detection model comprises a backbone feature extractor, a multi-scale feature pyramid, an adaptive attention module and a defect knowledge embedding module using defect prototype representation and dynamic optimization;

[0011] The training data is subjected to course sorting by using the environmental feature vectors and the defect prototype representation; a standard detection loss, a prototype contrast loss and an auxiliary loss are used for optimization, the auxiliary loss optimizes the target by minimizing the difference between the features in a predefined metric space; the adaptive detection model is subjected to adversarial training at the feature layer, the defect prototype embedding vector in the defect knowledge embedding module is continuously aligned and fine-tuned, and a trained adaptive detection model is obtained;

[0012] The trained adaptive detection model is deployed into a device to realize defect detection of underwater buildings.

[0013] Preferably, the environmental encoder in the generative adversarial network is designed to receive and process multi-modal underwater image data synchronously acquired and aligned from at least two types of sensors, the sensors comprising an imaging sonar device and at least one optical imaging device; the environmental encoder processes each modal data through independent feature extraction branches, including acoustic features and visual features, and outputs an environmental feature vector of a fixed dimension through multi-modal feature fusion modules for deep interaction and fusion of the features extracted by each independent branch, the environmental feature vector containing acoustic characteristics and visual environmental information.

[0014] Preferably, the defect prototype decoder in the generative adversarial network converts the defect prototype representation into a defect prototype latent vector, and the defect prototype latent vector and the environmental feature vector are fused through a cross-attention mechanism, and the fused vector is subjected to upsampling through multi-layer transposed convolution to gradually improve the spatial resolution, in the upsampling process, the weights of the convolution layer and the scaling parameters and translation parameters of the batch normalization layer are dynamically generated and applied by a small auxiliary network according to the environmental feature vector, so as to ensure that the generated defect instances conform to the imaging characteristics of the target underwater environment in terms of morphology, edge, acoustic reflection characteristics and texture.

[0015] Preferably, a fusion gate unit is built in the adaptive attention module in the adaptive detection module, the fusion gate unit receives a global feature vector obtained through a global average pooling operation and a semantic feature vector extracted from a multi-scale feature pyramid, and inputs the global feature vector and the semantic feature vector after splicing into a gate network containing a full connection layer and a nonlinear activation function, an output layer activation value of the gate network is normalized through a Softmax function to form a set of dynamic weighting coefficients with the same number of levels as each level of the multi-scale feature pyramid, and the dynamic weighting coefficients are respectively applied to feature maps of the corresponding levels.

[0016] Preferably, the defect knowledge embedding module in the adaptive detection model initializes a set of vectors with dimensions matched with image feature embedding dimensions as defect prototype embedding vectors by using defect prototype representations; during network forward propagation, the defect knowledge embedding module calculates similarity scores between each defect prototype embedding vector through a learnable linear transformation for input image region features, performs weighted summation on the defect prototype embedding vectors by using the similarity scores, fuses the result with the original image region features, and simultaneously updates the defect prototype embedding vectors based on gradient of a loss function during back propagation.

[0017] Preferably, the course sorting of the training data by using the environmental feature vector and the defect prototype representation includes: dividing all underwater image training samples according to weighted distance measurements between corresponding environmental feature vectors of the samples and reference environmental feature vectors of a set of different discrete levels defined in advance to form environmental difficulty buckets; within each environmental difficulty bucket, samples are sorted from easy to identify to difficult to identify according to comprehensive defect easy-to-identify evaluation results of the training samples; the comprehensive defect easy-to-identify evaluation includes considering initial classification confidence of a defect prototype representation associated with a training sample and frequency of correct identification of the training sample in an initial training stage; for an enhanced sample generated by the generative adversarial network, known easy-to-identify properties of a defect prototype used when the enhanced sample is generated are determined.

[0018] Preferably, the prototype contrast loss is specifically implemented as follows: image feature embeddings representing each defect region to be evaluated and extracted from the adaptive detection model are taken as anchor points; in the current training data batch, for the anchor points, a positive sample feature embedding is selected, which belongs to the same defect category as the anchor point and whose own feature embedding is closest to the defect prototype representation corresponding to the defect category; in addition, a negative sample feature embedding is selected, which belongs to a different defect category from the anchor point and whose own feature embedding is closest to the prototype representation of the defect category to which it belongs; the prototype contrast loss minimizes the L2 distance between the anchor point and the positive sample feature embedding, while maximizing the L2 distance between the anchor point and the negative sample feature embedding, so as to ensure that the difference between the L2 distance between the anchor point and the negative sample feature embedding and the L2 distance between the anchor point and the positive sample feature embedding is greater than a preset boundary margin.

[0019] Preferably, the introduction of the adversarial training in the feature layer of the adaptive detection model comprises: constructing a discriminator network composed of fully connected layers; the discriminator network receives a feature vector formed by global average pooling of the output of a convolutional layer block in the adaptive detection model as input; the training target of the discriminator network is to output a binary classification prediction probability to indicate whether the input feature received is from an input image with simple or complex underwater environment imaging characteristics, wherein the simplicity and complexity of the environment are defined according to the environment difficulty bucketing in curriculum learning; the adversarial loss transferred from the discriminator network through the gradient reversal layer is included in the loss function of the adaptive detection model; the adversarial loss guides the optimization of the feature extraction process of the adaptive detection model, and through the optimization, the intermediate layer feature vectors generated by the adaptive detection model when processing images with different underwater environment imaging characteristics are more similar in statistical distribution, thereby enabling the adaptive detection model to extract essential defect features that are not sensitive to environmental changes.

[0020] Compared with the prior art, the present application has the following beneficial effects:

[0021] 1. The generated adversarial network is used for data augmentation and feature extraction of multi-modal underwater image data. The environmental encoder deeply fuses acoustic and visual features to output environmental feature vectors containing acoustic and visual environmental information. The perception of complex underwater environments is enhanced, and the underwater environment can be more comprehensively represented to provide effective guidance for subsequent networks.

[0022] 2、The adaptive detection model is constructed, which comprises a backbone feature extractor, a multi-scale feature pyramid (FPN), an adaptive attention module and a defect knowledge embedding module. The adaptive attention module has a built-in fusion gate unit, which dynamically weights and fuses global and local saliency features according to the global context, optimizes the multi-scale feature fusion mode, enhances the adaptability of the model to complex underwater environments and the recognition accuracy of different defects. At the same time, the defect knowledge embedding module effectively integrates the defect prior knowledge extracted from the generative adversarial network into the detection network, and through the attention mechanism, the image features and the defect prior knowledge are interacted, the recognition accuracy of specific defect types is improved, and the sensitivity and discriminability of defect features are enhanced, especially in distinguishing visually similar or rare defects.

[0023] 3、The training data is sorted by courses using the environmental feature vector and the defect prototype representation. This improves the learning stability and the final performance of the model. A combined loss function including a standard detection loss, a prototype contrast loss that aligns guided features with prototypes, and an environmental feature consistency auxiliary loss is used for optimization. The prototype contrast loss and its adversarial dynamic margin adjustment mechanism impose adaptive learning pressure, driving the model to learn highly discriminative and robust feature representations for different defect types, effectively reducing confusion between different defect categories and further improving classification accuracy.

[0024] 4、Introduce adversarial training at the feature layer of the adaptive detection model, so that the model learns to extract essential defect features that are not sensitive to environmental changes, significantly enhancing the environmental robustness and practical application performance of the model. At the same time, the defect prototype embedding vector in the defect knowledge embedding module is continuously aligned and fine-tuned during training, so that it dynamically evolves to more accurately represent various defects, improving the efficiency of the prototype contrast loss and the accuracy of the final defect recognition. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The flowchart of the underwater building defect detection method based on the adaptive deep learning model training provided by the embodiment of the present application is provided;

[0026] Figure 2 The flowchart of the training sample course sorting provided by the embodiment of the present application is provided;

[0027] Figure 3 The training total loss diagram provided by the embodiment of the present application is provided. DETAILED DESCRIPTION

[0028] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0029] Please refer to Figures 1 to 3 The present application provides an underwater building defect detection method based on adaptive deep learning model training, and the technical solutions are as follows:

[0030] Embodiment one:

[0031] In order to improve the detection accuracy of underwater building defect detection, the underwater building defect detection method based on adaptive deep learning model training provided by the present application is introduced, and the specific flow chart is as shown in Figure 1

[0032] The underwater image data is obtained as a first training set, and the generative adversarial network is used for data enhancement and feature extraction on the first training set to obtain a second training set containing corresponding defect prototype representation and environmental feature vector.

[0033] The obtained underwater image mainly contains two parts

[0034] Imaging sonar: sonar can work effectively in muddy water and low visibility environment, and provide key structural outline and medium and long distance detection information.

[0035] Low-light camera: used to improve the imaging capability under insufficient illumination, obtain the optical image of the target area, and provide supplementary information for the fine features of the defect.

[0036] During the collection process, the synchronous acquisition and spatial alignment of the sonar image and the low-light camera image are realized, the low-light camera and the sonar equipment are integrated on the same underwater robot (ROV) platform, and through the positioning and attitude information of the ROV, combined with the known relative position and field of view parameters of the sensor, the sonar image and the optical image are preliminarily spatially registered to obtain multi-modal image data of the same target area. The image collection environment has diversity, covering from relatively clear to highly turbid water body, and the target surface may be attached by marine organisms or covered by sediments. The acoustic equipment is easily disturbed by various noises when collecting signals underwater; optical imaging also faces problems such as light scattering, absorption and color distortion.

[0037] ​The initial dataset, i.e., the first training set, contains original underwater sonar images and optical images from a low-light camera system. These images are manually screened and annotated for defects, including defect categories and their bounding boxes. The sonar data is converted in format and geometrically corrected; the optical images are de-fogged and color-corrected; and all images are resized and preliminarily denoised.

[0038] The environment encoder in the generative adversarial network receives underwater sonar images and low-light optical images as inputs. First, acoustic features of the sonar images and visual features of the optical images are extracted through a group of convolutional neural network (CNN) layers, respectively. Then, a cross-modal attention module is introduced. The sonar feature map is taken as the query (Query), and the optical feature map is taken as the key (Key) and value (Value). The similarity between the query and the key is calculated to assign weights to the values of the optical feature map, obtaining an optical feature representation modulated by the sonar information. Finally, the two modulated feature maps are concatenated with the original feature maps and processed through convolutional layers to form a multi-modal environment feature vector with more information and discriminability. The fused multi-modal features are processed through a self-attention mechanism to aggregate the global environmental context information. Finally, the environment encoder outputs an environment feature vector of a fixed dimension. The environment feature vector, which integrates the information of both sonar and optical modalities, can more comprehensively represent the underwater environment. Specifically, the environment feature vector contains acoustic environmental characteristics, including overall acoustic reflection intensity distribution statistical characteristics, turbidity-related texture details reflected by high-frequency components analyzed by fast Fourier transform, and signal non-uniformity indicators calculated by analyzing the skewness and kurtosis of the signal intensity histogram; the environment feature vector also contains optical environmental characteristics, including underwater light attenuation, color distortion, image contrast, and scattering effect strength evaluated from the color channel histogram. In this way, the environment encoder has a more robust and detailed understanding of the underwater environment.

[0039] By intelligently combining the penetration of sonar and the details of optics, the perception ability of complex underwater environments is significantly improved. The cross-modal attention mechanism can dynamically adapt to environmental changes, helping to enhance the recognition of ambiguous defects. The more accurate environment feature vector can provide more effective guidance for subsequent networks for defect sample generation and knowledge embedding, thereby striving to improve the overall accuracy and robustness of defect detection to address the challenge of insufficient single-sensor information.

[0040] In the generative adversarial network, the defect prototype decoder is responsible for generating defects according to the characteristics of the underwater environment. Its workflow begins with receiving two types of inputs: one is the defect prototype representation that represents different common defect categories of underwater buildings, which is converted into a defect prototype latent vector by an encoder; the other is an environmental feature vector output by the environment encoder, which represents the comprehensive characteristics of the current underwater environment. The decoder first fuses the two input vectors through a cross-attention mechanism. The attention mechanism enables the feature information of the defect prototype and the feature information of the environment to interact effectively, so that the generation of the defect can take into account both the inherent morphology of the defect itself and the influence of the current underwater environment. After cross-attention fusion, the vector is input into a network composed of multiple transposed convolution layers. This network gradually increases the spatial resolution of the feature map through layer-by-layer upsampling, and the ultimate goal is to directly generate a fused image region with simulated defects. A key technical point in this process is dynamic parameter generation. In the upsampling process, the weights of the transposed convolution layers inside the decoder network, as well as the scaling parameters and translation parameters of the batch normalization layers that follow them, are not fixed. They are calculated in real time and dynamically applied by a small auxiliary neural network according to the current environmental feature vector. This auxiliary network learns a mapping from the environmental feature vector to the decoder layer parameters.

[0041] Through this dynamic parameter adjustment mechanism, the final generated defect instance can truly conform to the comprehensive imaging characteristics of the target underwater environment in its fused image representation. This includes the acoustic reflection characteristics, edge sharpness or blurriness, color performance, and texture features of the defect region. In this way, the generative adversarial network can generate highly realistic and environment-adaptive defect data, providing high-quality samples for subsequent defect detection model training.

[0042] The adaptive detection model includes a backbone feature extractor, a multi-scale feature pyramid, an adaptive attention module, and a defect knowledge embedding module that utilizes defect prototype representations and performs dynamic optimization.

[0043] The backbone feature extractor is responsible for extracting hierarchical deep features from the input fused image. In this embodiment, EfficientNet-B3 is chosen as the backbone network. The backbone network has been pre-trained on a large image dataset, which helps the model learn a general image feature representation and speeds up convergence on the underwater defect detection task. The backbone network outputs feature maps at different stages, which have different spatial resolutions and semantic information strengths, laying the foundation for subsequent multi-scale feature analysis. The feature maps at shallower layers retain more spatial detail information, while the feature maps at deeper layers contain more rich semantic information.

[0044] To effectively detect defects of underwater structures with different sizes, an adaptive detection model constructs a multi-scale feature pyramid (FPN). FPN combines feature maps output by the backbone network at different depths to construct a series of feature layers with rich semantic information and multi-scale information. In specific implementation, FPN uses a top-down path to enlarge high-level semantic features through upsampling, and horizontally connects them with feature maps from lower layers that have higher spatial resolution but weaker semantic information. After channel alignment through 1x1 convolution, the features are spliced. The resulting multi-level feature pyramid corresponds to different receptive fields and scales at each level, allowing the model to focus on both large targets and small defects.

[0045] The core function of the adaptive attention module is to dynamically weight and fuse global and local saliency features based on global context to enhance the model's ability to perceive important features. The module has a built-in fusion gating unit. Specifically, the fusion gating unit first receives a global context feature vector output from the deep features of the backbone network after global average pooling. At the same time, it also receives a feature vector extracted from the feature map at the highest level of the multi-scale feature pyramid. These two feature vectors are concatenated in the channel dimension and then input into a gating network containing several fully connected layers and nonlinear activation functions. The activation values of the output layer of the gating network are normalized by the Softmax function to form a set of dynamic weighting coefficients. The number of these coefficients is the same as the number of levels in the multi-scale feature pyramid. Finally, these dynamically generated weighting coefficients are applied to the local saliency feature maps at the corresponding levels of the multi-scale feature pyramid. By broadcasting the coefficients to each spatial location of the feature map and performing element-wise multiplication, adaptive weighting and fusion of features of different scales are achieved. This allows the model to dynamically adjust the attention to features of different scales based on image content.

[0046] The fusion gating unit in the adaptive attention module optimizes the fusion of multi-scale features through its intelligent dynamic weighting mechanism, enhancing the model's ability to adapt to complex underwater environments and its recognition accuracy for different defects, thus providing a more effective and robust solution for underwater structure defect detection.

[0047] The defect knowledge embedding module aims to effectively integrate the defect prior knowledge extracted from the generative adversarial network into the detection network, and make the image features interact with the defect prior knowledge through the attention mechanism, so as to improve the recognition accuracy of the defect type. Specifically: first, the defect prototype representation representing different common defect categories of underwater buildings extracted from the generative adversarial network is used to initialize a set of learnable embedding vectors as defect prototype embedding vectors. The dimension of these embedding vectors matches the embedding dimension of the image features output by the FPN in the adaptive detection model. During the forward propagation of the network, for the image region features extracted from a certain level of FPN or the region of interest, the defect knowledge embedding module calculates the similarity between the image region features and each of the defect prototype embedding vectors. This similarity is calculated by a learnable linear transformation and then a similarity calculation, to obtain a similarity score. The step of calculating the similarity by the learnable linear transformation is: first, the module performs a linear transformation operation on the received input image region features. This transformation is performed by the first set of learnable parameters, i.e. weight values and bias values, which are adjusted and optimized by the back propagation algorithm during the overall training of the model, and its function is to map the image region features into a new transformed image feature representation. At the same time, the module also independently performs a similar linear transformation operation on each of the defect prototype embeddings. This transformation is performed by the second set of independent learnable parameters, i.e. weight values and bias values, and its purpose is to map each prototype embedding into a new transformed prototype feature representation. The learnable parameters used by the two sets of linear transformations are learned by the model from the training data, aiming to project the original features into a shared and more beneficial feature space for subsequent comparison. After obtaining the transformed image features and the transformed prototype features respectively, the module calculates the dot product between the two to calculate the similarity, and the dot product result is quantified as a similarity score. The similarity scores calculated are normalized by the Softmax function and used as attention weights. The attention weights are used to weight and sum the corresponding defect prototype embedding vectors to obtain a context vector that integrates prior defect knowledge. Finally, the context vector is fused with the original image region features by splicing and then processed by a fully connected layer. During the back propagation process, based on the gradient of the overall loss function, not only the network parameters of the image feature extraction part will be updated, but also these learnable defect prototype embedding vectors will be continuously aligned and fine-tuned, so that they can more accurately represent the essential features of each defect category.

[0048] By extracting defect prototypes from the generative adversarial network as learnable prior knowledge, and using attention mechanism to interact image features with them, the model can more accurately match known defect patterns. This enhances the sensitivity and discriminability of defect features, especially in distinguishing visually similar or rare defects. At the same time, it improves the generalization ability and robustness of the model in the case of sample imbalance, because even with few training samples, the model can still benefit from prototype knowledge. In addition, good initialization helps to accelerate model convergence, improve training efficiency, and provide certain explainability for model decision-making.

[0049] The training data is sorted by using the environmental feature vector and the defect prototype representation; a combined loss function including a standard detection loss, a prototype contrast loss for aligning guided features with prototypes, and an auxiliary loss is used for optimization; an adversarial training is introduced in the feature layer of the adaptive detection model to continuously align and fine-tune the defect prototype embedding vector in the defect knowledge embedding module, and a trained adaptive detection model is obtained;

[0050] In order to improve the training efficiency and model performance, before starting training the adaptive detection model using the second training set, it is first sorted, and the specific course sorting flowchart is as shown in Figure 2 The sorting process makes full use of the environmental feature vector and the defect prototype representation associated with each image in the second training set. The specific implementation is divided into two main stages. First, all training samples, including original images and enhanced images generated by the generative adversarial network, are preliminarily grouped according to their corresponding environmental feature vectors. This is achieved by calculating the weighted distance between the environmental feature vector of each sample and a set of pre-defined reference environmental feature vectors. These reference vectors represent different discrete levels of environmental conditions from extremely clear underwater environments to extremely turbid underwater environments. The reference environmental vectors are obtained by a data-driven clustering method. Specifically, first, a representative and diverse set of underwater images (a calibration dataset specially prepared for this purpose) is selected, and the environmental encoder in the generative adversarial network is used to process these images to extract their respective environmental feature vectors. Then, the K-Means clustering algorithm is applied to these extracted environmental feature vectors, where the number of clusters K is set to be consistent with the number of expected environmental difficulty discrete levels. The centers of the K clusters generated by clustering analysis are established as the pre-defined reference environmental feature vectors. This method ensures that the reference vectors are objectively generated based on data, and uses the same feature space as the features used in the subsequent learning process of the model, providing a robust and consistent basis for environmental difficulty assessment in curriculum learning. Finally, according to the distance metric, the samples are divided into several environmental difficulty buckets, including clear environment bucket, moderate turbidity environment bucket, and high turbidity environment bucket.

[0051] Subsequently, within each environmental difficulty bucket, the samples are further sorted from easy-to-identify defects to hard-to-identify defects. The sorting is based on a combination of the following aspects: for each sample, the initial classification confidence corresponding to the associated defect prototype representation is considered; for the original labeled data, the historical frequency of being correctly identified in the initial training phase is referred to; for the augmented images generated by the generative adversarial network, the known easy-to-identify degree characteristics of the prototype used to generate the defect are referred to. These aspects collectively constitute the criteria for determining the learning priority of the samples within the environmental difficulty level.

[0052] To determine the easy-to-identify degree characteristics of each defect prototype, the adaptive multi-scale model is used in the initial training phase to evaluate the learning efficiency of the real and labeled defect data in the second training set. Specifically, by analyzing the recognition accuracy, learning speed, and confidence of the adaptive detection model when training these real defect samples, a quantitative easy-to-identify score is assigned to each defect category. Defect categories that the model can quickly and accurately identify in the early stage will have higher easy-to-identify scores, and vice versa. When the generative adversarial network generates defect samples for data augmentation based on the prototype, the synthetic samples inherit the easy-to-identify score of their parent prototype. This score is then used in the second phase of curriculum learning to sort the synthetic samples from easy to hard within the same environmental difficulty bucket. This method not only has a data-driven characteristic, but also makes full use of existing real labeled data and core detection models, ensuring a high correlation between easy-to-identify evaluation and model learning characteristics, making the entire training curriculum more intelligent and efficient, and forming a closed-loop logic of learning difficulty from data and guiding subsequent learning based on learned difficulty.

[0053] Through this two-stage sorting strategy, the final training curriculum is formed. This method allows the model to first encounter samples with simple environments and easy-to-identify defects in the early stages of training, and gradually transition to samples with complex environments and hard-to-identify defects as training progresses, simulating the human learning process from easy to hard, which helps to improve the stability of learning and the final performance of the model.

[0054] The training of the adaptive detection model uses a carefully designed combined loss function for optimization, which includes a standard detection loss, an innovative prototype contrast loss, and an environmental feature consistency auxiliary loss. The specific total loss diagram is shown in Figure 3 .

[0055] The standard detection loss is used for the classification and localization of defects by the supervised model. For a specific detection head design, the defect class classification loss uses Focal Loss, and the defect bounding box regression loss uses GIoU Loss. In addition, a prototype contrast loss is used, which aims to guide the adaptive detection model to extract defect region image features embedded in alignment with the corresponding defect prototype and distinguish from other class prototypes. The specific implementation is as follows: the image feature embedding representing each defect region to be evaluated extracted from the adaptive detection model is defined as an anchor point; for the anchor point, a positive sample feature embedding is selected in the current training data batch, the positive sample feature embedding belongs to the same defect class as the anchor point, and the L2 distance between the feature embedding of the positive sample feature embedding and the defect prototype representation corresponding to the defect class to which the anchor point belongs is the smallest; a negative sample feature embedding is selected, the negative sample feature embedding belongs to a different defect class from the anchor point, and the L2 distance between the feature embedding of the negative sample feature embedding and the feature embedding of the anchor point is the smallest; the goal of the prototype contrast loss is to minimize the L2 distance between the anchor point and the positive sample feature embedding, and maximize the L2 distance between the anchor point and the negative sample feature embedding, to ensure that the difference between the two is greater than a preset boundary margin. This mechanism forces the model to learn features that are highly distinguishable between different defect types, and uses the learned prototypes as a guide.

[0056] The preset boundary margin is specifically a dynamically adjusted boundary margin, and the dynamic boundary margin adjustment method is to introduce an adversarial learning mechanism to set the margin value by constructing a special margin adjuster network and dynamically gaming with the main feature learning network. Under this framework, the margin adjuster is responsible for automatically generating a boundary margin that is most challenging to the main network but still achievable, aiming to encourage the main network to learn better feature representations; while the main feature learning network strives to optimize the feature embedding under the condition of the dynamically changing margin. The two networks are trained cooperatively, so that the setting of the boundary margin itself becomes an adaptive learning process and a dynamic adversarial process, rather than a passive adjustment of hyperparameters, thus continuously providing moderate learning pressure for the main network to drive it to learn more discriminative and robust features for complex situations.

[0057] Furthermore, the computation of the auxiliary loss aims to achieve representational stability across environments. This is achieved by processing paired samples, specifically samples from different real-world observation environments containing the same type of defect, and various variants generated by a generative adversarial network to simulate the morphology of the same defect under different environmental conditions. The intermediate layer features of each pair are extracted from the adaptive detection model. Then, the optimization objective of this loss is achieved by minimizing the differences between these paired features in a predefined metric space. Specifically, L2 distance is used to measure and minimize the differences between feature pairs, aiming to minimize their angular differences. This direct penalty for feature differences drives the model to learn more fundamental defect representations that are less sensitive to environmental changes, thus striving to keep these features as consistent as possible under different environmental influences. Moreover, when environmental factors inevitably cause systematic changes in features, this loss mechanism also helps to ensure that the change follows a stable pattern that can be learned and effectively compensated for by the model, ultimately guaranteeing the reliable extraction of core defect information. This directly enhances the model's robustness to changes in water clarity, lighting, and other environmental factors.

[0058] The overall loss function is a weighted sum of the losses, which mainly consist of standard detection loss, prototype comparison loss, and environmental feature consistency auxiliary loss. The weights of each loss are hyperparameters that need to be adjusted.

[0059] By leveraging the precise supervision of standard detection loss, efficient classification and accurate localization of underwater defects are achieved. An innovative prototype contrastive loss and its adversarial dynamic margin adjustment mechanism apply adaptive learning pressure, driving the model to learn highly discriminative and robust feature representations for different defect types. Environmental feature consistency auxiliary loss ensures the stability of features extracted from similar defect samples under different environmental conditions, significantly enhancing the model's robustness to complex underwater environmental factors. Overall, this design aims to comprehensively improve the accuracy, robustness, and generalization ability of the detection model to meet real-world underwater monitoring challenges.

[0060] Further, in order to further enhance the feature robustness of the adaptive detection model to environmental changes, an adversarial training is introduced at the feature layer of the model to ensure stable performance under variable underwater conditions. Specifically, a discriminator network is constructed. The discriminator is usually composed of multiple fully connected layers. It receives the feature vector output by the global average pooling after the convolutional layer block in the adaptive detection model, such as a certain output layer of the feature pyramid network or a deep layer of the backbone network, as input. The training target of the discriminator is to output a binary classification prediction probability to indicate whether the input feature received is from an image with simple underwater environment vision or acoustic characteristics or from an image with complex underwater environment vision or acoustic characteristics. The simplicity and complexity of the environment can be divided according to the definition of environment bucketing in the previous curriculum learning.

[0061] In the loss function of the adaptive detection model, an adversarial loss is passed from the discriminator through a gradient reversal layer. The gradient reversal layer will reverse the sign of the gradient from the discriminator during backpropagation. This adversarial loss drives the adaptive detection model to generate intermediate layer feature vectors that, in the view of the discriminator, are in the same distribution as the feature vectors generated when processing images with simple underwater environments, i.e., to deceive the discriminator so that it cannot distinguish the environment type of the feature source. This forces the adaptive detection model to learn to extract features that are insensitive to environmental changes and more essential defects, thereby significantly improving its performance in practical applications. The defect prototype embedding vector learned by the defect knowledge embedding module in the training process is continuously aligned and fine-tuned based on the gradient of the overall loss function. These prototypes are not just fixed prior knowledge, but can dynamically evolve according to the training data and task feedback. This allows them to more accurately and subtly represent the essential features of various defects, thereby better serving the prototype contrast loss and the final defect recognition task.

[0062] Through the synergistic effect of the above curriculum learning, combined loss function optimization, and innovative feature-level adversarial training, and continuous optimization of the defect prototype embedding vector, the adaptive detection model is fully trained. After training is complete, an adaptive detection model with high detection accuracy and strong environmental robustness for underwater building defects is obtained, which can be used for subsequent deployment and actual detection tasks.

[0063] By introducing adversarial training at the feature level, the adaptive detection model learns essential defect features that are not affected by underwater environmental changes, significantly enhancing the model's environmental robustness and practical application performance. At the same time, the defect prototype embedding vector is continuously aligned and fine-tuned during training, allowing it to dynamically evolve to more accurately represent various defects, thereby improving the efficiency of the prototype contrast loss and the accuracy of the final defect recognition. The synergy of these two with curriculum learning and combined loss ultimately ensures that the adaptive detection model has high detection accuracy and strong environmental adaptability, capable of complex deployment tasks.

[0064] The adaptive detection model trained is deployed into the device to realize the defect detection of the underwater building.

[0065] To obtain an efficient adaptive detection model in the training stage, dynamic pruning and structure adaptation technology in the training process is introduced. The core of this technology is to periodically adjust the sparse connection structure of the network in the model training iteration. Specifically, every certain number of iterations, the system identifies and removes the connections that contribute less to the current defect detection task according to the pre-set importance measurement standards, including the size of the neural connection weight and its gradient information and other indicators, to realize network sparsification. At the same time, to maintain the learning ability of the model and explore a more optimized sparse structure, the technology allows a part of the previously removed connections to be reactivated according to their potential importance evaluated subsequently, that is, to realize the resurrection of the connection, by judging the gradient change trend; and the system also allows the network to grow new connections in the position that contributes greatly to the improvement of the model performance. The whole process of dynamic pruning and connection regeneration is carried out under the constraint of the pre-set target network sparsity. In this way, the adaptive detection model learns to complete the defect detection task while its own network topology structure is continuously adapted and optimized. Compared with the traditional post-training pruning method, this technology of dynamically adjusting and optimizing the sparse structure in training gives the network model greater flexibility and adaptive adjustment opportunities, so as to have the potential to maintain a small detection precision loss while achieving a high model compression rate, and finally obtain an efficient and excellent performance sparse adaptive detection model, laying a foundation for subsequent deployment and efficient inference on resource-constrained platforms.

[0066] In the actual detection stage, the adaptive detection model optimized by the dynamic pruning and structure adaptation technology is first deployed on the underwater robot or the corresponding analysis platform. During operation, the imaging sonar and the low-light camera carried by the robot synchronously collect the images of the target underwater building. The obtained dual-mode image data is processed by the consistent multi-modal fusion processing in the training stage, and is sent into the deployed adaptive detection model as unified input. The model relies on its strong feature extraction, multi-scale analysis and defect recognition ability learned in the training to quickly analyze and process the fusion data, and finally outputs the detected defect type, accurate spatial position and related confidence information.

[0067] The method fuses acoustic and optical multi-modal image information and uses cross-modal attention mechanism to provide the model with comprehensive scene and subtle defect details far beyond a single sensor, laying a solid data foundation for subsequent accurate identification. The core adaptive multi-scale detection model integrates a dynamic attention module and a defect knowledge embedding module: the dynamic attention module can guide the model to adaptively focus on the image feature area and scale that is most critical to defect judgment, effectively excluding irrelevant information interference; the defect knowledge embedding module uses defect types extracted from the generative adversarial network and continuously optimized, thereby significantly enhancing the model's recognition accuracy and discrimination ability for defect types, especially fine-grained defects. At the training level, the standard detection loss directly optimizes the classification accuracy and positioning precision as the goal; while the innovative prototype contrast loss and its supporting adversarial dynamic margin adjustment mechanism strongly drive the model to learn high-discrimination defect feature representations, effectively reducing confusion between different defect categories, further improving classification accuracy. The comprehensive application of these mechanisms ensures that the model can not only accurately identify and precisely locate underwater building defects, but also continuously maintain high-precision detection performance in complex underwater environments.

[0068] Embodiment Two:

[0069] In this embodiment, the underwater image data is collected from three different types of sensors integrated into the same underwater robot (ROV / AUV) platform, ensuring time synchronization and spatial alignment during the collection process:

[0070] Imaging sonar equipment: provides detection capability for building structure contours and larger defects in medium and long distances and turbid water bodies.

[0071] Low-light camera: used to obtain optical images of the target area under insufficient lighting conditions, providing visual information for the fine features of defects.

[0072] Clear water making camera equipment: designed to improve imaging quality, enhance contrast, and suppress scattering in turbid water bodies, capable of obtaining structural details and defect features that are difficult to reveal under traditional optical imaging methods.

[0073] To fully utilize the complementary information of the three modalities, an enhanced multi-modal feature fusion mechanism is adopted before the data is input into the environmental encoder of the generative adversarial network. First, for the sonar image, the low-light optical image, and the image obtained by the turbid water clearing camera, their respective initial feature maps are extracted through independent convolutional neural network (CNN) branches. The network structure of each branch is optimized for the data characteristics of the corresponding modality. Second, a multi-modal fusion module is introduced. This module can be designed based on an attention mechanism, such as an extended cross-modal attention network or a multi-head attention layer, which can simultaneously receive feature maps from three independent branches. This module learns the complex correlation and complementarity between different modalities, dynamically assigns weights to the features of different modalities, and performs deep fusion. For example, the enhanced detail features of the turbid water clearing image can be used to enhance the blurred areas in the low-light image, and the structural outline information provided by the sonar can provide global context for the understanding of the optical image. Then, the fused multi-modal features are processed through several convolutional layers to finally form a fixed-dimensional environmental feature vector that is more informative and comprehensive in representing the underwater environment. This vector will serve as the output of the environmental encoder in the generative adversarial network.

[0074] Through this three-modal fusion strategy, the generated environmental feature vector can more accurately capture the complexity of the underwater environment, including the enhanced information provided by the turbid water clearing device under poor visual conditions, laying a solid foundation for subsequent defect sample generation and model training.

[0075] The input received by the environmental encoder in the generative adversarial network is a unified environmental representation formed by deep fusion of the three sensor data (sonar, low-light camera, and turbid water clearing camera) after independent feature extraction through a multi-modal attention mechanism. Since it has fused the enhanced information provided by the turbid water clearing camera under poor visual conditions, as well as the complementary data of sonar and low-light images, the environmental feature vector output by this encoder can more accurately and meticulously capture the complex conditions and subtle differences of extreme underwater environments.

[0076] The defect prototype decoder generates specific defect instances based on the latent representation of the defect prototype and the aforementioned enhanced multi-modal environmental feature vector obtained from the environmental encoder. The defect prototype decoder effectively fuses the inherent morphological information of the defect with the influence of the current underwater environment using a cross-attention mechanism. The weights of the internal key transpose convolutional layers and the parameters of the batch normalization layers are dynamically generated and applied by an auxiliary network based on the input environmental feature vector. Since the environmental feature vector contains more rich and accurate three-modal environmental information, the defect samples generated by the decoder can more realistically simulate and highly adapt to the target underwater environment in terms of morphology, edge sharpness, optical and acoustic reflection characteristics, and texture details.

[0077] In this way, the generative adversarial network can generate defect data with higher quality and stronger environmental adaptability, especially more realistic in the simulation of turbid water environment. These data, together with their corresponding defect prototype representation and refined environmental feature vector, jointly constitute the second training set for training the adaptive detection model, providing high-quality input for the model to learn stronger environmental robustness and defect recognition capability.

[0078] The adaptive detection model is trained using the second training set generated by the three-modal data fusion and the generative adversarial network. The key steps of the training process include: sorting the training data based on environmental and prototype perception-based curriculum learning; optimizing using a combined loss function containing standard detection loss, prototype contrast loss (with an adversarial dynamic adjustment mechanism for its boundary margin), and environmental feature consistency auxiliary loss; introducing feature-level adversarial training at the feature layer of the adaptive detection model to enhance environmental robustness; continuously aligning and fine-tuning the defect prototype embedding vectors in the defect knowledge embedding module; and integrating dynamic pruning and structure adaptive technology during the training phase to obtain an efficient model. Through this training process, the adaptive detection model obtained has further enhanced adaptability and defect recognition capability in complex underwater environments, especially turbid water, due to the richer sensor information from the training data.

[0079] Finally, the adaptive detection model trained on three-modal fusion data and optimized by dynamic pruning is deployed in actual underwater detection tasks. The deployment platform can be an embedded processing unit of an underwater robot (ROV / AUV). During the detection phase, the underwater robot carries a sonar, a low-light camera, and a turbid water clearing camera equipment to synchronously collect image data of the target underwater structure in real time. The real-time image data of these three modalities will first be processed according to the three-modal data fusion strategy to form unified multi-modal input features. Subsequently, the deployed lightweight adaptive detection model receives and efficiently processes these fused multi-modal image data. The model uses its superior ability obtained through training on enhanced data to perform in-depth analysis on the input fused images, accurately identify various underwater structure defects, and output information such as defect category, location, and confidence. To demonstrate the superiority of the proposed method, the proposed method is compared with YOLOV11 and RT-DETRV2 on the same test set, and the results are shown in Table 1. As can be seen from Table 1, the accuracy of the proposed method is much better than that of YOLOV11 and RT-DETRV2 detection models, indicating that the proposed method has obvious advantages in underwater structure defect detection.

[0080] Table 1 Comparison of detection effects of existing methods and the proposed method

[0081] Parameters YOLOV11 RT-DETRV2 The method of the invention Image size 512*512 512*512 512*512 Precision P 88.2 92.1 95.9 Recall R 83.8 84.5 87.5 mAP 87.4 91.2 94.3 F1 score 0.859 0.881 0.915

[0082] While embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and variations can be made to these embodiments without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for detecting defects in underwater structures based on an adaptive deep learning model, characterized in that, include: Multimodal underwater image data is acquired as the first training set. Generative adversarial network is used to perform data augmentation and feature extraction on the first training set to obtain a second training set containing corresponding defect prototype representations and environmental feature vectors. An adaptive detection model is constructed, which includes a backbone feature extractor, a multi-scale feature pyramid, an adaptive attention module, and a defect knowledge embedding module that utilizes defect prototype representation and performs dynamic optimization. The second training set is ranked using environmental feature vectors and defect prototype representations. This ranking includes: dividing all underwater image training samples into environmental difficulty bins based on the weighted distance between their corresponding environmental feature vectors and a predefined set of reference environmental feature vectors at different discrete levels; within each environmental difficulty bin, ranking the samples from easy to identify to difficult to identify based on the comprehensive defect identifiability evaluation results; the comprehensive defect identifiability evaluation includes, for the originally labeled training samples, considering the initial classification confidence of the defect prototype representation associated with the training samples and the frequency with which the training samples were correctly identified during the initial training phase; for the enhanced samples generated by the generative adversarial network, determining the ranking based on the known identifiability characteristics of the defect prototypes used in their generation; optimizing using a combination of standard detection loss, prototype contrast loss, and auxiliary loss, where the auxiliary loss optimizes the objective by minimizing the differences between features in a predefined metric space; introducing adversarial training into the feature layer of the adaptive detection model to continuously align and fine-tune the defect prototype embedding vectors in the defect knowledge embedding module, resulting in the trained adaptive detection model. The trained adaptive detection model is deployed to the device to detect defects in underwater structures.

2. The underwater structure defect detection method based on adaptive deep learning model training according to claim 1, characterized in that, The environment encoder in the generative adversarial network is designed to receive and process multimodal underwater image data acquired and aligned synchronously from at least two types of sensors, including imaging sonar devices and at least one optical imaging device. The environment encoder processes each modal data, including acoustic features and visual features, through independent feature extraction branches. A multimodal feature fusion module performs deep interaction and fusion of the features extracted from each independent branch to output a fixed-dimensional environment feature vector, which contains acoustic characteristics and visual environment information.

3. The underwater structure defect detection method based on adaptive deep learning model training according to claim 2, characterized in that, The defect prototype decoder in the generative adversarial network converts the defect prototype representation into a defect prototype latent vector, which is then fused with the environmental feature vector through a cross-attention mechanism. The fused vector is then upsampled through multiple transposed convolutions to gradually improve the spatial resolution. During the upsampling process, the weights of the convolutional layers and the scaling and translation parameters of the batch normalization layer are dynamically generated and applied by a small auxiliary network based on the environmental feature vector, ensuring that the generated defect instances conform to the imaging characteristics of the target underwater environment in terms of morphology, edges, acoustic reflection properties, and texture.

4. The underwater structure defect detection method based on adaptive deep learning model training according to claim 1, characterized in that, The adaptive attention module in the adaptive detection module has a built-in fusion gating unit. The fusion gating unit receives the global feature vector obtained by global average pooling and the semantic feature vector extracted from the multi-scale feature pyramid. The global feature vector and the semantic feature vector are concatenated and then input into a gating network containing fully connected layers and non-linear activation functions. The activation values ​​of the output layer of the gating network are normalized by the Softmax function to form a set of dynamic weighting coefficients with the same number as each level of the multi-scale feature pyramid. The dynamic weighting coefficients are applied to the feature maps of the corresponding levels.

5. The underwater structure defect detection method based on adaptive deep learning model training according to claim 1, characterized in that, The defect knowledge embedding module in the adaptive detection model uses defect prototype representation to initialize a set of vectors whose dimensions match the image feature embedding dimensions, as defect prototype embedding vectors. During forward propagation of the network, for the input image region features, the defect knowledge embedding module calculates the similarity score between each defect prototype embedding vector through a learnable linear transformation, uses the similarity score to perform a weighted summation of the defect prototype embedding vectors, and fuses the result with the original image region features. At the same time, during backpropagation, the defect prototype embedding vectors are updated based on the gradient of the loss function.

6. The underwater structure defect detection method based on adaptive deep learning model training according to claim 1, characterized in that, The prototype contrast loss is specifically implemented as follows: the image feature embeddings representing each defect region to be evaluated, extracted from the adaptive detection model, are defined as anchor points; within the current training data batch, for the anchor point, a positive sample feature embedding is selected, the positive sample feature embedding and the anchor point belong to the same defect category, and the L2 distance between the positive sample feature embedding's own feature embedding and the defect prototype representation corresponding to the defect category to which the anchor point belongs is minimized; a negative sample feature embedding is selected, the negative sample feature embedding and the anchor point belong to different defect categories, and the L2 distance between the negative sample feature embedding's own feature embedding and the anchor point's feature embedding is minimized; the objective of the prototype contrast loss is to minimize the L2 distance between the anchor point and the positive sample feature embedding, while maximizing the L2 distance between the anchor point and the negative sample feature embedding, ensuring that the difference between the two is greater than a preset boundary margin.

7. The underwater structure defect detection method based on adaptive deep learning model training according to claim 1, characterized in that, The introduction of adversarial training into the feature layer of the adaptive detection model includes: constructing a discriminator network composed of fully connected layers, wherein the discriminator network receives the output of the convolutional layer blocks in the adaptive detection model and the feature vector formed by global average pooling as input; the training objective of the discriminator network is to output a binary classification prediction probability to indicate whether the input features it receives originate from the imaging characteristics of simple and complex underwater environments, wherein the simplicity and complexity of the environment are divided according to the environmental difficulty binning definition in the course learning; the loss function of the adaptive detection model includes an adversarial loss passed from the discriminator network through a gradient inversion layer; the adversarial loss guides the adaptive detection model to optimize the feature extraction process, and through the optimization, the intermediate layer feature vectors generated by the adaptive detection model when processing images from different underwater environment imaging characteristics are more statistically similar, thereby enabling the adaptive detection model to extract essential defect features that are insensitive to environmental changes.

Images