Bridge underwater disease image data set construction method based on generative adversarial network

Abstract

The invention discloses a generative adversarial network-based bridge underwater disease image data set construction method, which comprises the following steps of: A, acquiring an underwater structure disease image, and carrying out pixel-level fine labeling on a disease region to generate a disease mask image; performing size normalization and data standardization preprocessing on the original image and the mask image, and constructing a standardized training data set; b, constructing a deep convolutional generative adversarial network DCGAN as a mask generation model, and performing adversarial training by taking a real disease mask image as a training sample; using the trained DCGAN model to generate bridge underwater disease mask composite images in batches; c, constructing a ControlNet condition generative adversarial network model, inputting the generated disease mask composite image into the trained ControlNet model, and outputting a bridge underwater disease composite image; d, establishing a corresponding file naming relationship for the mask image and the underwater disease image; and storing the paired images and generating a data set index file, and constructing a complete bridge underwater disease image data set.

Description

Technical Field

[0001] This invention relates to a method for constructing an underwater bridge defect image dataset based on generative adversarial networks (GANs). The method uses GANs to generate defect mask images. Based on the synthesized defect mask images, a conditional GAN ​​model is used to generate corresponding underwater bridge defect images, thereby expanding the scale of the underwater bridge defect dataset and improving the performance of the underwater defect detection model. This invention belongs to the field of structural defect detection. Background Technology

[0002] The long-term safe operation of bridge structures is crucial for public safety and economic benefits. Due to the complex and variable underwater environment, the surface of underwater bridge structures is prone to various defects such as cracks, spalling, and wear, jeopardizing the structure's load-bearing capacity and service life. Therefore, regular and effective inspections of underwater bridge defects are necessary to assess the bridge's health status.

[0003] In recent years, deep learning-based computer vision technology has made significant progress in the field of structural defect detection. The performance of deep learning-based defect detection models relies on large-scale, high-quality labeled datasets. Currently, applying deep learning-based defect detection technology to underwater bridge defect detection still faces challenges. Underwater detection operations are costly, the environment is complex, and factors such as water visibility and lighting conditions make it difficult to obtain clear and usable images, resulting in difficulties in establishing large-scale underwater bridge defect datasets. Some studies have attempted to use image generation models to synthesize underwater bridge defect images to expand the training dataset. However, underwater images suffer from severe color distortion and low contrast, and image generation models pre-trained in common aquatic environments fail to learn the unique properties of the underwater environment, making it difficult to directly transfer and generate simulated underwater defect images. Furthermore, existing methods are mostly limited to generating defect images, while defect location information still requires manual annotation, resulting in low dataset construction efficiency. The scarcity and incompleteness of underwater bridge defect datasets limit the defect detection capabilities of deep learning models based on computer vision.

[0004] Therefore, underwater bridge defect detection based on deep learning and computer vision technologies requires a method that can generate high-quality underwater bridge defect images and their corresponding defect annotations. This would overcome the limitation of scarce underwater bridge defect image data and provide sufficient data support for training robust and efficient underwater bridge defect detection models. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a method for constructing an underwater bridge defect image dataset based on generative adversarial networks, which can be used to accurately identify defects in underwater bridge components.

[0006] A method for constructing an underwater bridge damage image dataset based on generative adversarial networks includes the following steps:

[0007] A. Obtain raw underwater structural damage images from multiple sources, and perform pixel-level fine annotation on the damaged areas in the images to generate damage mask images; perform size normalization and data standardization preprocessing on the raw images and mask images to construct a standardized training dataset;

[0008] B. Construct a deep convolutional generative adversarial network (DCGAN) as a mask generation model, and perform adversarial training using real bridge lesion mask images as training samples; use the trained DCGAN model to generate a variety of synthetic images of underwater bridge lesion masks in batches.

[0009] C. Construct a conditional generative adversarial network model ControlNet, using real defect mask images as spatial conditions and corresponding underwater defect images as generation targets. Train the model to learn the mapping relationship between the mask and the image. Input the generated defect mask composite image into the trained ControlNet model and output a composite image of underwater bridge defects.

[0010] D. Establish corresponding file naming relationships between all generated masked images and their corresponding underwater defect images; store the paired images and generate a dataset index file to construct a complete underwater defect image dataset for bridges.

[0011] Furthermore, step A specifically includes:

[0012] A1. Collect original underwater images of bridge defects using underwater robots, divers, or publicly available databases;

[0013] A2. Pixel-level annotation of the defect areas in the original underwater bridge structure image to generate a true defect mask image;

[0014] A3. Perform preprocessing operations such as size normalization and data format standardization on the original underwater structure image and its corresponding real defect mask image;

[0015] A4. Pair the preprocessed images with the masked images to form a training dataset for model training.

[0016] Furthermore, step B specifically includes:

[0017] B1. Construct a mask generation model based on Deep Convolutional Generative Adversarial Network (DCGAN). The generator of this mask generation model is used to learn the feature distribution of real disease masks and generate mask images, while the discriminator is used to distinguish between real masks and masks generated by the generator. The generator takes a random noise vector as input and outputs a synthesized mask image through multiple convolutional layers. The discriminator takes the mask images synthesized by the generator as input and outputs the authenticity judgment results of these synthesized images.

[0018] B2. Train the mask generation model DCGAN by using arbitrary bridge real-world defect mask images as the training set, and continuously optimize the generator and discriminator of the model through adversarial training.

[0019] B3. Input random noise vectors that conform to a preset distribution into the trained mask generation model DCGAN. Through the forward propagation of the mask generation model DCGAN, batch-generate composite images of disease masks with similar morphological features to real disease masks.

[0020] Furthermore, step C specifically includes:

[0021] C1. Construct a conditional generative adversarial network model, ControlNet, based on mask conditions. This model takes a mask image of underwater bridge defects as input, uses a pre-trained Stable Diffusion (SD) network as its base, adds a trainable control branch to process the mask input, and outputs a synthetic image of underwater bridge defects consistent with the mask space structure.

[0022] C2. The collected real defect mask images are used as spatial conditions, and the corresponding original underwater bridge defect images are used as generation targets to form training sample pairs;

[0023] C3. The ControlNet conditional image generation model is trained, and the encoder extracts spatial control features from the input mask image. These features are then fused into the encoder of the base network Stable Diffusion through zero-convolutional layers, guiding the model to generate underwater disease images that correspond to the disease locations in the input mask during the decoding process.

[0024] C4. Input the composite image with the defect mask into the trained conditional image generation model ControlNet to generate composite images of underwater bridge defects corresponding to the mask in batches.

[0025] Furthermore, step D specifically includes:

[0026] D1. Establish the correspondence between the generated defect mask composite image and the composite underwater defect image of the bridge;

[0027] D2. Store the paired masked images and the underwater structure images of the bridge in a common data format that can be directly read by deep learning models;

[0028] D3. Generate a dataset index file that records the storage paths of all paired images and their corresponding relationships, including image size, generation parameters, and other data. Ultimately, construct a large-scale, high-quality underwater bridge defect image dataset to provide data support for training the underwater defect detection model.

[0029] The present invention has the following advantages:

[0030] (1) Through a two-level generative model architecture, the automatic generation of disease masks and underwater bridge disease images was realized. This design effectively avoids the pattern confusion and training instability problems that may occur when disease images and their masks are generated simultaneously, and improves the generation quality of underwater bridge disease dataset images and the accuracy of mask annotation;

[0031] (2) By generating a mask image and then generating a disease image based on the mask, the consistency between the generated underwater disease image of the bridge and the generated disease mask in terms of position and shape can be automatically achieved without manual annotation or correction, thus improving the efficiency of dataset construction.

[0032] (3) The proposed dataset generation method reduces the dependence on training data. The mask generation model only requires a general bridge defect mask and does not require specific underwater data; while the image generation model only requires a small number of underwater bridge defect images and masks for transfer learning to generate underwater environmental features, thereby constructing a high-quality underwater bridge defect image dataset. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method of the present invention;

[0034] Figure 2 This invention describes the process of generating disease mask images based on DCGAN.

[0035] Figure 3 This invention describes the process of generating underwater bridge damage images based on the ControlNet conditional generative adversarial network model.

[0036] Figure 4a This is the original real mask image input into the mask generation model in this invention. Figure 4b The disease mask image is generated using the method proposed in this invention. Figure 4c These are underwater bridge damage images generated based on these masked images. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings.

[0038] The implementation method of the present invention is as follows Figure 1 As shown, a method for constructing an underwater bridge damage image dataset based on generative adversarial networks specifically includes the following steps:

[0039] A. Obtain raw underwater structural damage images from multiple sources, and perform pixel-level fine annotation on the damaged areas in the images to generate damage mask images; perform size normalization and data standardization preprocessing on the raw images and mask images to construct a standardized training dataset;

[0040] B. Construct a deep convolutional generative adversarial network (DCGAN) as a mask generation model, and perform adversarial training using real bridge lesion mask images as training samples; use the trained DCGAN model to generate a variety of synthetic images of underwater bridge lesion masks in batches.

[0041] C. Construct a conditional generative adversarial network model ControlNet, using real defect mask images as spatial conditions and corresponding underwater defect images as generation targets. Train the model to learn the mapping relationship between the mask and the image. Input the generated defect mask composite image into the trained ControlNet model and output a composite image of underwater bridge defects.

[0042] D. Establish corresponding file naming relationships between all generated masked images and their corresponding underwater defect images; store the paired images and generate a dataset index file to construct a complete underwater defect image dataset for bridges.

[0043] Step A specifically includes:

[0044] A1. Use underwater robots to capture videos and extract keyframes from bridge pile foundations, abutments, and other parts; organize professional divers to inspect the underwater structure of the bridge and take high-definition photos; select images that match the characteristics of underwater bridge defects from public databases and establish an original image library of underwater bridge defects covering different water quality and lighting conditions.

[0045] A2. Using an image annotation tool, perform pixel-level fine annotation on the diseased areas in the original image to generate a binary mask image with the same resolution as the original image. In this mask image, the value of diseased pixels is set to 1, and the value of non-diseased pixels is set to 0.

[0046] A3. All collected raw underwater disease images and their corresponding mask images are uniformly scaled to a fixed resolution (e.g., 512×512 pixels), and pixel values ​​are normalized to the range [0, 1]. Finally, a normalized training dataset is constructed. ,in This represents the i-th original underwater defect image. This represents the corresponding actual disease mask image.

[0047] Step B specifically includes:

[0048] B1. A deep convolutional generative adversarial network (DCGAN) model is used as the mask generation model. For example... Figure 2 As shown, its input is a random noise vector that conforms to a normal distribution, and after multi-layer convolution operation, the output is a synthetic lesion mask image with a resolution of 512×512.

[0049] B2. Use the actual defect mask image as input. The defect mask image can be an annotated image of the collected underwater defects, or a defect mask image of any bridge. Some input images are shown below. Figure 4a As shown, areas with a pixel value of 1 are displayed in white for easier visualization. The mask generation model is iteratively optimized through adversarial training, prompting the generator to continuously learn the morphological characteristics of real-world diseases.

[0050] B3. After model training, generate random noise vectors conforming to a standard normal distribution N(0,1), with the vector dimension consistent with the generator input requirements. Input the random noise vectors into the trained DCGAN generator, and generate diverse composite disease mask images in batches through forward propagation. The generated disease mask images are as follows. Figure 4b As shown, these synthetic masked images are morphologically similar to real diseases and exhibit diversity.

[0051] Step C specifically includes:

[0052] C1. The ControlNet model is used to learn the mapping relationship between the mask and the image. The model structure is as follows: Figure 3 As shown, this model uses a pre-trained Stable Diffusion v1.5 as its base network, adding a trainable control branch to its U-Net encoder. The control branch is connected to the backbone network through zero convolutional layers, ensuring that the introduction of control conditions in the early stages of training does not affect the pre-trained model.

[0053] C2. Using 150 real-world bridge defect mask images from the training dataset as spatial conditions and 150 corresponding original underwater bridge defect images as generation targets, training sample pairs are formed. Since the generated bridge defect images are of the same category, the ControlNet text prompts are all set to the same value.

[0054] C3. During training, real disease mask images are input into the ControlNet conditional encoder, and multi-level spatial control features are extracted through a convolutional neural network. These features are then fused into the base network StableDiffusion (SD) U-Net encoder through zero-convolutional layers. The model is then guided to generate underwater disease images that correspond to the disease locations in the input mask during decoding.

[0055] C4. Input the composite image of the defect mask generated in step B3 into the trained ControlNet model sequentially. The output is a composite image of the bridge's underwater defects that strictly corresponds to the input mask in terms of defect location and shape, and has realistic underwater texture. The output underwater bridge defect image is as follows. Figure 4c As shown.

[0056] Step D specifically includes:

[0057] D1. Establish the correspondence between the composite image of the crack mask and the composite image of the underwater structure crack. This relationship is achieved through a unified and related file name. For example, the mask image is named "mask_001" and the corresponding composite image is named "image_001".

[0058] D2. Store the paired masked images and underwater bridge structure images in PNG format so that the deep learning model can read them directly;

[0059] D3. Generate a dataset index file, which records the storage paths of all paired images and their corresponding relationships in CSV format. It also includes metadata such as image size and generation parameters, which facilitates loading and use in subsequent machine learning tasks.

[0060] This invention achieves the synthesis of underwater bridge defect dataset images and corresponding location mask images through a two-stage image generation framework. It effectively overcomes the shortcomings of high cost and insufficient data volume in real underwater bridge data acquisition, and can provide a dataset foundation for training high-performance underwater bridge defect detection models.

[0061] The embodiments described in this specification are merely examples of how to implement the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms described in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for constructing a bridge underwater disease image dataset based on a generative adversarial network, characterized in that, The method comprises the following steps: A. Obtain original underwater structure disease images from multiple sources, and perform pixel-level fine labeling on the disease areas in the images to generate disease mask images; perform size normalization and data standardization preprocessing on the original images and the mask images, and construct a standardized training data set; B. Construct a deep convolutional generative adversarial network DCGAN as a mask generation model, and perform adversarial training using real disease mask images as training samples; use the trained DCGAN model to batch generate diverse bridge underwater disease mask synthetic images; C. Construct a conditional generative adversarial network model ControlNet, use real disease mask images as spatial conditions, and corresponding underwater disease images as generation targets to train the model to learn the mapping relationship between the mask and the image; input the generated disease mask synthetic image into the trained ControlNet model, and output the bridge underwater disease synthetic image; D. Establish a corresponding file naming relationship for all generated mask images and corresponding underwater disease images; Store the paired images and generate a data set index file to construct a complete bridge underwater disease image data set. 2.The bridge underwater disease image dataset construction method based on a generative adversarial network of claim 1, wherein, Step A specifically comprises: A1. Collect original bridge underwater disease images through underwater robots, divers, or public databases; A2. Perform pixel-level labeling on the disease areas in the original bridge underwater images to generate real disease mask images, with different values for disease pixels and non-disease pixels; A3. Perform size normalization, data format standardization, and other preprocessing operations on the original bridge underwater disease images and their corresponding disease mask images; A4. Pair the preprocessed images and mask images to form a training data set for model training. 3.The bridge underwater disease image dataset construction method based on a generative adversarial network of claim 1, wherein, Step B specifically comprises: B1. Construct a mask generation model based on the deep convolutional generative adversarial network DCGAN, which includes a generator for learning the feature distribution of real disease masks and generating mask images, and a discriminator for distinguishing between real masks and masks generated by the generator; the generator takes a random noise vector as input and outputs synthetic mask images through multiple convolutional layers, and the discriminator takes the synthetic mask images generated by the generator as input and outputs the authenticity judgment results of these synthetic images; B2. Train the mask generation model DCGAN using any real bridge disease mask image as a training set, and continuously optimize the generator and discriminator of the model through adversarial training; B3. Input a random noise vector conforming to a preset distribution into the trained mask generation model DCGAN, and batch generate disease mask synthetic images with similar morphological features as real disease masks through forward propagation of the mask generation model DCGAN. 4.The bridge underwater disease image dataset construction method based on a generative adversarial network of claim 1, wherein, Step C specifically comprises: C1. Construct a conditional generative adversarial network model ControlNet based on the mask condition. The ControlNet model inputs the disease mask image of the bridge underwater structure, uses a pre-trained Stable Diffusion (SD) as the base network, adds a trainable control branch to process the mask input, and outputs an underwater disease synthetic image consistent with the mask spatial structure; C2. The collected real disease mask image is used as a spatial condition, and the corresponding original bridge underwater disease image is used as a generation target to form a training sample pair; C3. Train the conditional image generation model ControlNet, and the encoder extracts the spatial control features of the input mask image. These spatial control features are fused into the encoder of the basic network Stable Diffusion through a zero convolution layer, guiding the model to generate underwater disease images consistent with the input mask disease position during the decoding process; C4. Input the disease mask synthetic image into the trained conditional image generation model ControlNet to batch generate bridge underwater disease synthetic images corresponding to the mask.

5. The bridge underwater disease image dataset construction method based on a generative adversarial network according to claim 1, characterized in that, Step D specifically includes: D1. Establish the correspondence between the generated disease mask synthetic image and the bridge underwater disease synthetic image, and realize it through a unified and related file name; D2. Store the paired mask image and bridge underwater structure image in a general image format for direct reading by a deep learning framework; D3. Generate a data set index file to record the storage path and corresponding relationship of all paired images, including image size, generation parameters, and other data. Finally, a large-scale and high-quality bridge underwater disease image data set is constructed to provide data support for the training of underwater disease detection models.

Images