Dam underwater concrete spalling detection method based on weak supervision small sample learning

Abstract

The invention discloses a dam underwater concrete spalling detection method based on weak supervision small sample learning, and relates to the technical field of dam underwater target detection. According to the method, a deep convolutional generative adversarial network is used as a generative model, details in an image can be effectively processed, a high-quality composite image can be generated, and the problem that underwater concrete spalling images are rare is solved; a Faster R-CNN model with ResNet50 as a main network is used as a student model and a teacher model, and a self-training strategy of weak supervised learning is comprehensively used to reduce the adverse effect of the difference between synthetic data and real data distribution on the model, so that a high-precision dam underwater concrete spalling detection model is constructed. According to the method, high-precision underwater concrete peeling (new class) detection can be realized under the condition that only part of underwater concrete peeling (new class) images are marked, and the method can be further popularized and applied to structural defect detection of various wading hydraulic structures.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater target detection of dams, and in particular to a method for detecting underwater concrete spalling of dams based on weakly supervised small sample learning. Background Art

[0002] Water conservancy infrastructure is related to the national economy and people's livelihood, and has played an important role in economic development, social progress and environmental improvement. However, due to the long construction period, inadequate management and maintenance, natural disasters and other reasons, some water conservancy infrastructure currently has a high rate of danger and a severe safety situation. Therefore, it is necessary to further investigate the safety hazards of reservoir dams and strengthen risk removal and reinforcement work.

[0003] Among the hazards of dams, leakage caused by cracks and spalling is the main safety hazard of underwater concrete structures of dams, and cracks are the main hazard. However, spalling can reduce the effective anti-seepage area of ​​the dam and reduce its anti-seepage capacity, thereby increasing the risk of leakage. The underwater imaging technology carried by existing underwater robots is widely used in underwater defect detection due to its high resolution and detection accuracy. The target detection algorithm based on deep learning can replace manual processing of image information and extract underwater defect information end-to-end, greatly improving the detection efficiency and avoiding the influence of subjective factors.

[0004] However, most deep learning-based object detection algorithms require massive datasets with fine manual annotations for training in order to obtain a model that meets the expected detection effect. This method is suitable for underwater concrete crack detection of dams with a sufficient number of samples. When the existing underwater concrete crack detection model is applied to underwater concrete spalling detection with a small number of samples, it is often necessary to manually and finely annotate the underwater concrete spalling images and then retrain them. In addition, due to the small number of underwater concrete spalling images, the false detection rate and missed detection rate are high, which makes it difficult to meet the actual engineering needs. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned technologies, the present invention provides a method for detecting underwater concrete spalling of dams based on weakly supervised small sample learning. The present invention combines image generation technology and weakly supervised learning methods to achieve high-precision underwater concrete spalling (new class) detection by only annotating part of the underwater concrete spalling (new class) images.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] A method for detecting underwater concrete spalling of dams based on weakly supervised small sample learning includes the following steps:

[0008] Step 1: Exploring the underwater concrete structure of the dam, obtaining images of cracks and spalling, taking cracks as the base class to obtain a base class dataset, and taking spalling as the new class to obtain a new class dataset; all images in the base class dataset are finely annotated, and some images in the new class dataset are finely annotated, and the remaining images do not provide annotation information, which is used to generate synthetic images and pseudo labels;

[0009] Step 2: Initialize the deep convolutional generative adversarial network, build a generator for generating synthetic images and a discriminator for distinguishing between real images and generated images, and use the new class dataset to generate synthetic images;

[0010] Step 3: Select a model and train it using the "pre-training-fine-tuning" training method to obtain a teacher model;

[0011] Step 4: Reasoning the unlabeled image and the synthetic image through the teacher model to generate pseudo labels;

[0012] Step 5: Combine the dataset with labeled information, unlabeled images, synthetic images, and pseudo labels corresponding to the synthetic images to form a new dataset, use the new dataset to train the student model, and update the teacher model through the exponential moving average formula of the student model parameters;

[0013] Step 6: Repeat steps 4 and 5 to iteratively optimize the updated teacher model. The student model obtained after iterative optimization is the underwater concrete spalling detection model for dams.

[0014] Step 7: In actual application, the acquired image of the underwater concrete structure is input into the underwater concrete spalling detection model of the dam to identify spalling in the image.

[0015] Preferably, in step 1, the ratio of the number of crack and spalling images is 10:1; and in the spalling images of the new class data set, the ratio of annotated images to unannotated images is 1:4.

[0016] Preferably, in step 2, during the adversarial training of the deep convolutional adversarial generative network, the loss functions of the generator and the discriminator both adopt the cross entropy loss function.

[0017] Preferably, the model in step 3 is a Faster R-CNN model with ResNet50 as the backbone network, and the teacher model is obtained after training in a "pre-training-fine-tuning" training mode.

[0018] Preferably, the "pre-training-fine-tuning" includes a pre-training stage and a fine-tuning stage; the pre-training stage is to use a large number of labeled samples of the base class data set for training; the fine-tuning stage is to use a small balanced set of samples of the base class data set and the labeled new class data set to fine-tune the pre-trained model.

[0019] Preferably, the small sample balance set is 10 instance set samples randomly selected from the base class data set and the labeled new class data set.

[0020] Preferably, the step 4 further includes processing the generated pseudo-labels using a frame jittering technique to improve the quality of the pseudo-labels.

[0021] Preferably, the frame jitter technology includes translation jitter and scaling jitter.

[0022] Preferably, the student model in step 5 is a FasterR-CNN model with ResNet50 as the backbone network.

[0023] Preferably, the exponential moving average formula in step 5 is:

[0024] θ teacher =α·θ teacher +(1-α)·θ student ;

[0025] Where: θ teacher are the parameters of the teacher model; θ student are the parameters of the student model; α is the smoothing coefficient.

[0026] Preferably, in step 6, the number of iterative optimization times is determined by the actual engineering precision requirement, and the iterative optimization process can be stopped when the iterative optimization meets the actual engineering precision requirement, thereby obtaining a dam underwater concrete spalling detection model.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention uses a deep convolutional generative adversarial network (DCGAN) with relatively low model complexity as a generative model, which can effectively process the details in the image and generate high-quality synthetic images to alleviate the problem of scarce underwater concrete spalling images;

[0029] The present invention uses the FasterR-CNN model with ResNet50 as the backbone network as the student model and the teacher model, and adopts the self-training strategy of weakly supervised learning to train the detection model to reduce the adverse effects of the difference between the distribution of synthetic data and real data on the model;

[0030] The method of the present invention combines image generation technology and weakly supervised learning methods, and can achieve high-precision underwater concrete spalling (new class) detection when only part of the underwater concrete spalling (new class) images are labeled. It can be further promoted and applied to structural defect detection of various types of water-related hydraulic structures, greatly reducing the cost of manual labeling work, and has broad practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a workflow diagram of a method for detecting underwater concrete spalling of dams based on weakly supervised small sample learning according to the present invention;

[0032] Figure 2 A schematic diagram of the structure of a deep convolutional generative adversarial network (DCGAN network) in the present invention;

[0033] Figure 3 It is a schematic diagram of the structure of the FasterR-CNN model with ResNet50 as the backbone network in the present invention;

[0034] Figure 4 This is an example diagram of detection of underwater concrete spalling images using the method of the present invention. DETAILED DESCRIPTION

[0035] In order to better explain the present invention, the main contents of the present invention are further explained below in conjunction with the drawings and specific embodiments, but the contents of the present invention are not limited to the following embodiments.

[0036] The present invention provides a method for detecting underwater concrete spalling of dams based on weakly supervised small sample learning. The specific working process is as follows: Figure 1 As shown, the following steps are included:

[0037] Step 1: Exploring the underwater concrete structure of the dam, obtaining crack and spalling images, and obtaining base class data sets and new class data sets; taking cracks as the base class, all images are finely annotated, while taking spalling as the new class, only some images are finely annotated, and the remaining images are not provided with annotation information, which are used to generate synthetic images and pseudo labels;

[0038] As a preferred technical solution of the present invention, in step S1, the ratio of underwater concrete crack images (base class) to underwater concrete spalling images (new class) is 10:1, and in underwater concrete spalling images, the ratio of annotated images to unannotated images is 1:4.

[0039] Step 2: Initialize the deep convolutional generative adversarial network (DCGAN), build a generator for generating synthetic images and a discriminator for distinguishing between real images and generated images, and use the new class dataset (including labeled and unlabeled images) to generate high-quality synthetic images;

[0040] As a preferred technical solution of the present invention, in step 2, the structural diagram of the deep convolutional generative adversarial network (DCGAN) is as follows: Figure 2 As shown in Figure 2, the deep convolutional generative adversarial network (DCGAN) can be decomposed into two parts: the generator and the discriminator.

[0041] like Figure 2 As shown in the figure, the generator includes five transposed convolutional layers (ConvTrans), and the batch normalization layer (BN) and ReLU activation function are connected in sequence between adjacent transposed convolutional layers. The last transposed convolutional layer is connected to the Tanh function and then output; the convolution kernel size of each transposed convolutional layer is 4×4, and the stride of the second to fifth transposed convolutional layers is 2, and the padding is 1.

[0042] The discriminator consists of five convolutional layers (Conv), and the batch normalization layer (BN) and the Leaky ReLU activation function are connected in sequence between adjacent transposed convolutional layers. The last transposed convolutional layer is connected to the sigmoid function and then output. The convolution kernel size of each convolutional layer is 4×4, and the stride of the first to fourth convolutional layers is 2, and the padding is 1.

[0043] The deep convolutional generative adversarial network (DCGAN) adopted in the present invention is a generative data augmentation method selected for the small amount of data available for training. Compared with models with too low complexity, such as the original GAN ​​model, the image quality generated by it is better. At the same time, compared with high-complexity models such as StyleGAN, it is easier to converge during the training process and the generated images are more diverse.

[0044] The generator takes two-dimensional random noise as input, and after multiple transposed convolutions, batch normalization, and ReLU activation functions, generates realistic pseudo images to simulate underwater concrete spalling areas that are difficult to capture in actual scenes, thereby expanding the training data set, increasing sample diversity, and helping the model learn more robust underwater spalling feature representations. The discriminator takes real images or pseudo images generated by the generator as input, and after multiple convolutions, batch normalization, leaky ReLU activation functions, and sigmoid functions, it outputs the probability of the image being true or false. In the continuous adversarial training process, the model continuously learns the ability to distinguish between real images and generated images, which helps to enhance the model's ability to distinguish between underwater concrete spalling and non-underwater concrete spalling (background) features.

[0045] During adversarial training, the loss functions of both the generator and the discriminator adopt the cross entropy loss function. The training of the generator and the discriminator is performed alternately until the predetermined number of iterations is reached.

[0046] Among them, the formula of the cross entropy loss function is as follows:

[0047] min G max D L(D,G)=E x～pdata(x) [logD(x)]+E z～pz(z) [log(1-D(G(z)))]

[0048] Where: min G To minimize the loss function of the generator; max D is the loss function of maximizing the discriminator; L(D,G) is the loss function of the discriminator D and the generator G; E x～pdata(x) is the expected loss of the discriminator for the real underwater concrete spalling image x; p data (x) is the distribution of real underwater concrete spalling image data; D(x) is the probability that the judger judges that x is a real underwater concrete spalling image; E z～pz(z) is the expected loss of the discriminator for the generated image; p z (z) is the distribution of input noise; G(z) is the data generated by the generator based on the noise z; D(G(z)) is the probability that the discriminator judges that G(z) is a real underwater concrete spalling image.

[0049] Step 3: Select the teacher model and train it using the "pre-training-fine-tuning" method: In the pre-training stage, a large number of labeled samples of the base class data set are used for training; in the fine-tuning stage, the pre-trained model is fine-tuned using a small balanced set of labeled base and new classes (10 instance set samples are randomly selected from the base and new class data sets);

[0050] As a preferred technical solution of the present invention, in step 3, the teacher model used is a FasterR-CNN model with ResNet50 as the backbone network, and the corresponding structural diagram is shown in Figure 3 .

[0051] like Figure 3 As shown in the figure, the Faster R-CNN model with ResNet50 as the backbone network includes the backbone network ResNet50, the region proposal network (RPN), the region of interest pooling (RoIPooling), as well as the classifier and the bounding box regressor.

[0052] The specific processing process of the FasterR-CNN model with ResNet50 as the backbone network is as follows:

[0053] First, the image is scaled to meet the needs of subsequent processing. Then, the pre-trained ResNet50 network is used as a feature extractor to extract the image features of underwater concrete spalling. The generated feature map is then passed into the region proposal network (RPN) to generate candidate proposal boxes, that is, areas that may contain underwater concrete spalling. Each region of interest generated by the RPN is then converted into a feature map of a fixed size for unified subsequent processing. Finally, two parallel fully connected layers are used to classify and regress the bounding box of each region of interest, so as to determine whether the category of the target in the area is underwater concrete spalling and accurately locate the coordinate information of the target.

[0054] Step 4: Use the trained teacher model to infer the unlabeled images and the synthetic images generated by the generator to generate pseudo labels, and use the box jittering technique to improve the quality of the pseudo labels;

[0055] As a preferred technical solution of the present invention, in step 4, the frame jitter technology adopted includes translation jitter and scaling jitter, wherein the translation jitter is achieved by randomly translating the horizontal and vertical positions of the border in a small amplitude, and the scaling jitter is achieved by randomly scaling the width and height of the border in a small amplitude.

[0056] Step 5: Combine the dataset with labeled information, unlabeled images, synthetic images and their corresponding pseudo labels to form a new dataset, train the student model on the new dataset, and update the teacher model by the exponential moving average (EMA) of the student model parameters.

[0057] As a preferred technical solution of the present invention, in step S5, the student model used is the same detection model as the teacher model, that is, the Faster R-CNN model with ResNet50 as the backbone network. The exponential moving average (EMA) formula for updating the teacher model parameters is:

[0058] θ teacher =α·θ teacher +(1-α)·θ student

[0059] Where: θ teacher are the parameters of the teacher model; θ student is the parameter of the student model; α is the smoothing coefficient, which is generally close to 1 and is used to control the weight between the teacher model and the student model parameters.

[0060] Step 6: Repeat steps 4 and 5 for iterative optimization using the updated teacher model. After multiple iterations of optimization, the student model that meets the actual engineering accuracy requirements is used as the underwater concrete spalling detection model for dams.

[0061] Step 7: In actual application, the acquired image of the underwater concrete structure is input into the underwater concrete spalling detection model of the dam to distinguish the underwater cracks and spalling in the image, so as to identify the spalling in the image.

[0062] The underwater concrete spalling detection model of the dam constructed by the above method is used to detect the underwater concrete structure of the dam. The detection results are as follows: Figure 4 As shown, the results show that the method of the present invention can accurately identify underwater concrete spalling in various scenarios, and the confidence of the predicted bounding box is higher than 0.8, which means that the model has more than 80% confidence that the target box is underwater concrete spalling.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0064] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in this field.

Claims

1. A method for detecting underwater concrete spalling of dams based on weakly supervised small sample learning, characterized by: The following steps are involved: Step 1: Exploring the underwater concrete structure of the dam, obtaining images of cracks and spalling, taking cracks as the base class to obtain a base class dataset, and taking spalling as the new class to obtain a new class dataset; all images in the base class dataset are finely annotated, and some images in the new class dataset are finely annotated, and the remaining images do not provide annotation information, which is used to generate synthetic images and pseudo labels; Step 2: Initialize the deep convolutional generative adversarial network, build a generator for generating synthetic images and a discriminator for distinguishing between real images and generated images, and use the new class dataset to generate synthetic images; Step 3: Select a model and train it using the "pre-training-fine-tuning" training method to obtain a teacher model; Step 4: Reasoning the unlabeled image and the synthetic image through the teacher model to generate pseudo labels; Step 5: Combine the dataset with labeled information, unlabeled images, synthetic images, and pseudo labels corresponding to the synthetic images to form a new dataset, use the new dataset to train the student model, and update the teacher model through the exponential moving average formula of the student model parameters; Step 6: Repeat steps 4 and 5 to iteratively optimize the updated teacher model. The student model obtained after iterative optimization is the underwater concrete spalling detection model for dams. Step 7: In actual application, the acquired image of the underwater concrete structure is input into the underwater concrete spalling detection model of the dam to identify spalling in the image.

2. The method for detecting underwater concrete spalling of a dam according to claim 1, characterized in that: In the step 1, the ratio of the number of crack and spalling images is 10:1; in the spalling images of the new class data set, the ratio of labeled images to unlabeled images is 1:

4.

3. The method for detecting underwater concrete spalling of dams according to claim 1, characterized in that: In step 2, during the adversarial training of the deep convolutional adversarial generative network, the loss functions of the generator and the discriminator both adopt the cross entropy loss function.

4. The method for detecting underwater concrete spalling of a dam according to claim 1, characterized in that: The model in step 3 is a Faster R-CNN model with ResNet50 as the backbone network. After training in a "pre-training-fine-tuning" training mode, a teacher model is obtained.

5. The method for detecting underwater concrete spalling of a dam according to claim 4, characterized in that: The "pre-training-fine-tuning" includes a pre-training stage and a fine-tuning stage; the pre-training stage is to use a large number of labeled samples of the base class data set for training; the fine-tuning stage is to use a small balanced set of samples of the base class data set and the labeled new class data set to fine-tune the pre-trained model.

6. The method for detecting underwater concrete spalling of a dam according to claim 5, characterized in that: The small sample balance set is 10 instance set samples randomly selected from the base class data set and the labeled new class data set.

7. The method for detecting underwater concrete spalling of a dam according to claim 1, characterized in that: The step 4 also includes processing the generated pseudo-labels using a frame jittering technique to improve the quality of the pseudo-labels.

8. The method for detecting underwater concrete spalling of a dam according to claim 7, characterized in that: The frame jittering technique includes translation jittering and scaling jittering.

9. The method for detecting underwater concrete spalling of a dam according to claim 1, characterized in that: The student model in step 5 is a Faster R-CNN model with ResNet50 as the backbone network.

10. The method for detecting underwater concrete spalling of dams according to claim 1, characterized in that: The exponential moving average formula in step 5 is: i teacher =a·θ teacher +(1-a)·θ student ; Where: θ teacher are the parameters of the teacher model; θ student are the parameters of the student model; α is the smoothing coefficient.

11. The method for detecting underwater concrete spalling of dams according to claim 1, characterized in that: In step 6, the number of iterative optimizations is determined by the actual engineering precision requirements. When the iterative optimization meets the actual engineering precision requirements, the iterative optimization process can be stopped to obtain the underwater concrete spalling detection model for the dam.

Images