Dam underwater defect image generation method based on deep learning and underwater optical model

Abstract

The invention discloses a dam underwater defect image generation method based on deep learning and an underwater optical model, and the method comprises the following steps: shooting an apparent defect image of a dam, and constructing a dam overwater defect image data set; underwater images of the dam are collected, and an underwater style image data set is constructed; combining the two data sets to obtain a non-paired dam overwater and underwater image data set; building an unsupervised generative adversarial network, and inputting a dam overwater and underwater image data set for training to obtain an overwater and underwater image style migration model; establishing an underwater optical imaging model, and introducing an underwater image attenuation operator to obtain an underwater optical imaging attenuation model; and performing style migration on the dam overwater defect image by using the overwater and underwater image style migration model and the underwater optical imaging attenuation model, and converting the dam overwater defect image into a dam underwater defect image data set with a real underwater image style. According to the invention, the problems of difficult dam underwater defect image acquisition and small sample size in various deep learning tasks can be solved.

Description

Technical Field

[0001] The present invention belongs to the fields of dam safety detection, deep learning and image generation, and specifically relates to a method for generating underwater defect images of dams based on deep learning and underwater optical models. Background Art

[0002] Reservoir dams play an important role in major livelihood industries that promote economic and social development, such as flood control, power generation and irrigation. Dam inspection and maintenance are necessary means to protect the safe and stable operation of dams. At present, there are many schemes for defect detection of the above-water part of dams. However, underwater defect detection of dams mainly relies on divers carrying tools to enter the water to inspect the surface of underwater dam bodies, energy dissipation basins, water cushion ponds and other hydraulic structures. However, the underwater part of large reservoir dams is large in area and wide in range. This detection method is not only time-consuming and labor-intensive, but also dangerous. Underwater robots can carry a variety of detection equipment to perform underwater defect detection tasks, and are becoming an important technical means for underwater defect detection of dams.

[0003] Based on deep learning and computer vision technology, by detecting underwater images taken by underwater robots, it is possible to determine whether there are defects. Fan et al. proposed an underwater dam crack detection and segmentation algorithm based on multi-level adversarial transfer learning; Li et al. proposed a real-time underwater damage detection method based on deep learning and computer vision, which can effectively detect underwater structural damage; Ma et al. proposed a defuzzification algorithm and defect segmentation method to detect damage to sewage pipes in real time. These methods based on deep learning technology require a large amount of underwater defect image data to train and optimize deep learning models. However, it is difficult to obtain underwater defect images of dams. It is necessary to operate underwater equipment such as underwater robots to shoot in the underwater environment of the dam, and the number of underwater defect samples of dams is small, which limits the development of advanced underwater defect detection methods for dams based on deep learning technology. Therefore, it is difficult to obtain underwater defect image data of dams, which cannot meet the growing development needs of research on underwater defect detection methods for dams based on deep learning. Summary of the invention

[0004] In view of this, the purpose of the present invention is to propose a method for generating underwater defect images of dams based on deep learning and underwater optical models, so as to solve the problems of difficulty in acquiring underwater defect images of dams and small sample size in various deep learning tasks.

[0005] The technical solution of the present invention:

[0006] A method for generating underwater defect images of a dam based on deep learning and underwater optical model comprises the following steps:

[0007] S1: Use optical equipment (drones and cameras, etc.) to take photos of the apparent defects of the dam, and establish a dataset of dam water defect images after processing;

[0008] S2: Use an underwater robot equipped with an underwater high-definition camera to take underwater images of the dam, filter images with underwater style features, and establish an underwater style image dataset;

[0009] S3: Processing the dam above-water defect image dataset and the underwater style image dataset to establish an unpaired dam above-water and underwater image dataset;

[0010] S4: building an unsupervised generative adversarial network, using the above-water and underwater image dataset of the dam to train the generative adversarial network, obtaining and saving the above-water and underwater image style transfer model;

[0011] S5: establishing an underwater optical imaging model, introducing an underwater optical attenuation operator, and obtaining an underwater optical imaging attenuation model;

[0012] S6: Use the above-water and underwater image style transfer model and the underwater optical imaging attenuation model to perform style transfer on the above-water defect images of the dam, establish an underwater defect image dataset of the dam with a real underwater image style, and use the underwater defect image dataset of the dam to perform various deep learning tasks.

[0013] Furthermore, the step of establishing a dataset of dam water defect images described in step S1 includes: cropping and screening images of dam surface defects taken by equipment such as drones and cameras, so that the defects account for a large proportion of the target in the image and the defect features are obvious.

[0014] Furthermore, the step of establishing an underwater style image dataset described in step S2 includes: operating an underwater robot to shoot different hydraulic structures (such as the surface of a dam, a cushion pond, and an energy dissipation basin) at different water depths in the reservoir, and intercepting and cropping the acquired video to obtain underwater images containing problems such as uneven lighting, color cast, and blur degradation.

[0015] Furthermore, the step of establishing an unpaired dam above-water and underwater image dataset in step S3 includes: placing the dam above-water defect image dataset and the underwater style image dataset in folders A and B respectively, and placing the A and B folders in the dam above-water and underwater image dataset up2un folder.

[0016] Furthermore, the step of obtaining the above-water and underwater image style transfer model in step S4 includes:

[0017] Build a generative adversarial network, where the adversarial loss can be expressed as:

[0018]

[0019] Among them, G is the generator, D is the discriminator, X represents the dam water defect dataset, and Y represents the underwater style image dataset. represents the mathematical expectation of function f(x) under distribution X. It represents the mathematical expectation of function f(y) on y under distribution Y.

[0020] The dam water and underwater image dataset up2un is divided into a training set and a test set in proportion, and the appropriate number of training epochs and hyperparameter λ are set and optimized to train the optimal water and underwater image style transfer model.

[0021] Furthermore, the step of obtaining the underwater optical imaging attenuation model in step S5 includes:

[0022] The underwater optical imaging model is established according to the underwater optical imaging formula:

[0023] I c (x) = J c (x)T c (x)+B c (x)(1-T c (x)),c∈{R,G,B}

[0024] Among them, x is the pixel coordinate, I c (x) is the c channel of the underwater image captured by the camera, R, G, B represent the RGB color channels of the image respectively, and J c (x) is the undegraded image, B c (x) is the global background light of color c channel, T c (x) is the medium transmittance of channel c, expressed as:

[0025] T c (x) = exp(-β c d(x))c∈{R,G,B}

[0026] Among them, β c is the attenuation coefficient of color channel c, and d(x) represents the distance from the optical sensor to the object.

[0027] An underwater optical attenuation operator including uneven illumination, color cast and blur is introduced into the underwater optical imaging model.

[0028] Taking into account the fill light and camera carrier movement in the real underwater shooting process, the attenuation operators Mask(·) and Blurring(·) are introduced into the underwater optical imaging model. The Mask(·) operator is used to simulate the uneven image brightness degradation caused by the fill light of a single light source underwater. The illumination mask is used to simulate the attenuation effect of the light source in the water, so that the image looks closer to the real underwater environment and obtains a non-uniform illumination image. The attenuation effect of the light source is simulated by creating a brightness map that gradually decreases from the center to the outside. The illumination mask uses a Gaussian distribution to calculate the brightness value. The calculation formula is as follows. First, the distance from each pixel to the center of the light source is calculated.

[0029]

[0030] Among them, (x c ,y c ) is the coordinate of the center of the light source, and (x,y) is the coordinate of the current pixel.

[0031] Secondly, the light source intensity is calculated according to the Gaussian distribution:

[0032]

[0033] Among them, σ is the standard deviation of the Gaussian distribution, and intensity is the maximum light source intensity. This formula indicates that the light source intensity decays exponentially with increasing distance.

[0034] Finally, the mask value is normalized to the range [0, 1].

[0035] The Blurring (·) operator is used to simulate the motion blur of underwater images caused by the movement of the carrier during the exposure time, using Gaussian blur:

[0036]

[0037] Among them, GaussianBlur represents Gaussian blur, Represents convolution.

[0038] Color cast is caused by the different attenuation rates of light of different wavelengths in water. The images of the three color channels of R, G, and B are attenuated according to the physical laws of underwater optical imaging and fused in different proportions. After integration, the final formula for generating the underwater defect image of the dam is:

[0039] I(x)=Blurring{Mask[I R (x)+I G (x)+I B (x)]}

[0040] Furthermore, step S6 of using the method to establish a dam underwater defect image dataset with a real underwater image style includes: inputting the dam above-water defect image dataset into an above-water and underwater image style transfer model and an underwater optical imaging attenuation model, and selecting images with a real underwater style based on manual subjective screening and objective evaluation indicators PSNR and SSIM. The selected images not only contain the characteristics of the dam's apparent defects, but also conform to the underwater imaging style, and can be used for training deep learning models of dam underwater defects such as intelligent recognition, target detection, and semantic segmentation.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] Based on deep learning generative adversarial networks and underwater imaging physical models, the present invention converts dam water defect images into dam underwater defect images with the style of real underwater images with uneven lighting, color cast and blur characteristics, solving the problems of difficulty in acquiring dam underwater defect images and small sample size in various deep learning tasks, and providing data support for the training of dam underwater defect deep learning models for downstream tasks such as intelligent recognition, target detection and semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a method flow chart provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of the network structure of the generative adversarial network in the present invention;

[0045] Figure 3 Schematic diagram of underwater optical imaging attenuation model in the present invention;

[0046] Figure 4 It is the above-water defect image of the dam and the corresponding generated underwater defect image in the present invention. DETAILED DESCRIPTION

[0047] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0048] In order to deepen the understanding of the present invention by those skilled in the art, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. Figure 1 FIG. 1 is a flow chart of a method provided by an embodiment of the present invention, and the method specifically comprises the following steps:

[0049] (1) Establishing a dam water defect image dataset

[0050] Optical equipment such as drones and cameras are used to photograph the apparent defects of the dam, and after processing, a dataset of dam surface defect images is established. The processing steps include: cropping and screening the acquired dam apparent defect images so that the defects account for a large proportion of the target in the image and the defect features are obvious;

[0051] (2) Establishing an underwater style image dataset

[0052] In order to ensure the diversity and high quality of the generated underwater defect images of the dam, it is necessary to operate an underwater robot to shoot different hydraulic structures (such as the surface of the dam body, cushion pond and stilling basin) at different water depths in the reservoir, intercept and crop the acquired videos to obtain underwater images with uneven illumination, color cast, blur and degradation problems, and establish an underwater style image dataset;

[0053] (3) Establish an unpaired dam above-water and underwater image dataset

[0054] The dam water defect image dataset and the underwater style image dataset are placed in folders A and B respectively. A represents the dam water defect image and B represents the underwater style image. The A and B folders are placed in the up2un folder of the dam water and underwater dataset.

[0055] (4) Build a style transfer model for underwater and above-water images

[0056] First, build an unsupervised generative adversarial network. Figure 2 As shown in the figure, the unsupervised generative adversarial network is established, which includes contrast loss. Through training, the style of underwater images is transferred to the images of dam water defects. The adversarial loss of the generative adversarial network is expressed as:

[0057]

[0058] Among them, G is the generator, D is the discriminator, X represents the dam water defect dataset, and Y represents the underwater style image dataset. represents the mathematical expectation of function f(x) under distribution X. It represents the mathematical expectation of function f(y) on y under distribution Y.

[0059] Secondly, the unsupervised generative adversarial network is trained and optimized. The up2un dataset of dam water and underwater images is divided into training set and test set in a ratio of 9:1. The model is optimized through training, and the appropriate number of training epochs and hyperparameters λ are set. The generated images are comprehensively evaluated based on manual subjective screening and objective evaluation indicators PSNR and SSIM. The subjective judgment standard is to have underwater imaging style, and the standard of objective evaluation indicators is that the higher the PSNR and SSIM, the higher the quality of the generated image. The optimal water and underwater image style transfer model is finally obtained through screening and saved.

[0060] (5) Establishing underwater optical imaging attenuation model

[0061] like Figure 3 As shown, this is a schematic diagram of the underwater optical imaging attenuation theory. It can be seen that light of different wavelengths has different attenuation rates in water. As the distance light propagates in water increases, red light disappears first, followed by orange light, yellow light, purple light, and green light. Blue light has the shortest wavelength and is transmitted the farthest in water, so underwater images exhibit a blue-green color cast.

[0062] The underwater optical imaging model is established according to the underwater optical imaging formula:

[0063] I c (x) = J c (x)T c (x)+B c (x)(1-T c (x)),c∈{R,G,B}

[0064] Among them, x is the pixel coordinate, I c (x) is the c channel of the underwater image captured by the camera, R, G, B represent the RGB color channels of the image respectively, and J c (x) is the undegraded image, B c (x) is the global background light of color c channel, T c (x) is the medium transmittance of channel c, expressed as:

[0065] T c (x) = exp(-β c d(x))c∈{R,G,B}

[0066] Among them, β c is the attenuation coefficient of color channel c, and d(x) represents the distance from the optical sensor to the object.

[0067] An underwater optical attenuation operator including uneven illumination, color cast and blur is introduced into the underwater optical imaging model.

[0068] Taking into account the fill light and camera carrier movement in the real underwater shooting process, the attenuation operators Mask(·) and Blurring(·) are introduced into the underwater optical imaging model. The Mask(·) operator is used to simulate the uneven image brightness degradation caused by the fill light of a single light source underwater. The illumination mask is used to simulate the attenuation effect of the light source in the water, so that the image looks closer to the real underwater environment and obtains a non-uniform illumination image. The attenuation effect of the light source is simulated by creating a brightness map that gradually decreases from the center to the outside. The illumination mask uses a Gaussian distribution to calculate the brightness value. The calculation formula is as follows. First, the distance from each pixel to the center of the light source is calculated.

[0069]

[0070] Among them, (x c ,y c ) is the coordinate of the center of the light source, and (x,y) is the coordinate of the current pixel.

[0071] Secondly, the light source intensity is calculated according to the Gaussian distribution:

[0072]

[0073] Among them, σ is the standard deviation of the Gaussian distribution, and intensity is the maximum light source intensity. This formula indicates that the intensity of the light source decays exponentially with increasing distance.

[0074] Finally, the mask value is normalized to the range [0, 1].

[0075] The Blurring (·) operator is used to simulate the motion blur of underwater images caused by the movement of the carrier during the exposure time, using Gaussian blur:

[0076]

[0077] Among them, GaussianBlur represents Gaussian blur, Represents convolution.

[0078] Color cast is caused by the different attenuation rates of light of different wavelengths in water. The images of the three color channels of R, G, and B are attenuated according to the physical laws of underwater optical imaging and fused in different proportions. After integration, the final dam underwater defect image generation formula is:

[0079] I(x)=Blurring{Mask[I R (x)+I G (x)+I B (x)]}

[0080] (6) Generate underwater defect images of dams

[0081] like Figure 4 As shown, the steps of generating underwater defect images of a dam using the above-water and underwater image style transfer model and the underwater optical imaging attenuation model.

[0082] Firstly, the images in the dam water defect image dataset are input into the underwater style transfer model and the underwater optical imaging attenuation model respectively. Then, the images with more realistic underwater style are selected from the dam underwater defect images generated by the two models based on the artificial subjective screening and objective evaluation indicators PSNR and SSIM. The selected images should meet the standard: they contain not only the features of the dam surface defects but also the underwater imaging style. Subsequently, a dam underwater defect image dataset is established, which can be used for the training of deep learning models of dam underwater defects such as intelligent recognition, target detection and semantic segmentation.

[0083] The embodiments described above are merely implementation methods of the present invention and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for generating underwater defect images of dams based on deep learning and underwater optical models, characterized in that: The following steps are involved: S1: Use optical equipment to take photos of the apparent defects of the dam, and establish a dataset of dam surface defect images after processing; S2: Use an underwater robot equipped with an underwater camera to take underwater images of the dam, filter images with underwater style features, and establish an underwater style image dataset; S3: Processing the dam above-water defect image dataset and the underwater style image dataset to establish an unpaired dam above-water and underwater image dataset; S4: building an unsupervised generative adversarial network, using the above-water and underwater image dataset of the dam to train the generative adversarial network, obtaining and saving the above-water and underwater image style transfer model; S5: establishing an underwater optical imaging model, introducing an underwater optical attenuation operator, and obtaining an underwater optical imaging attenuation model; S6: Use the above-water and underwater image style transfer model and the underwater optical imaging attenuation model to perform style transfer on the above-water defect images of the dam, establish an underwater defect image dataset of the dam with a real underwater image style, and use the underwater defect image dataset of the dam to perform various deep learning tasks.

2. The method for generating underwater defect images of dams according to claim 1, characterized in that: The step S1 of establishing a dataset of dam surface defect images includes cropping and screening the captured dam surface defect images so that the defects account for a large proportion of the target in the image and the defect features are obvious.

3. The method for generating underwater defect images of dams according to claim 1, characterized in that: The step S2 described in establishing an underwater style image dataset includes operating an underwater robot to shoot different hydraulic structures at different water depths in the reservoir, intercepting and cropping the acquired videos to obtain underwater images containing problems such as uneven illumination, color cast, and blur degradation.

4. The method for generating underwater defect images of dams according to claim 1, characterized in that: The step S3 of establishing an unpaired dam above-water and underwater image dataset includes placing the dam above-water image dataset and the underwater style image dataset in folders A and B respectively, and placing the A and B folders in the dam above-water and underwater defect image dataset up2un folder.

5. The method for generating underwater defect images of dams according to claim 1, characterized in that: The above-water and underwater image style transfer model is obtained, including building a generative adversarial network, where the adversarial loss is expressed as: Among them, G is the generator, D is the discriminator, X represents the dam water defect dataset, and Y represents the underwater style image dataset. represents the mathematical expectation of function f(x) under distribution X. It represents the mathematical expectation of function f(y) on y under distribution Y; The up2un dataset of dam underwater defect images is divided into training and test sets in proportion. The appropriate number of training epochs and hyperparameter λ are set and optimized to obtain the optimal style transfer model for underwater and above-water images.

6. The method for generating underwater defect images of dams according to claim 1, characterized in that: The method for obtaining the underwater optical imaging attenuation model includes: The underwater optical imaging model is established according to the underwater optical imaging formula: I c (x)=J c (x)T c (x)+B c (x)(1-T c (x)),c∈{R,G,B} Among them, x is the pixel coordinate, I c (x) is the c channel of the underwater image captured by the camera, R, G, B represent the RGB color channels of the image respectively, and J c (x) is the undegraded image, B c (x) is the global background light of color c channel, T c (x) is the medium transmittance of channel c, expressed as: T c (x)=exp(-β c d(x))c∈{R,G,B} Among them, β c is the attenuation coefficient of color c channel, d(x) represents the distance from the optical sensor to the object; Introducing underwater optical attenuation operators including uneven illumination, color cast and blur into underwater optical imaging models; Taking into account the fill light and camera carrier movement in the real underwater shooting process, the attenuation operators Mask(·) and Blurring(·) are introduced into the underwater optical imaging model; the Mask(·) operator is used to simulate the uneven image brightness degradation caused by the fill light of a single light source underwater, and the illumination mask is used to simulate the attenuation effect of the light source in water, so that the image looks closer to the real underwater environment and obtains a non-uniform illumination image; the attenuation effect of the light source is simulated by creating a brightness map that gradually decreases from the center to the outside; the illumination mask uses Gaussian distribution to calculate the brightness value, and the calculation formula is as follows; first, the distance from each pixel to the center of the light source is calculated; Among them, (x c ,y c ) is the coordinate of the center of the light source, (x, y) is the coordinate of the current pixel; Secondly, the light source intensity is calculated according to the Gaussian distribution: Among them, σ is the standard deviation of the Gaussian distribution, and intensity is the maximum light source intensity. This formula indicates that the light source intensity decays exponentially with increasing distance; Finally, normalize the mask value to the range [0,1]; The Blurring (·) operator is used to simulate the motion blur of underwater images caused by the movement of the carrier during the exposure time, using Gaussian blur: Among them, GaussianBlur represents Gaussian blur, represents convolution; Color cast is caused by the different attenuation rates of light of different wavelengths in water. The images of the three color channels of R, G, and B are attenuated according to the physical laws of underwater optical imaging and fused in different proportions. After integration, the final dam underwater defect image generation formula is: I(x)=Blurring{Mask[I R (x)+I G (x)+I B (x)]}。 7. The method for generating underwater defect images of dams according to claim 1, characterized in that: The method for establishing a dam underwater defect image dataset with a real underwater image style by using an above-water and underwater image style transfer model and an underwater optical imaging attenuation model comprises inputting the dam above-water defect image dataset into the above-water and underwater image style transfer model and the underwater optical imaging attenuation model, and selecting images with a real underwater style based on artificial subjective screening and objective evaluation indicators PSNR and SSIM. The selected images not only contain features of the dam's apparent defects, but also conform to the underwater imaging style, and can be used for intelligent recognition, target detection and semantic segmentation of dam underwater defect deep learning model training.

Images