A method for enhancing concealed pipeline image data based on generative adversarial network

Abstract

A method for enhancing concealed pipeline image data based on generative adversarial networks includes two key parts: reconstructing the generative adversarial network structure for generating concealed pipeline radar image data and constructing a fuzzy min-max neural network for controlling the quality of the generated images. The present invention reconstructs the structure of the generative adversarial network to form an extended generative adversarial network. The real concealed pipeline image dataset obtained by ground penetrating radar is used as the input of the extended generative adversarial network, and the generated images are screened by the fuzzy min-max neural network to obtain effective enhanced image data. The enhanced images are mixed with the real concealed pipeline image dataset to train, test, and validate the public classifier. Compared with traditional methods, the image data generated by this method has a higher verification accuracy, and the data enhancement effect is better than that of traditional generative adversarial network models.

Description

Technical Field

[0001] The present invention relates to the field of machine learning data augmentation, and particularly to a method for augmenting hidden pipeline image data based on generative adversarial networks. Background Art

[0002] Since hidden pipelines are usually buried underground or deep within building structures, it is relatively difficult to obtain a large number of hidden pipeline image datasets that can be used for machine learning model training. In addition, there may be various interference factors in pipeline images, such as light changes, shadows, occlusions, etc., making it difficult to ensure the diversity and quality of the dataset. Traditional data augmentation methods often perform poorly in dealing with these challenges because they are difficult to simulate the complex scenarios of pipeline images in the real world.

[0003] In this context, the introduction of Generative Adversarial Network (GAN) provides an innovative solution for augmenting hidden pipeline image data. GAN is a deep learning model composed of a generator and a discriminator, which can generate realistic new samples through adversarial training. By training the generator, more diverse and realistic hidden pipeline images can be generated, thus making up for the deficiencies of the original dataset. The advantage of GAN is that it can learn and generate complex pipeline scenarios, including various interferences and changes, thereby improving the effect of data augmentation and providing more generalized model training data for tasks such as hidden pipeline detection. Therefore, using GAN to augment hidden pipeline images can not only solve the problems of data scarcity and complexity, but also improve the performance and robustness of the model in actual application scenarios.

[0004] The fuzzy neural network combines the advantages of neural networks and fuzzy logic, and has great superiority in dealing with problems such as fuzziness and non-linearity, which has attracted more and more experts and scholars to invest in this field and achieved fruitful results. The Fuzzy Min-Max Neural Network (FMM) is an improved fuzzy neural network model that shows strong performance in classification tasks. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above technical drawbacks and deficiencies, and provide a method for augmenting hidden pipeline image data based on generative adversarial networks.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A method for augmenting hidden pipeline image data based on generative adversarial networks, comprising the following steps:

[0008] Step 1: Reconstruct the generative adversarial network model, construct an extended generative adversarial network model (Extend Generative Adversarial Network, ExGAN) suitable for generating radar images of concealed pipelines, input the real concealed pipeline image data, and generate forged data of concealed pipeline radar images;

[0009] Step 2: Construct a fuzzy min-max neural network as a classifier, take the output of ExGAN after feature extraction as the input of the fuzzy min-max neural network, and complete the construction of the entire data augmentation model based on the fuzzy min-max neural network and the extended generative adversarial network (Fuzzy minimum maximum and extended generative adversarial data augmentation, FAExG-DA). Take the real concealed pipeline radar images as the training data set and input them into the fuzzy min-max neural network to train the model;

[0010] Step 3: Input the forged data of the concealed pipeline radar images into the fuzzy min-max neural network classifier for screening;

[0011] Step 4: Mix the screened image data set with the real concealed pipeline images for comparative testing, and verify and feedback the performance and problems of the data augmentation model.

[0012] Step 1: Reconstruct the generative adversarial network model, construct an extended generative adversarial network model suitable for generating radar images of concealed pipelines, input the real concealed pipeline image data, and generate forged data of concealed pipeline radar images. The specific steps are as follows:

[0013] Step 1.1: Use a deep convolutional neural network to reconstruct the generator in the generative adversarial network model to obtain an extended generative adversarial network model. The network model includes a generative network and a discriminative network. The generative network includes 5 convolutional layers, 5 batch normalization layers, 4 upsampling layers, 4 activation layers, and 1 fully connected layer. The discriminative network includes 4 convolutional layers and 1 fully connected layer. The generator G converts random noise into a virtual image, and the discriminator D discriminates the virtual image according to the real image data, and optimizes the network parameters of the generator G and the discriminator D according to the discrimination result;

[0014] Step 1.2: Input the real concealed pipeline image data into ExGAN, adjust the model parameters, and after multiple rounds of training, obtain the output forged image data after the model converges.

[0015] Step 2: Construct a fuzzy min-max neural network as a classifier. After feature extraction of the output of ExGAN, use it as the input of the fuzzy min-max neural network to complete the construction of the entire data augmentation model based on the fuzzy min-max neural network and the extended generative adversarial network. Use real concealed pipeline radar images as the training dataset and input them into the fuzzy min-max neural network to train the model. The specific steps are as follows:

[0016] Step 2.1: Select the convolutional layers of the AlexNet convolutional neural network to extract features from the input image data, including 5 convolutional layers, 4 batch normalization layers, 5 activation layers, 3 max pooling layers, and 1 flattening layer, which are used to convert the image data into multi-dimensional statistical feature data to form an original dataset {X}, where X = (x h1 , x h2 ,..., x hn ) represents the extracted image feature pattern, and n is the data dimension;

[0017] Step 2.2: Perform dimensionality reduction and normalization on the extracted multi-dimensional statistical data, remove the dimensions containing 0 values, and obtain the training dataset {X'}, where X h ' = (x h1 , x h2 ,..., x hn' ), and n' is the data dimension after dimensionality reduction;

[0018] Step 2.3: Create hyperboxes centered on each input pattern. Calculate hb min and hb max

[0019]

[0020]

[0021] where l represents the radius of the hyperbox. In this method, take l = 0.05 to obtain the hyperbox set {B}, where B i = {V i , W i} (i = 1, 2... N), V i is the minimum point of the hyperbox, and W i is the maximum point of the hyperbox;

[0022] Step 2.4: Combine the reconstructed ExGAN and the constructed fuzzy min-max neural network to form a data augmentation model based on the fuzzy min-max neural network and the extended generative adversarial network.

[0023] Step 3: Input the forged data of the concealed pipeline radar image into the fuzzy min-max neural network classifier for screening. The specific steps are as follows:

[0024] Step 3.1: The generated forged images are also subjected to feature extraction and dimensionality reduction through the AlexNet convolutional neural network to obtain a feature data set {T}, where T i =(t1,t2,...,t n' );

[0025] Step 3.2: For a piece of data in the feature data set, traverse all the hyperboxes in the hyperbox set. According to the formula

[0026] t i ≥V ji ,t i ≤W ji (j = 1,2...N)

[0027] Judge whether the data is inside the hyperbox. If it is inside the hyperbox, it is marked as a valid enhanced image; otherwise, it is an invalid enhanced image. Denote the set of valid enhanced images as T V , and the set of invalid enhanced images as T inV ;

[0028] Step 3.3: Repeat the above steps until each piece of data in the feature data set has been processed.

[0029] Step 4: Mix the screened image data set with the real concealed pipeline images and conduct a comparative test to verify and feedback the performance and problems of the data enhancement model. The specific steps are as follows:

[0030] Step 4.1: Select a certain number of images from the valid generated image data set T V , mix them with the real concealed pipeline data set, and use them as the data set for training and testing the neural network. The image labels in the real concealed pipeline data set and the set T V are 1, and random noise images are added as images with label 0;

[0031] Step 4.2: Use it as the training set and test set for testing the neural network VGG16. Use the model after 30 rounds of training to predict the validation set and test its accuracy to verify the effect of data enhancement;

[0032] Step 4.3: Input the original real concealed pipeline radar image data set into the VGG16 convolutional neural network model for training and validation, and compare its accuracy with the accuracy of FAExG-DA;

[0033] Step 4.4: Input the dataset of the concealed pipeline radar images generated by the original GAN model into the VGG16 convolutional neural network model for training and validation, and compare its accuracy with that of FAExG-DA;

[0034] Step 4.5: Feedback the performance of FAExG-DA based on the comparison results to determine whether the model needs to be modified.

[0035] Advantages and beneficial effects of the present invention

[0036] The present invention reconstructs the structure of the traditional generative adversarial network and introduces a fuzzy neural network to control the quality of the generated forged data, aiming to improve the data augmentation effect of the concealed pipeline radar images. By inputting the generated concealed pipeline forged radar images into the VGG16 neural network model for training and validating in the validation set, the accuracy of the recognition model is improved, achieving the data augmentation effect, indicating that the extended data augmentation method for concealed pipeline images based on generative adversarial has better performance. Brief description of the drawings

[0037] Figure 1 is the algorithm flowchart of a data augmentation method for concealed pipeline images based on generative adversarial of the present invention;

[0038] Figure 2 is the architecture diagram of the proposed data augmentation model (FAExG-DA) based on fuzzy min-max neural network and extended generative adversarial network. Detailed implementation manners

[0039] The present invention will be described in detail below with reference to the drawings and examples.

[0040] A data augmentation method for concealed pipeline images based on generative adversarial augments the dataset of real concealed pipeline images, and inputs the original data and the generated data into the VGG16 neural network for testing and validation. The specific implementation plan includes the following contents and steps:

[0041] Step 1: Reconstruct the generative adversarial network model, construct an extended generative adversarial network model suitable for generating radar images of concealed pipelines, input the real concealed pipeline image data, and generate forged data of concealed pipeline radar images, as Figure 2 shown. The specific steps are as follows:

[0042] Step 1.1: Reconstruct the generator in the GAN model using a deep convolutional neural network to obtain ExGAN. Specifically, in ExGAN, the generator constructs a 1*100 random noise matrix, which is transformed into a 256*16*16 matrix through a fully connected layer as the input to the generator in ExGAN. Next, (1) after batch normalization and 2-fold upsampling operations on the matrix, it is input into a convolutional layer with a convolutional kernel size of 3*3 and a stride of 1 for convolution operation, and the output size is 256*32*32; (2) after batch normalization, LeakyReLU activation, and 2-fold upsampling operations on the matrix output in the previous step, it is input into a convolutional layer with a convolutional kernel size of 3*3 and a stride of 1 for convolution operation, and the output size is 128*64*64; (3) after batch normalization, LeakyReLU activation, and 2-fold upsampling operations on the matrix output in the previous step, it is input into a convolutional layer with a convolutional kernel size of 3*3 and a stride of 1 for convolution operation, and the output size is 128*128*128; (4) after batch normalization, LeakyReLU activation, and 2-fold upsampling operations on the matrix output in the previous step, it is input into a convolutional layer with a convolutional kernel size of 3*3 and a stride of 1 for convolution operation, and the output size is 64*256*256; (5) after batch normalization and LeakyReLU activation operations on the matrix output in the previous step, it is input into a convolutional layer with a convolutional kernel size of 3*3 and a stride of 1 for convolution operation, and the output size is 3*256*256. Finally, after passing through a Tanh activation layer, the generated image is obtained. This image has 3 channels and a size of 256*256. For the discriminator of ExGAN, it receives an image with a size of 3*256*256 as the input. Next, (1) it passes through a convolutional layer with a convolutional kernel of 3*3 and a stride of 2, and batch normalization is performed, and the output size is 16*128*128; (2) the output of the previous step passes through a convolutional layer with a convolutional kernel of 3*3 and a stride of 2, and batch normalization is performed, and the output size is 32*64*64; (3) the output of the previous step passes through a convolutional layer with a convolutional kernel of 3*3 and a stride of 2, and batch normalization is performed, and the output size is 64*32*32; (4) the output of the previous step passes through a convolutional layer with a convolutional kernel of 3*3 and a stride of 2, and batch normalization is performed, and the output size is 128*16*16; (5) the output of the previous step is mapped to a 1-dimensional scalar through a fully connected layer, and finally, through a Sigmoid function, the scalar is converted to 0 or 1 as the discrimination label of the image;

[0043] Step 1.2: Input the real concealed pipeline radar image data into ExGAN, adjust the model parameters, and after multiple rounds of training, obtain the output enhanced image data after the model converges. Specifically, the real concealed pipeline images are detected on-site by a ground penetrating radar and uniformly cropped to a size of 256*256. These images are used as a dataset to be input into ExGAN for training. Set the learning rate to 0.0005, use Adam as the optimizer, and output an image every 10 rounds to view the generation effect until the model converges.

[0044] Step 2: Construct a fuzzy min-max neural network as a classifier, and use the output of ExGAN as the input of the fuzzy min-max neural network after feature extraction to complete the construction of the entire data enhancement model based on the fuzzy min-max neural network and the extended generative adversarial network. Use the real concealed pipeline radar images as the training dataset and input them into the fuzzy min-max neural network to train the model. The specific steps are as follows:

[0045] Step 2.1: Select the convolutional layer of the AlexNet convolutional neural network to extract features from the input image data. First, compress the size of the image to 3*224*224 to adapt to AlexNet. Next, (1) pass through a convolutional layer with a convolutional kernel of 11*11 and a stride of 4, and perform ReLU activation layer and batch normalization operations, as well as a max-pooling layer with a convolutional kernel of 3*3 and a stride of 2, and the output size is 96*27*27; (2) pass through a convolutional layer with a convolutional kernel of 5*5 and a stride of 1, and perform ReLU activation layer and batch normalization operations, as well as a max-pooling layer with a convolutional kernel of 3*3 and a stride of 2, and the output size is 256*13*13; (3) pass through a convolutional layer with a convolutional kernel of 3*3 and a stride of 4, and perform ReLU activation layer and batch normalization operations, and the output size is 384*13*13; (4) pass through a convolutional layer with a convolutional kernel of 3*3 and a stride of 1, and perform ReLU activation layer, and the output size is 192*13*13; (5) pass through a convolutional layer with a convolutional kernel of 3*3 and a stride of 1, and perform ReLU activation layer and batch normalization operations, as well as a max-pooling layer with a convolutional kernel of 3*3 and a stride of 2, and the output size is 256*6*6; (6) perform a flatten operation on the output of the previous step to obtain a 4096-dimensional vector, which is the feature of the image. Repeat this step for all images in the dataset to obtain the training dataset {X};

[0046] Step 2.2: Perform dimensionality reduction and normalization processing on the extracted multi-dimensional statistical data, remove the dimensions containing 0 values, and obtain the training dataset {X'}, where X h '=(x h1 ,x h2 ,…,x hn'), where n' is the data dimension after dimensionality reduction. Specifically, for each dimension of the image, if there is a value of 0, then that dimension is removed. In the normalization step, according to the formula

[0047]

[0048] normalize the data, restricting the data of each dimension between 0 and 1;

[0049] Step 2.3: Create a hyperbox centered on each input pattern. Calculate hb according to the following formula min and hb max

[0050]

[0051]

[0052] where l represents the radius of the hyperbox. In this method, take l = 0.05. The obtained hyperbox set is {B}, where B i ={V i ,W i}(i = 1, 2…N), V i is the minimum point of the hyperbox, and W i is the maximum point of the hyperbox;

[0053] Step 2.4: Combine the reconstructed ExGAN and the constructed fuzzy min-max neural network to form a data augmentation model based on the fuzzy min-max neural network and the extended generative adversarial network.

[0054] Step 3: Input the concealed pipeline radar image forgery data into the fuzzy min-max neural network classifier for screening. The specific steps are as follows:

[0055] Step 3.1: The enhanced image is also subjected to feature extraction and dimensionality reduction through the AlexNet convolutional neural network to obtain the feature dataset {T};

[0056] Step 3.2: For a piece of data in the feature dataset, traverse all the hyperboxes in the hyperbox set. According to the formula

[0057] t i ≥V ji ,t i ≤W ji (j = 1, 2…N)

[0058] Judge whether the data is inside the hyperbox. If it is inside the hyperbox, mark it as a valid enhanced image; otherwise, it is an invalid enhanced image. Denote the set of valid enhanced images as T V , and the set of invalid enhanced images as T inV ;

[0059] Step 3.3: Repeat the above steps until each piece of data in the feature dataset has been processed;

[0060] Step 4: Mix the filtered image dataset with real concealed pipeline images for comparative testing, verify and feedback on the performance and problems of the data augmentation model. The specific steps are as follows:

[0061] Step 4.1: Select a certain amount of images from T V and mix them with the real concealed pipeline dataset to form a dataset for training and testing the neural network. The image labels in the real concealed pipeline dataset and set T V are 1, and random noise images are added as images with label 0. The quantity ratio of the two types of images is 2:1;

[0062] Step 4.2: Use it as the training set and test set for testing the neural network VGG16. Use the model after 30 rounds of training to predict the validation set and test its accuracy to verify the effect of data augmentation. Specifically, the neural network used is the standard VGG16, the number of training rounds is 20, the learning rate is 0.0005, and Adam is used as the neural network optimizer;

[0063] Step 4.3: Input the original real concealed pipeline radar image dataset into the VGG16 convolutional neural network model for training and validation, and compare its accuracy with that of FAExG-DA;

[0064] Step 4.4: Use the traditional GAN model to perform image augmentation in the same way, manually screen out effective and ineffective augmented images, and also input them into the VGG16 convolutional neural network model for training and validation, and compare its accuracy with that of ExGAN;

[0065] Step 4.5: Construct a comparative experiment to compare the data augmentation effect of the data augmentation method provided by the present invention (i.e., FAExGAN-DA) with the data augmentation method of the original GAN model and the dataset without data augmentation. In this experiment, the generated forged concealed pipeline radar image data is mixed with the real concealed pipeline radar image data as the training set and test set of the VGG16 neural network. After 30 rounds of iteration, the model is saved, and the validation set composed entirely of real concealed pipeline data is used to verify the model. Compare the accuracies obtained by the two methods. The experimental results are shown in the table. The experimental results show that the accuracy of the concealed pipeline images generated by the present invention on the validation set is better than that of the original GAN method after training in the VGG16 neural network, and it has better model classification performance.

[0066]

[0067] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various modifications and changes can be made to the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims

1. A hidden pipeline image data enhancement method based on generative adversarial methods comprises the following steps: Step 1: Reconstruct the generative adversarial network model and construct an extended generative adversarial network model suitable for generating radar images of concealed pipelines. Input the real concealed pipeline image data to generate forged radar image data of concealed pipelines. Step 2: Construct a data enhancement model based on the fuzzy minimum-maximum neural network and the extended generative adversarial network. By selecting the convolutional layer of the AlexNet convolutional neural network, the real hidden pipeline radar image is converted into multi-dimensional statistical feature data to achieve feature extraction of the input image data. The extracted multi-dimensional statistical data is subjected to dimensionality reduction and normalization to form an original data set as a training data set. Each input pattern of the original data set is used to create a hyperbox with itself as the center, thereby constructing a fuzzy minimum-maximum neural network as a classifier. The output of the extended generative adversarial network model constructed in step 1 is used as the input of the fuzzy minimum-maximum neural network after feature extraction, that is, a convolutional layer of the AlexNet convolutional neural network is added to the output part of the extended generative adversarial network, and a fuzzy minimum-maximum neural network is combined after the convolutional layer, thereby completing the construction of the entire data enhancement model FAExG-DA based on the fuzzy minimum-maximum neural network and the extended generative adversarial network. Step 3: Input the forged data of the concealed pipeline radar image into the fuzzy minimum maximum neural network classifier for screening; Step 4: Mix the filtered image dataset with the real hidden pipeline images and conduct comparative tests to verify and provide feedback on the performance and problems of the data augmentation model.

2. The method for enhancing hidden pipeline image data based on generative adversarial methods according to claim 1, characterized in that: In step 1, the traditional generative adversarial network model is improved to construct an extended generative adversarial model suitable for hidden pipeline image data enhancement. The real hidden pipeline image data is input to obtain the enhanced hidden pipeline image data. The specific steps are as follows: Step 1.1: Use a deep convolutional neural network to reconstruct the generator in the generative adversarial network model to obtain an extended generative adversarial network model. The network model includes a generator network and a discriminator network. The generator network includes 5 convolutional layers, 5 batch normalization layers, 4 upsampling layers, 4 activation layers, and 1 fully connected layer. The discriminator network includes 4 convolutional layers and 1 fully connected layer. The generator G converts random noise into a virtual image, and the discriminator D discriminates the virtual image based on the real image data. The network parameters of the generator G and the discriminator D are optimized based on the discrimination results. In step 1.2, the real hidden pipeline image data is input into the extended generative adversarial network, the model parameters are adjusted, and after multiple rounds of training, the output forged image data is obtained after the model converges.

3. The method for enhancing hidden pipeline image data based on generative adversarial methods according to claim 1, characterized in that: Step 2: Construct a fuzzy min-max neural network as a classifier. Use the output of the extended generative adversarial network as input to the fuzzy min-max neural network after feature extraction, completing the construction of the data enhancement model based on the fuzzy min-max neural network and the extended generative adversarial network. Use real radar images of concealed pipelines as training data sets and input them into the fuzzy min-max neural network to train the model. The specific steps are as follows: Step 2.1: Select the convolutional layer of the AlexNet convolutional neural network to extract features from the input image data, including 5 convolutional layers, 4 batch normalization layers, 5 activation layers, 3 maximum pooling layers, and 1 flattening layer, which are used to convert image data into multidimensional statistical feature data to form an original data set {X}, where X = (x h1 ,x h2 ,...,x hn ) represents the extracted image feature pattern, and n is the data dimension; Step 2.2: Perform dimensionality reduction and normalization on the extracted statistical data, remove the dimension containing 0 values, and obtain the training data set {X'}, where X h '=(x h1 ,x h2 ,...,x hn' ), n' is the dimension of the data after dimensionality reduction; Step 2.3: For each input pattern, a hyperbox is created with itself as the center, and hb is calculated according to the following formula: min and hb max Where l represents the radius of the hyperbox. In this method, l = 0.05, and the resulting hyperbox set is {B}, where B i ={V i ,W i }(i=1,2...N), V i is the minimum point of the hyperbox, W i is the maximum point of the hyperbox; In step 2.4, the reconstructed extended generative adversarial network is combined with the constructed fuzzy minimum maximum neural network to form a data enhancement model based on the fuzzy minimum maximum neural network and the extended generative adversarial network.

4. The method for enhancing hidden pipeline image data based on generative adversarial methods according to claim 1, characterized in that: As mentioned in step 3, the forged data of the concealed pipeline radar image is input into the fuzzy minimum maximum neural network classifier for screening. The specific steps are as follows: Step 3.1: The generated fake image is also subjected to feature extraction and dimensionality reduction processing by the AlexNet convolutional neural network to obtain the feature dataset {T}, where T i =(t1,t2,...,t n' ); Step 3.2: For a piece of data in the feature data set, traverse all the hyperboxes in the hyperbox set and calculate the value according to the formula t i ≥V ji ,t i ≤W ji (j=1,2...N) Determine whether the data is inside the hyperbox. If it is, it is marked as a valid enhanced image, otherwise it is an invalid enhanced image. The set of valid enhanced images is T V , the invalid enhanced image set is T inV ; Step 3.3: Repeat the above steps until every data in the feature dataset has been processed.

5. The method for enhancing hidden pipeline image data based on generative adversarial methods according to claim 1, characterized in that: In step 4, the filtered image dataset is mixed with the real hidden pipeline image and compared with each other to verify and provide feedback on the performance and problems of the data augmentation model. The specific steps are as follows: Step 4.1: Generate an image dataset T from the valid V A certain amount of images are selected from the dataset and mixed with the real hidden pipeline dataset as the dataset for training and testing the neural network. V The image label in is 1, and a random noise image is added as the image with label 0; Step 4.2: As the training set and test set for testing the neural network VGG16, use the model after 30 rounds of training to predict the validation set and test its accuracy to verify the effect of data augmentation; In step 4.3, the original real concealed pipeline radar image dataset is also input into the VGG16 convolutional neural network model for training and verification, and its accuracy is compared with that of the FAExG-DA. In step 4.4, the hidden pipeline radar image dataset generated by the original GAN model is also input into the VGG16 convolutional neural network model for training and verification, and its accuracy is compared with that of the FAExG-DA. Step 4.5: Compare the performance of the result-feedback FAExG-DA to determine whether the model needs to be modified.

Images