Ground penetrating radar image data augmentation methods, devices, electronic equipment and storage media

Abstract

The present invention discloses a data augmentation method, and specifically relates to a ground penetrating radar image data augmentation method, device, electronic device and storage medium. The method includes: obtaining multiple categories of GPR B-scan images corresponding to multiple underground targets one by one; constructing a multi-category auxiliary generative adversarial network; using multiple categories of GPR B-scan images as training data sets to iteratively train the multi-category auxiliary generative adversarial network to obtain a trained multi-category auxiliary generative adversarial network; inputting noise vectors and category labels into the trained multi-category auxiliary generative adversarial network to generate target GPR B-scan images corresponding to the category labels. Through the above method, high-quality GPR B-scan images of corresponding categories can be generated while only requiring a small number of training data sets.

Description

Technical Field

[0001] The present invention relates to a data augmentation method, and in particular to a ground penetrating radar image data augmentation method, device, electronic equipment and storage medium. Background Art

[0002] Ground Penetrating Radar (GPR) is a non-destructive detection technology that uses high-frequency electromagnetic waves to detect underground objects. It can obtain information about underground structures and targets non-destructively. When detecting along the surface line, GPR determines the spatial position, shape, size, dielectric properties and other parameters of underground media and targets by emitting electromagnetic wave pulses and receiving their reflected signals.

[0003] At present, deep learning has shown great potential in the field of GPR data processing. However, a large amount of GPR data is required to train a reliable deep learning model. Due to the complexity of field measurement conditions and the high cost of data acquisition, it is often very difficult to obtain sufficient field data. Most GPR target detection networks use only thousands of training samples, which are usually collected under the same experimental settings. The similarity between the data is high and lacks diversity. In addition, there may be imbalance problems in data of different categories, which brings difficulties to network feature learning. Simulation is an effective means to increase GPR data, but it is difficult to reproduce the complex and changeable real scenes and medium distribution during simulation modeling, and it cannot fully simulate the influence of various factors, resulting in the simulation data not being realistic enough. On the other hand, simulation calculations also require a certain amount of computing time to obtain results. If the accuracy of the simulation results is to be improved, it is necessary to divide the spatial sampling interval and the time sampling interval into finer ones, and the consumption of computing resources will increase sharply. Therefore, how to achieve multi-category GPR B-scan image data augmentation is still an urgent problem to be solved.

[0004] Data augmentation methods in the prior art include traditional methods and methods based on deep learning. Traditional methods include operations such as translation, flipping, and scaling, as well as adding various noises to images. Traditional methods only process images in existing data sets and do not generate new features, resulting in data that is too similar and prone to overfitting problems during network training. Currently, data augmentation methods based on deep learning mainly use generative adversarial networks to learn the data distribution of training samples and generate samples with new features. However, existing GAN methods still have some shortcomings when generating GPR data, such as the generated data is limited to a single category, and it is prone to mode collapse, resulting in a lack of diversity in generated samples. Therefore, the GPR B-scan images generated by existing data augmentation methods have the problems of single category and high similarity, resulting in poor quality of the augmented training data set. Summary of the invention

[0005] The technical problem to be solved by the present invention is that the GPRB-scan images generated by the existing data augmentation methods have the problems of single type and high similarity, resulting in poor quality of the augmented training data set. In order to solve the above problems, the present invention provides a ground penetrating radar image data augmentation method, device, electronic device and storage medium.

[0006] The content of the present invention includes:

[0007] In a first aspect, an embodiment of the present invention provides a method for augmenting ground penetrating radar image data, comprising:

[0008] Acquire GPR B-scan images of multiple categories corresponding to multiple underground targets;

[0009] Constructing a multi-category auxiliary generative adversarial network, the multi-category auxiliary generative adversarial network includes a generator and a discriminator, the generator includes a residual transposed convolution block, the discriminator includes a residual convolution block, the residual transposed convolution block and the residual convolution block are used to deepen the network structure to learn deep-level features of various categories of images, the generator and the discriminator both include a CBAM attention mechanism layer, a spectral normalization layer and an instance normalization layer, the CBAM attention mechanism layer includes a channel attention mechanism and a spatial attention mechanism, which are used to adaptively extract image features, and the spectral normalization layer and the instance normalization layer are used to stabilize the output of the network;

[0010] Iteratively training the multi-category auxiliary generative adversarial network using the multiple categories of GPR B-scan images as training data sets to obtain a trained multi-category auxiliary generative adversarial network;

[0011] The noise vector and the category label are input into the trained multi-category auxiliary generative adversarial network to generate a target GPR B-scan image corresponding to the category label.

[0012] Optionally, the generator comprises:

[0013] Two parallel transposed convolution blocks, one of which receives the noise vector as input and the other receives the category label as input;

[0014] A first processing block, the first processing block is used to process the outputs of the two parallel transposed convolution blocks;

[0015] The CBAM attention mechanism layer, the CBAM attention mechanism layer is used to adaptively extract features from the output of the first processing block;

[0016] A first transposed convolutional layer, wherein the first transposed convolutional layer is used to generate a target GPR B-scan image based on the output of the CBAM attention mechanism layer.

[0017] Optionally, the first processing block includes 5 first processing sub-blocks connected in sequence, and the first processing sub-block includes the residual transposed convolution block and the upsampling layer connected in sequence.

[0018] Optionally, the residual transposed convolution block includes sequentially connected:

[0019] A first residual part, wherein the first residual part includes two connected transposed convolutional layer groups, wherein the transposed convolutional layer group includes a second transposed convolutional layer, a first spectral normalization layer, and a first instance normalization layer connected in sequence;

[0020] A first direct mapping part, wherein the first direct mapping part includes a first convolutional layer, and the first convolutional layer is used to adjust the number of channels of the input feature map;

[0021] A first processing layer group, wherein the first processing layer group is used to add the output of the first residual part and the output of the first direct mapping part and output them through an activation function.

[0022] Optionally, the discriminator includes:

[0023] a second processing block, the second processing block being used to process an input image;

[0024] The CBAM attention mechanism layer, the CBAM attention mechanism layer is used to adaptively extract features from the output of the second processing block;

[0025] A third processing block, the third processing block is used to process the output of the CBAM attention mechanism layer;

[0026] an output layer, the output layer being used to determine the probability that the input image is a real image based on the output of the third processing block;

[0027] An auxiliary classifier, the auxiliary classifier is used to determine the category probability of the input image based on the output of the third processing block.

[0028] Optionally, the third processing block includes 5 second processing sub-blocks connected in sequence, and the second processing sub-block includes the residual convolution block and the global average pooling layer connected in sequence.

[0029] Optionally, the residual convolution block includes:

[0030] A second residual part, wherein the second residual part includes two connected convolutional layer groups, and the convolutional layer group includes a second convolutional layer, a second spectral normalization layer, and a second instance normalization layer connected in sequence;

[0031] A second direct mapping part, wherein the second direct mapping part includes a third convolution layer, and the third convolution layer is used to adjust the number of channels of the input feature map;

[0032] The second processing layer group is used to add the output of the second residual part and the output of the second direct mapping part and then output them through an activation function and a discard layer.

[0033] In a second aspect, an embodiment of the present invention provides a ground penetrating radar image data augmentation device, comprising:

[0034] An acquisition module, used for acquiring GPR B-scan images of multiple categories corresponding to multiple underground targets;

[0035] A construction module is used to construct a multi-category auxiliary generative adversarial network, wherein the multi-category auxiliary generative adversarial network includes a generator and a discriminator; the generator includes a residual transposed convolution block, the discriminator includes a residual convolution block, the residual transposed convolution block and the residual convolution block are used to deepen the network structure to learn deep-level features of images of various categories, the generator and the discriminator both include a CBAM attention mechanism layer, a spectral normalization layer and an instance normalization layer, the CBAM attention mechanism layer includes a channel attention mechanism and a spatial attention mechanism, which are used to adaptively extract image features, and the spectral normalization layer and the instance normalization layer are used to stabilize the output of the network;

[0036] An iterative training module, used for iteratively training the multi-category auxiliary generative adversarial network using the multiple categories of GPR B-scan images as training data sets to obtain a trained multi-category auxiliary generative adversarial network;

[0037] A generation module is used to input the noise vector and the category label into the trained multi-category auxiliary generative adversarial network to generate a target GPR B-scan image corresponding to the category label.

[0038] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the ground penetrating radar image data augmentation method as described in the first aspect.

[0039] In a fourth aspect, an embodiment of the present invention provides a readable storage medium for storing a program, wherein when the program is executed by a processor, the steps in the method for augmenting ground penetrating radar image data as described in the first aspect are implemented.

[0040] In an embodiment of the present invention, multiple categories of GPR B-scan images corresponding to multiple underground targets are obtained; a multi-category auxiliary generative adversarial network is constructed, and the multi-category auxiliary generative adversarial network includes a generator and a discriminator; the multi-category auxiliary generative adversarial network is iteratively trained using the multiple categories of GPR B-scan images as training data sets to obtain a trained multi-category auxiliary generative adversarial network; a noise vector and a category label are input into the trained multi-category auxiliary generative adversarial network to generate a target GPR B-scan image corresponding to the category label. The beneficial effect of the present invention is that high-quality GPR B-scan images of corresponding categories can be generated with only a small number of training data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Attached Figure 1 A flow chart of a method for augmenting ground penetrating radar image data provided by an embodiment of the present invention;

[0042] Attached Figure 2 A network structure diagram of a generator provided by an embodiment of the present invention;

[0043] Attached Figure 3 A network structure diagram of a residual transposed convolution block provided in an embodiment of the present invention;

[0044] Attached Figure 4 A network structure diagram of a discriminator provided in an embodiment of the present invention;

[0045] Attached Figure 5 A network structure diagram of a residual convolution block provided in an embodiment of the present invention;

[0046] Attached Figure 6 A network structure diagram of the CBAM attention mechanism provided by an embodiment of the present invention;

[0047] Attached Figure 7 A processing flow chart of a multi-category auxiliary generative adversarial network provided by an embodiment of the present invention;

[0048] Attached Figure 8 GPR B-scan image training set data of three target categories in the embodiment provided in the embodiment of the present invention;

[0049] Attached Fig. 9 GPR B-scan images of three target categories generated in the embodiments provided in the embodiments of the present invention;

[0050] Attached Fig.10 A schematic diagram of a ground penetrating radar image data augmentation device provided by an embodiment of the present invention;

[0051] Attached Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In the embodiments of the present application, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In the embodiments of the present application, the term "multiple" refers to two or more, and other quantifiers are similar. The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] See also Figure 1 , Figure 1 The present invention provides a method for augmenting ground penetrating radar image data, which includes the following steps:

[0055] Step 101, obtaining GPR B-scan images of multiple categories corresponding to multiple underground targets.

[0056] Step 102, construct a multi-category auxiliary generative adversarial network, the multi-category auxiliary generative adversarial network includes a generator and a discriminator, the generator includes a residual transposed convolution block, the discriminator includes a residual convolution block, the residual transposed convolution block and the residual convolution block are used to deepen the network structure to learn deep-level features of images of various categories, the generator and the discriminator both include a CBAM attention mechanism layer, a spectral normalization layer and an instance normalization layer, the CBAM attention mechanism layer includes a channel attention mechanism and a spatial attention mechanism, which are used to adaptively extract image features, and the spectral normalization layer and the instance normalization layer are used to stabilize the output of the network.

[0057] Step 103: using the GPR B-scan images of the multiple categories as training data sets to iteratively train the multi-category auxiliary generative adversarial network to obtain a trained multi-category auxiliary generative adversarial network.

[0058] Step 104: input the noise vector and the category label into the trained multi-category auxiliary generative adversarial network to generate a target GPR B-scan image corresponding to the category label.

[0059] Specifically, step 101 includes: collecting multiple categories of GPR echo data through simulation or actual measurement to obtain multiple categories of GPR B-scan images. GPR scans the underground area where underground tubular targets of different materials are buried, and the scanning survey line is perpendicular to the axial direction of the underground tubular target. The GPR B-scan image obtained by scanning the underground area where underground tubular targets of the same material are buried is an image of the same category. By scanning the underground area where different underground tubular targets of different materials are buried, GPR B-scan images of different categories can be obtained. Several scanning points are set on the survey line, and the GPR emits electromagnetic waves at each scanning point and receives its reflected echo to form a GPR A-scan echo. The A-scan echoes of all scanning points are merged in sequence to obtain multiple categories of GPR B-scan images of different underground target areas.

[0060] In the specific implementation, in order to meet the needs of network training, it is necessary to preprocess the GPR B-scan images of multiple categories, unify the sizes of the GPR B-scan images of multiple categories, and set the number of channels of all images to 1, that is, set them as grayscale images. The specific processing method is not described here. The processed GPR B-scan images of multiple categories constitute the training data sets of the corresponding categories.

[0061] The specific structure of the multi-category auxiliary generative adversarial network is introduced below. As a generative adversarial network (GAN), the multi-category auxiliary generative adversarial network includes a generator (denoted as G) and a discriminator (denoted as D). The generator G is used to receive noise vectors and label information, and output the generated image of the corresponding category. The discriminator D is used to distinguish between the generated image and the real image, and to discriminate the category of the input image.

[0062] A multi-category auxiliary generative adversarial network is constructed, and residual transposed convolution blocks and residual convolution blocks are designed to deepen the network structure to learn the deep features of images of various categories. The CBAM attention mechanism module is used to adaptively extract image features, and spectral normalization and instance normalization are used to stabilize the network output.

[0063] Specifically, the generator G is composed of a transposed convolution block, a residual transposed convolution block, an upsampling layer, the CBAM attention mechanism layer and a first transposed convolution layer. Optionally, in some embodiments, the generator includes:

[0064] Two parallel transposed convolution blocks, one of which receives the noise vector as input and the other receives the category label as input;

[0065] A first processing block, the first processing block is used to process the outputs of the two parallel transposed convolution blocks;

[0066] The CBAM attention mechanism layer, the CBAM attention mechanism layer is used to adaptively extract features from the output of the first processing block;

[0067] A first transposed convolutional layer, wherein the first transposed convolutional layer is used to generate a target GPR B-scan image based on the output of the CBAM attention mechanism layer.

[0068] As a specific example, see Figure 2 , the first layer of generator G is the two parallel transposed convolution blocks, and the second to eleventh layers of generator G are the first processing blocks. Optionally, the first processing block includes five first processing sub-blocks connected in sequence, and the first processing sub-blocks include residual transposed convolution blocks (G_Res_block) and upsampling layers (Upsampling) connected in sequence. That is, the second, fourth, sixth, eighth, and tenth layers of generator G are residual transposed convolution blocks, and the third, fifth, seventh, ninth, and eleventh layers of generator G are upsampling layers. Among them, the sampling kernel size of all upsampling layers is 2×2. The twelfth layer of generator G is the CBAM attention mechanism layer, which is used to extract features adaptively. The thirteenth layer of generator G is the first transposed convolution layer, and the number of convolution kernels of the first transposed convolution layer is 1, the size is 4×4, and the step size is 2, which is used to output the final generated data, that is, the target GPR B-scan image.

[0069] See also Figure 2 As a specific embodiment, the two parallel transposed convolution blocks set in the first layer of the generator G are used to process the input noise vector (Input z) and the category label (Input c) respectively. The transposed convolution block consists of a transposed convolution layer (ConvT), a spectral normalization layer (SN), an instance normalization layer (IN) and a ReLU activation function. The number of convolution kernels of the transposed convolution layer contained in the transposed convolution block is 1024, the size is 4×4, and the step size is 1.

[0070] Optionally, in some embodiments, the residual transposed convolution block includes:

[0071] A first residual part, wherein the first residual part includes two connected transposed convolutional layer groups, wherein the transposed convolutional layer group includes a second transposed convolutional layer, a first spectral normalization layer, and a first instance normalization layer connected in sequence;

[0072] A first direct mapping part, wherein the first direct mapping part includes a first convolutional layer, and the first convolutional layer is used to adjust the number of channels of the input feature map;

[0073] A first processing layer group, wherein the first processing layer group is used to add the output of the first residual part and the output of the first direct mapping part and output them through an activation function.

[0074] See also Figure 3 , the residual transposed convolution block consists of the first residual part (such as Figure 3 ) and the first direct mapping portion (as shown on the right side of Figure 3 ) is used to deepen the network depth and learn deeper features of various categories of images. The first residual part has two connected transposed convolution layer groups. The number of convolution kernels in the transposed convolution layer group (called the second transposed convolution layer) is half the number of channels of the feature map input to the residual transposed convolution block. The convolution kernel size is 3×3, the step size is 1, and the padding is 1. Each second transposed convolution layer is connected to a spectral normalization layer (called the first spectral normalization layer) and an instance normalization layer (called the first instance normalization layer). The output of the network is more stable through double normalization. The first direct mapping part includes a convolution layer (Conv) (called the first convolution layer). The first convolution layer is a convolution layer with a convolution kernel size of 1×1 and a step size of 1, which is used to adjust the number of channels of the input feature map. Finally, the first processing layer group is used to add the outputs of the first residual part and the first direct mapping part, and process them through the ReLU activation function as the final output of the residual transposed convolution block.

[0075] The discriminator D is composed of a fully connected block, a convolution block, the CBAM attention mechanism layer, a residual convolution block and a downsampling layer. Optionally, in some embodiments, the discriminator includes:

[0076] a second processing block, the second processing block being used to process an input image;

[0077] The CBAM attention mechanism layer, the CBAM attention mechanism layer is used to adaptively extract features from the output of the second processing block;

[0078] A third processing block, the third processing block is used to process the output of the CBAM attention mechanism layer;

[0079] an output layer, the output layer being used to determine the probability that the input image is a real image based on the output of the third processing block;

[0080] An auxiliary classifier, the auxiliary classifier is used to determine the category probability of the input image based on the output of the third processing block.

[0081] Specifically, see Figure 4 , the first and second layers of the discriminator D constitute the second processing block. Specifically, the first layer of the discriminator D is a fully connected block, which is composed of a linear layer (Linear), a spectral normalization layer (SN) and a LeakyReLU activation function, and is used to process category labels. The second layer of the discriminator D is a convolution block, which is composed of a convolution layer (Conv) with 32 convolution kernels, a size of 4×4, a step size of 2, and a padding of 1, a spectral normalization layer (SN) and a LeakyReLU activation function. The third layer of the discriminator D is the CBAM attention mechanism layer. The fourth to thirteenth layers of the discriminator D constitute the third processing block. Optionally, as a specific embodiment, the third processing block includes 5 second processing sub-blocks connected in sequence, and the second processing sub-block includes a residual convolution block (D_Res_block) and a global average pooling layer (AvgPooling) connected in sequence. That is, the 4th, 6th, 8th, 10th, and 12th layers of the discriminator D are residual convolution blocks, and the 5th, 7th, 9th, 11th, and 13th layers are global average pooling layers. The 14th layer of the discriminator D is the output layer, which is a convolution layer with 1 convolution kernel, a size of 4×4, and a step size of 1. The 15th layer of the discriminator D is the auxiliary classifier composed of a linear layer and a softmax layer. The category label after dimension transformation is concatenated with the input image in the first dimension and input to the second layer. The pooling kernel size of all global average pooling layers is 2×2 and the step size is 2. The output of the 13th layer is respectively input to the parallel output layer and the auxiliary classifier to obtain the probability of the discriminator judging whether the input image is a real image and the category probability of the input image.

[0082] Optionally, in some embodiments, the residual convolution block includes:

[0083] A second residual part, wherein the second residual part includes two connected convolutional layer groups, and the convolutional layer group includes a second convolutional layer, a second spectral normalization layer, and a second instance normalization layer connected in sequence;

[0084] A second direct mapping part, wherein the second direct mapping part includes a third convolution layer, and the third convolution layer is used to adjust the number of channels of the input feature map;

[0085] The second processing layer group is used to add the output of the second residual part and the output of the second direct mapping part and then output them through an activation function and a discard layer.

[0086] See also Figure 5 , the residual convolution block consists of the second residual part (such as Figure 5 ) and a second direct mapping portion (as shown on the right side of Figure 5 The second residual part consists of two convolutional layer groups. The number of convolution kernels in the convolutional layer group (called the second convolutional layer) is twice the number of channels of the feature map input to the residual convolution block. The convolution kernel size is 3×3, the step size is 1, and the padding is 1. The second convolutional layer is connected to a spectral normalization layer (called the second spectral normalization layer) and an instance normalization layer (called the second instance normalization layer) to make the convolution output and parameters of the network more stable. The second direct mapping part includes a convolutional layer (called the third convolutional layer). The third convolutional layer is a convolutional layer with a convolution kernel size of 1×1 and a step size of 1, which is used to adjust the number of channels of the input feature map. The second processing layer group is used to add the outputs of the second residual part and the second direct mapping part, pass through the LeakyReLU activation function, and then pass through the Dropout layer as the final output of the residual convolution block to reduce overfitting.

[0087] See also Figure 6 , the CBAM attention mechanism layer combines the channel attention mechanism and the spatial attention mechanism to adaptively adjust the network's attention to channel and spatial information, and extract features with better representation and discrimination. The CBAM channel attention mechanism layer performs global average pooling and global maximum pooling on the input feature layer, and then uses the shared convolution block to process the pooling results. The outputs are added and processed by the Sigmoid function to obtain the weight of each channel of the input feature layer. Each channel weight is multiplied by the input feature layer to obtain the weighted feature layer F1. There are two convolutional layers in the shared convolutional block. The weighted feature layer F1 is input to the spatial attention mechanism. The spatial attention mechanism obtains the weight of each feature point by stacking the maximum and average values ​​of each feature point in the input feature map, and then processes the convolution layer with the adjusted number of channels and the Sigmoid function. The weight of each feature point is then multiplied by F1 to obtain the final weighted feature layer F2. The convolution layer with the adjusted number of channels has 1 convolution kernel, 7×7 convolution kernel size, 1 stride, and 3×3 padding.

[0088] For the convenience of description and distinction, the CBAM attention mechanism layer in the generator G is called the first CBAM attention mechanism layer, and the CBAM attention mechanism layer in the discriminator D is called the second CBAM attention mechanism layer. As an optional implementation, the network structure of the second CBAM attention mechanism layer is the same as that of the first CBAM attention mechanism layer. For details, see Figure 7 Among them, the only difference is that the number of convolution kernels of the two convolution layers contained in the shared convolution block of the second CBAM attention mechanism layer and the first CBAM attention mechanism layer is different. The number of convolution kernels of the two convolution layers in the CBAM attention mechanism layer (i.e., the second CBAM attention mechanism layer) in the discriminator D is 4 and 32 respectively, the convolution kernel size is 1×1, and the step size is 1. The number of convolution kernels of the two convolution layers in the CBAM attention mechanism layer (i.e., the first CBAM attention mechanism layer) in the generator G is 8 and 64 respectively, the convolution kernel size is 1×1, and the step size is 1.

[0089] The following is an explanation of the training process of the multi-category assisted generative adversarial network.

[0090] As a specific embodiment, a multi-category auxiliary generative adversarial network is trained using the GPR B-scan image dataset of the multiple target categories, and the training steps are as follows:

[0091] S1: Construct a multi-category auxiliary generative adversarial network and initialize the training weight parameters.

[0092] S2: Assume that there are Y GPR B-scan images to be trained in the training data set (the processed GPR B-scan images of multiple categories obtained in step 101), including echo data of multiple categories of targets. Divide the training sample images into p groups, the first p-1 groups correspond to b untrained GPR B-scan images, the pth group corresponds to b1 untrained GPR B-scan images, Y, p, b, b1 are positive integers, b1=Y-(p-1)*b, where Indicates rounding up. That is, one epoch cycle will perform p batch training, the first p-1 batch training will train b images, and the last batch training will train b1 images. At the beginning of each epoch cycle, the training data set is randomly shuffled. In addition, in each batch of training images, the number and order of various categories are also randomly determined to ensure that the network can obtain better training results.

[0093] S3: In each batch training, a normally distributed noise vector with the same number as the images to be trained is randomly generated, and the noise vector and the GPR B-scan image to be trained in the training data set and the category label corresponding to the GPR B-scan image are input into a multi-category auxiliary generative adversarial network, and the loss error of the generator G and the discriminator D is gradually reduced through the optimizer to train the multi-category auxiliary generative adversarial network.

[0094] S4: Repeat steps S2-S3 until one epoch of training is completed.

[0095] S5: Repeat steps S2 to S4 until the loss error of the multi-category auxiliary generative adversarial network tends to be stable, and a trained multi-category auxiliary generative adversarial network is obtained.

[0096] Specifically, in each batch training, the parameters of the generator G are first fixed, and the parameters of the discriminator D are trained. Untrained GPR B-scan images (i.e., real images), GPR B-scan images generated by the generator G (i.e., generated images), and the category labels corresponding to each image are used as input, and the input real images and generated images are translated, cropped, flipped, and other operations are performed with an adaptive probability to improve the robustness and generalization ability of the discriminator D. The loss function of the discriminator D is minimized by the optimizer so that it can correctly distinguish between real images, generated images, and the corresponding categories corresponding to the images; then the parameters of the discriminator D are fixed, and the parameters of the generator G are trained. The generated image is output by the input random noise vector and the category label. The loss function of the generator G is minimized by the optimizer so that the generated image of the corresponding category is closer to the real image; this is repeated alternately to gradually reduce the loss error of the generator G and the discriminator D.

[0097] During specific training, the loss function of the discriminator D includes a first adversarial loss and a first classification loss. The loss function L of the discriminator D is D satisfy:

[0098] L D =L DS +L DC

[0099] L DS =E[(D(x,c)-1) 2 ]+E[(D(G(z,c))) 2 ]

[0100]

[0101] The loss function of the generator G includes a second adversarial loss and a second classification loss. The loss function L of the generator G Gsatisfy:

[0102] L G =L GS +L GC

[0103] L GS =E[(D(G(z,c))-1) 2 ]

[0104]

[0105] Among them, L DS is the adversarial loss of the discriminator D, L DC is the classification loss of the discriminator D, L GS is the adversarial loss of the generator G, L GC is the classification loss of the generator G, E is the average loss of this batch of training samples, D(x,c) is the output value obtained by inputting the real image of the corresponding category into the discriminator D to measure the true and false, G(z,c) is the generated GPRB-scan image of the corresponding category obtained by inputting the noise vector and label information into the generator G, and D(G(z,c)) is the output value obtained by inputting the generated GPRB-scan image into the discriminator D to measure the true and false. p(C=c|x) represents the probability that the discriminator judges that the input image category is c when the real image x is input; p c is the true probability of category c; p(C=c|G(z,c)) represents the probability that the discriminator outputs category c when the input generated image G(z,c); N represents the total number of categories.

[0106] Use the optimizer to gradually reduce the loss error of the generator G and the discriminator D. When minimizing the loss function L of the discriminator D D , the input real image and the generated image are translated, cropped, flipped, etc. with an adaptive probability to improve the robustness and generalization ability of the discriminator D. The calculation formula of the adaptive probability is:

[0107]

[0108] Where p is the adaptive probability, which means that the image is translated, cropped, flipped, etc. with the probability of p. If the loss function L of the discriminator D is D If the value is small, overfitting is likely to occur, and the p value should be increased accordingly. On the contrary, if the loss function L D The larger the value, the smaller the p value will be; until L D ≥1, the p-value is 0, indicating that no operation is performed on the image.

[0109] After iterative training to obtain a trained multi-category auxiliary generative adversarial network, a noise vector that obeys the normal distribution is randomly generated, and a category code is randomly generated according to the category information of the real image as a condition. Both are input into the generator G at the same time to obtain the GPRB-scan image of the corresponding category with the same size as the dataset image.

[0110] In an embodiment of the present invention, GPRB-scan images of various target categories are obtained through simulation or actual measurement; a multi-category auxiliary generative adversarial network is constructed, and the network structure is deepened by residual transposed convolution blocks and residual convolution blocks in the multi-category auxiliary generative adversarial network to learn deep-level features of images of various categories, a CBAM attention mechanism module is used to adaptively extract image features, and spectral normalization and instance normalization are used to stabilize the network output; the multi-category auxiliary generative adversarial network is trained using GPRB-scan images of various target categories to obtain a trained multi-category auxiliary generative adversarial network.

[0111] Through the method provided by the embodiment of the present invention, on the one hand, multiple different categories of GPRB-scan image data can be quickly generated in batches only by inputting noise vectors and different category labels, without the need to use simulation and actual measurement methods to obtain GPRB-scan image data of each category one by one, and without the need to repeatedly train the network, which improves the convenience of data generation and reduces the amount of data required for network training. On the other hand, this method generates new images through a multi-category assisted generative adversarial network, rather than augmenting the images in the existing data set, so the GPRB-scan image data obtained by this method can generate new features, which makes the generated GPRB-scan image data more diverse and the quality of the obtained data set higher.

[0112] The present invention effectively solves the problem of insufficient simulation and measured data, and can simultaneously generate multiple categories of GPRB-scan image data. The GPRB-scan image data obtained by this method can be used for GPR data set production, training and testing of GPR image processing methods based on deep learning, and can also provide rich GPR data support for GPR clutter suppression, target detection, imaging inversion and other research.

[0113] For ease of understanding, a specific embodiment is used as an example for description below.

[0114] See also Figure 7In this example, data of three target categories, underground steel bars, underground hollow PVC pipes, and underground water pipes, were collected. A sandbox was used as the test site, and quartz sand was used as the background medium. Steel bars, hollow PVC pipes, and mineral water bottles filled with water were used as targets, and buried according to different burial positions and depths, including single-target and dual-target cases. The acquired data were preprocessed, including zero bias correction, automatic gain, etc. After screening and processing, 110 measured GPR B-scan images of underground steel bars (i.e., GPR B-scan images of category 1), 107 measured GPR B-scan images of underground hollow PVC pipes (i.e., GPR B-scan images of category 2), and 103 measured GPR B-scan images of underground water pipes (i.e., GPR B-scan images of category 3) were obtained. To meet the training needs, the dimension of each image is set to 1×256×256, which means that the number of channels of each GPR B-scan image is 1, and the size is uniformly 256×256. These processed images constitute the measured GPRB-scan image dataset of underground steel bars, the measured GPRB-scan image dataset of underground hollow PVC pipes, and the measured GPR B-scan image dataset of underground water pipes (i.e., three categories of GPR B-scan images). Moreover, most of the data sets in these three categories are single-target images. The measured GPR B-scan image dataset of underground steel bars, the measured GPRB-scan image dataset of underground hollow PVC pipes, and the measured GPRB-scan image dataset of underground water pipes together constitute the training dataset for multi-category auxiliary generative adversarial network training.

[0115] A Multi-class Assisted GenerativeAdversarial Network (MA-GAN) was constructed. The network used the Adam optimizer, with the beta1 parameter of the Adam optimizer set to 0.5, the beta2 parameter set to 0.999, and the learning rate of the network set to 0.0001. A total of 3000 epochs were trained, with 40 batch trainings per epoch, 8 samples per batch training, and the model saved every 50 epochs. The saved and converged model was used for data generation. In addition, the training set was shuffled at the beginning of each epoch, and the images of the three target categories to be trained were randomly arranged. At the same time, in each batch, the number and order of the measured GPRB-scan images of underground steel bars, the measured GPRB-scan images of underground hollow PVC pipes, and the measured GPRB-scan images of underground water pipes were also randomly determined.

[0116] Figure 8GPR B-scan image training set data for three target categories, Figure 8 (a-1) to (a-5) are measured GPR B-scan images of underground steel bars. Figure 8 (b-1) to (b-5) are measured GPR B-scan images of underground hollow PVC pipes. Figure 8 (c-1) to (c-5) are measured GPR B-scan images of underground water pipes. Fig. 9 The generated GPRB-scan images of three target categories are shown, correspondingly, Fig. 9 (a-1) to (a-5) are the generated steel bar GPRB-scan images. Fig. 9 (b-1) to (b-5) are the generated GPR B-scan images of the hollow PVC pipe. Fig. 9 (c-1) to (c-5) are the generated GPRB-scan images of water pipes. Visually, it can be seen that the multi-category assisted generative adversarial network can well generate high-quality measured images of three target categories, and the generated images also include single-target and dual-target cases.

[0117] In order to further verify the relevance of the images generated by the network and the real images, the present invention uses the mFSIM (meanFeature Similarity Index Measure) indicator for quantitative analysis. mFSIM focuses on the similarity between features rather than the similarity between images, and its range is usually between 0 and 1. The closer to 1, the more similar the features of the two images are. The definition of mFSIM is as follows:

[0118]

[0119] Among them, s i is the i-th sample in the generated dataset, r j is the jth sample in the real dataset. r 、N s are the total number of samples in the real dataset and the generated dataset, respectively.

[0120] In order to verify the advancedness of the network, a comparative experiment was conducted. In the designed MA-GAN, an improved residual transposed convolution block and CBAM attention mechanism layer were designed in the generator G, a CBAM attention mechanism layer and an improved residual convolution block were designed in the discriminator D, and spectral normalization and instance normalization were used in the backbone of the network.

[0121] The generator of the existing ACGAN consists of 8 transposed convolutional layers, which generates data through the input random noise vector and category information output; the discriminator consists of 6 convolutional layers, 1 output layer and 1 auxiliary classifier, and the output is the probability that the input image belongs to the real image and its category probability.

[0122] The existing MGPR-GAN adds a self-attention mechanism module based on CGAN, establishes long-range connections between data, and can generate data for multiple categories of targets.

[0123] The ACGAN and MGPR-GAN were trained using the aforementioned data sets, and GPR B-scan images were generated. The images generated by the MA-GAN in this embodiment were compared with the images generated by the ACGAN and MGPR-GAN, and the generated data were evaluated according to the aforementioned indicators. The experimental results are shown in Table 1:

[0124] Table 1 Comparison of mFSIM of images generated by different GAN models

[0125]

[0126] The results in Table 1 show that the GPR B-scan images of the three target categories generated by the multi-category assisted generative adversarial network all achieved the highest mFSIM. Compared with ACGAN, the mFSIM index of MA-GAN increased by an average of 13.76%; compared with MGPR-GAN, the mFSIM index of MA-GAN increased by an average of 9.75%.

[0127] In summary, it can be seen that the method provided by the embodiment of the present invention can generate high-quality GPR B-scan images of the corresponding category while only requiring a small number of training data sets, thereby expanding the measured data set.

[0128] See also Fig.10 The embodiment of the present invention provides a ground penetrating radar image data augmentation device 1000, comprising:

[0129] An acquisition module 1001 is used to acquire GPR B-scan images of multiple categories corresponding to multiple underground targets;

[0130] A construction module 1002 is used to construct a multi-category auxiliary generative adversarial network, wherein the multi-category auxiliary generative adversarial network includes a generator and a discriminator, wherein the generator includes a residual transposed convolution block, and the discriminator includes a residual convolution block. The residual transposed convolution block and the residual convolution block are used to deepen the network structure to learn deep-level features of images of various categories. The generator and the discriminator both include a CBAM attention mechanism layer, a spectral normalization layer, and an instance normalization layer. The CBAM attention mechanism layer includes a channel attention mechanism and a spatial attention mechanism, which are used to adaptively extract image features. The spectral normalization layer and the instance normalization layer are used to stabilize the output of the network.

[0131] An iterative training module 1003 is used to iteratively train the multi-category auxiliary generative adversarial network using the multiple categories of GPR B-scan images as training data sets to obtain a trained multi-category auxiliary generative adversarial network;

[0132] The generation module 1004 is used to input the noise vector and the category label into the trained multi-category auxiliary generative adversarial network to generate a target GPR B-scan image corresponding to the category label.

[0133] Optionally, the generator comprises:

[0134] Two parallel transposed convolution blocks, one of which receives the noise vector as input and the other receives the category label as input;

[0135] A first processing block, the first processing block is used to process the outputs of the two parallel transposed convolution blocks;

[0136] The CBAM attention mechanism layer, the CBAM attention mechanism layer is used to adaptively extract features from the output of the first processing block;

[0137] A first transposed convolutional layer, wherein the first transposed convolutional layer is used to generate a target GPR B-scan image based on the output of the CBAM attention mechanism layer.

[0138] Optionally, the first processing block includes 5 first processing sub-blocks connected in sequence, and the first processing sub-block includes the residual transposed convolution block and the upsampling layer connected in sequence.

[0139] Optionally, the residual transposed convolution block includes sequentially connected:

[0140] A first residual part, wherein the first residual part includes two connected transposed convolutional layer groups, wherein the transposed convolutional layer group includes a second transposed convolutional layer, a first spectral normalization layer, and a first instance normalization layer connected in sequence;

[0141] A first direct mapping part, wherein the first direct mapping part includes a first convolutional layer, and the first convolutional layer is used to adjust the number of channels of the input feature map;

[0142] A first processing layer group, wherein the first processing layer group is used to add the output of the first residual part and the output of the first direct mapping part and output them through an activation function.

[0143] Optionally, the discriminator includes:

[0144] a second processing block, the second processing block being used to process an input image;

[0145] The CBAM attention mechanism layer, the CBAM attention mechanism layer is used to adaptively extract features from the output of the second processing block;

[0146] A third processing block, the third processing block is used to process the output of the CBAM attention mechanism layer;

[0147] an output layer, the output layer being used to determine the probability that the input image is a real image based on the output of the third processing block;

[0148] An auxiliary classifier, the auxiliary classifier is used to determine the category probability of the input image based on the output of the third processing block.

[0149] Optionally, the third processing block includes 5 second processing sub-blocks connected in sequence, and the second processing sub-block includes the residual convolution block and the global average pooling layer connected in sequence.

[0150] Optionally, the residual convolution block includes:

[0151] A second residual part, wherein the second residual part includes two connected convolutional layer groups, and the convolutional layer group includes a second convolutional layer, a second spectral normalization layer, and a second instance normalization layer connected in sequence;

[0152] A second direct mapping part, wherein the second direct mapping part includes a third convolution layer, and the third convolution layer is used to adjust the number of channels of the input feature map;

[0153] The second processing layer group is used to add the output of the second residual part and the output of the second direct mapping part and then output them through an activation function and a discard layer.

[0154] The ground penetrating radar image data augmentation device 1000 provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0155] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0157] like Fig.11 As shown, an embodiment of the present application provides an electronic device 1100, including: a memory 1102, a processor 1101, and a program stored in the memory 1102 and executable on the processor 1101; the processor 1101 is used to read the program in the memory 1102 to implement the steps in the ground penetrating radar image data augmentation method as described above.

[0158] The embodiment of the present application also provides a readable storage medium, on which a program is stored. When the program is executed by a processor, each process of the above-mentioned ground penetrating radar image data augmentation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here. Among them, the readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as compact disk (CD), digital video disc (DVD), Blu-ray Disc (BD), high-definition versatile disc (HVD), etc.), and semiconductor memory (such as read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read only memory (EEPROM), non-volatile memory (NAND FLASH), solid-state drive (Solid State Disk or Solid State Drive, SSD)), etc.

[0159] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0161] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for augmenting ground penetrating radar image data, characterized in that: include: Acquire GPR B-scan images of multiple categories corresponding to multiple underground targets; Constructing a multi-category auxiliary generative adversarial network, the multi-category auxiliary generative adversarial network includes a generator and a discriminator, the generator includes a residual transposed convolution block, the discriminator includes a residual convolution block, the residual transposed convolution block and the residual convolution block are used to deepen the network structure to learn deep-level features of various categories of images, the generator and the discriminator both include a CBAM attention mechanism layer, a spectral normalization layer and an instance normalization layer, the CBAM attention mechanism layer includes a channel attention mechanism and a spatial attention mechanism, which are used to adaptively extract image features, and the spectral normalization layer and the instance normalization layer are used to stabilize the output of the network; Iteratively training the multi-category auxiliary generative adversarial network using the multiple categories of GPR B-scan images as training data sets to obtain a trained multi-category auxiliary generative adversarial network; Inputting the noise vector and the category label into the trained multi-category auxiliary generative adversarial network to generate a target GPR B-scan image corresponding to the category label; Wherein, the generator includes: Two parallel transposed convolution blocks, one of which receives the noise vector as input and the other receives the category label as input; A first processing block, the first processing block is used to process the outputs of the two parallel transposed convolution blocks; A first CBAM attention mechanism layer, wherein the first CBAM attention mechanism layer is used to adaptively extract features from the output of the first processing block; A first transposed convolutional layer, the first transposed convolutional layer is used to generate a target GPR B-scan image based on the output of the first CBAM attention mechanism layer; Wherein, the discriminator includes: a second processing block, the second processing block being used to process an input image; A second CBAM attention mechanism layer, wherein the second CBAM attention mechanism layer is used to adaptively extract features from the output of the second processing block; A third processing block, the third processing block is used to process the output of the second CBAM attention mechanism layer; an output layer, the output layer being used to determine the probability that the input image is a real image based on the output of the third processing block; An auxiliary classifier, the auxiliary classifier is used to determine the category probability of the input image based on the output of the third processing block.

2. The method according to claim 1, characterized in that: The first processing block includes five first processing sub-blocks connected in sequence, and the first processing sub-block includes the residual transposed convolution block and the upsampling layer connected in sequence.

3. The method according to claim 2, characterized in that: The residual transposed convolution block includes the following sequentially connected: A first residual part, wherein the first residual part includes two connected transposed convolutional layer groups, wherein the transposed convolutional layer group includes a second transposed convolutional layer, a first spectral normalization layer, and a first instance normalization layer connected in sequence; A first direct mapping part, wherein the first direct mapping part includes a first convolutional layer, and the first convolutional layer is used to adjust the number of channels of the input feature map; A first processing layer group, wherein the first processing layer group is used to add the output of the first residual part and the output of the first direct mapping part and output them through an activation function.

4. The method according to claim 1, characterized in that: The third processing block includes five second processing sub-blocks connected in sequence, and the second processing sub-block includes the residual convolution block and the global average pooling layer connected in sequence.

5. The method according to claim 4, characterized in that: The residual convolution block includes: A second residual part, wherein the second residual part includes two connected convolutional layer groups, and the convolutional layer group includes a second convolutional layer, a second spectral normalization layer, and a second instance normalization layer connected in sequence; A second direct mapping part, wherein the second direct mapping part includes a third convolution layer, and the third convolution layer is used to adjust the number of channels of the input feature map; The second processing layer group is used to add the output of the second residual part and the output of the second direct mapping part and then output them through an activation function and a discard layer.

6. A ground penetrating radar image data augmentation device, characterized in that: include: An acquisition module, used for acquiring GPR B-scan images of multiple categories corresponding to multiple underground targets; A construction module is used to construct a multi-category auxiliary generative adversarial network, wherein the multi-category auxiliary generative adversarial network includes a generator and a discriminator, wherein the generator includes a residual transposed convolution block, and the discriminator includes a residual convolution block. The residual transposed convolution block and the residual convolution block are used to deepen the network structure to learn deep-level features of images of various categories. The generator and the discriminator both include a CBAM attention mechanism layer, a spectral normalization layer, and an instance normalization layer. The CBAM attention mechanism layer includes a channel attention mechanism and a spatial attention mechanism, which are used to adaptively extract image features. The spectral normalization layer and the instance normalization layer are used to stabilize the output of the network. An iterative training module, used for iteratively training the multi-category auxiliary generative adversarial network using the multiple categories of GPR B-scan images as training data sets to obtain a trained multi-category auxiliary generative adversarial network; A generation module, used for inputting the noise vector and the category label into the trained multi-category auxiliary generative adversarial network to generate a target GPR B-scan image corresponding to the category label; Wherein, the generator includes: Two parallel transposed convolution blocks, one of which receives the noise vector as input and the other receives the category label as input; A first processing block, the first processing block is used to process the outputs of the two parallel transposed convolution blocks; A first CBAM attention mechanism layer, wherein the first CBAM attention mechanism layer is used to adaptively extract features from the output of the first processing block; A first transposed convolutional layer, the first transposed convolutional layer is used to generate a target GPR B-scan image based on the output of the first CBAM attention mechanism layer; Wherein, the discriminator includes: a second processing block, the second processing block being used to process an input image; A second CBAM attention mechanism layer, wherein the second CBAM attention mechanism layer is used to adaptively extract features from the output of the second processing block; A third processing block, the third processing block is used to process the output of the second CBAM attention mechanism layer; an output layer, the output layer being used to determine the probability that the input image is a real image based on the output of the third processing block; An auxiliary classifier, the auxiliary classifier is used to determine the category probability of the input image based on the output of the third processing block.

7. An electronic device comprising: A memory, a processor, and a program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps in the method for augmenting ground penetrating radar image data as described in any one of claims 1 to 5.

8. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps in the method for augmenting ground penetrating radar image data according to any one of claims 1 to 5 are implemented.

Images