Transmission line information extraction method from UAV images based on generative adversarial network

Abstract

A method for extracting transmission line information from images collected by drones based on generative adversarial networks relates to a transmission line information extraction technology, in order to solve the problem of poor recognition efficiency of existing image transmission line information through manual or semi-artificial line defects. The present invention generates a preprocessed image by preprocessing the original image; classifies and annotates the generated preprocessed image to generate a label image; selects multiple original images and paired label images to form a training set; constructs a generative adversarial network ensemble learning model; inputs the formed training set into the generative adversarial network ensemble learning model, and combines it with the real application data of the transmission line to obtain a solidified generative adversarial network ensemble learning model; inputs the drone collected image to be extracted into the solidified generative adversarial network ensemble learning model to extract the transmission line information in the drone collected image to be extracted. The beneficial effect is high recognition efficiency.

Description

Technical Field

[0001] The invention relates to a technology for extracting information of a power transmission line. Background Art

[0002] As an important power grid infrastructure, transmission lines play an important role in the process of power transmission. With the large-scale use of high-resolution drone photography technology, remote aerial photography has become a conventional means of line inspection, but how to quickly and accurately extract transmission line and tower information from aerial images has attracted attention. In terms of transmission line defect identification methods, most cases currently still use manual means to directly observe the image for judgment, or use semi-automatic extraction methods, that is, simple image recognition plus manual judgment. However, due to the existence of massive aerial image data and complex image information, this manual or semi-manual line defect identification process is not only inefficient, untimely and inaccurate, but also leads to decision delays. Failure to repair line defects in a timely manner will lead to defect expansion, which is easy to cause power grid accidents. Summary of the invention

[0003] The purpose of the present invention is to solve the problem that the existing image transmission line information is identified through manual or semi-artificial line defects, and the recognition efficiency is poor. A method for extracting transmission line information from images collected by drones based on a generative adversarial network is proposed.

[0004] The method for extracting power line information from images collected by a drone based on a generative adversarial network of the present invention comprises the following steps:

[0005] Step 1: preprocessing the original image to generate a preprocessed image;

[0006] Step 2: Classify and annotate the preprocessed image generated in step 1 to generate a label image;

[0007] Step 3: Select multiple original images and their corresponding label images for pairing to form a training set;

[0008] Step 4: Build a generative adversarial network ensemble learning model;

[0009] Step 5: Input the training set formed in step 3 into the generative adversarial network ensemble learning model constructed in step 4, and solidify the parameters of the generative adversarial network ensemble learning model constructed in step 2 in combination with the real application data of the transmission line, so as to obtain the solidified generative adversarial network ensemble learning model;

[0010] Step 6: Input the drone-collected image to be extracted into the generative adversarial network ensemble learning model solidified in step 5 for image recognition, and extract the power transmission line information from the drone-collected image to be extracted.

[0011] Furthermore, generating the pre-processed image in step 1 includes: performing pitch adjustment, roll operation or yaw mode image correction on the original image.

[0012] Furthermore, the label image generated in step 2 is annotated with the tower area, sky area, forest area, river area and open space area on the pre-processed image.

[0013] Furthermore, the construction process of the generative adversarial network ensemble learning model in step 4 includes

[0014] Step 41: On the PyTorch platform, use the residual network to build multiple generators;

[0015] Step 42: Introduce a collaborative working mechanism between the multiple generators built in step 41, allowing multiple generators to exchange information;

[0016] Step 43: Use the noise in the label image as the input of multiple generators, and fuse the outputs of multiple generators into feature maps to generate the final image;

[0017] Step 44: Input the final image generated in step 43 into the discriminant network to complete the construction of the generative adversarial network ensemble learning model.

[0018] Furthermore, the specific method of obtaining the solidified generative adversarial network ensemble learning model in step 5 is:

[0019] Step 51: Input the real application data of the power transmission line into the discriminant network for comparison with the final image generated in step 43; the discriminant network outputs the discriminant error;

[0020] Step 52: The discrimination error output in step 51 is transmitted back to the multiple generators built in step 41 to update the parameters of the multiple generators, thereby completing the solidification of the parameters of the generative adversarial network ensemble learning model constructed in step 4.

[0021] Furthermore, the residual network in step 4-1 is composed of residual blocks;

[0022] The residual block includes a weight layer 1, a weight layer 2 and an adder;

[0023] The input x is used as the input of the weight layer 1, and the output F(x) is obtained after passing through the weight layer 1; where F(x) represents the output of x after the weighted and activated functions of the 2 layers;

[0024] The output F(x) of weight layer 1 is then input to weight layer 2, and after passing through weight layer 2, the output is F(x)+x;

[0025] The output F(x)+x is used as one input of the adder, and the input x is used as another input of the adder after passing through the label function.

[0026] Furthermore, the discriminant network in step 44 includes convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, fully connected layer 1 and fully connected layer 2;

[0027] Among them, convolution layer 1 has 64 convolution kernels, convolution layer 2 has 128 convolution kernels, convolution layer 3 has 256 convolution kernels, convolution layer 4 has 512 convolution kernels, fully connected layer 1 has 1024 output dimensions, and fully connected layer 2 has 1 output dimension.

[0028] The beneficial effects of the present invention are as follows: the information extraction method introduces a multi-generator generative adversarial network model with an integrated learning idea, which includes multiple generative networks, each of which is built using a residual network, and a collaborative working mechanism is introduced between the generative networks, allowing multiple generative networks to exchange information and accelerate network learning. Finally, the feature maps of the multiple generative networks are fused as the final image and input into the discriminant network; the solidified generative adversarial network integrated learning model has high recognition efficiency and has a good recognition effect on extracting transmission line information using high-resolution drone images. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of the method for extracting power line information from images collected by drones based on a generative adversarial network as described in the first specific implementation mode;

[0030] Figure 2 This is a schematic diagram of the structure of the generative adversarial network ensemble learning model in the fourth specific implementation mode;

[0031] Figure 3 Schematic diagram of the structure of the residual block in the sixth specific implementation mode. DETAILED DESCRIPTION

[0032] Specific implementation method 1: Combination Figure 1 This embodiment describes the method for extracting power line information from drone-collected images based on a generative adversarial network, which includes the following steps:

[0033] Step 1: preprocessing the original image to generate a preprocessed image;

[0034] Step 2: Classify and annotate the preprocessed image generated in step 1 to generate a label image;

[0035] Step 3: Select multiple original images and their corresponding label images for pairing to form a training set;

[0036] Step 4: Build a generative adversarial network ensemble learning model;

[0037] Step 5: Input the training set formed in step 3 into the generative adversarial network ensemble learning model constructed in step 4, and solidify the parameters of the generative adversarial network ensemble learning model constructed in step 2 in combination with the real application data of the transmission line, so as to obtain the solidified generative adversarial network ensemble learning model;

[0038] Step 6: Input the drone-collected image to be extracted into the generative adversarial network ensemble learning model solidified in step 5 for image recognition, and extract the power transmission line information from the drone-collected image to be extracted.

[0039] In this embodiment, step three selects 5000 original images and their corresponding label images for pairing; in step five, as the number of training iterations increases, the difference loss value between the real application data and the label image becomes smaller and smaller, and the entire generative adversarial network ensemble learning model is in a convergence state. At this time, the parameters of the generative adversarial network ensemble learning model are solidified.

[0040] Specific implementation method 2: This implementation method further limits the method for extracting transmission line information from images collected by a drone based on a generative adversarial network described in specific implementation method 1. In this implementation method, generating a preprocessed image in step 1 includes: performing pitch adjustment, roll operation or yaw mode image correction on the original image.

[0041] Specific implementation method three: This implementation method further limits the method for extracting transmission line information from images collected by drones based on a generative adversarial network as described in specific implementation method one. In this implementation method, the label image generated in step two is a label that marks the tower area, sky area, forest area, river area and open space area on the pre-processed image.

[0042] Specific implementation method four: Combination Figure 2 This embodiment further defines the method for extracting transmission line information from drone-collected images based on a generative adversarial network described in the first embodiment. In this embodiment, the construction process of the generative adversarial network ensemble learning model in step 4 includes:

[0043] Step 41: On the PyTorch platform, use the residual network to build multiple generators;

[0044] Step 42: Introduce a collaborative working mechanism between the multiple generators built in step 41, allowing multiple generators to exchange information;

[0045] Step 43: Use the noise in the label image as the input of multiple generators, and fuse the outputs of multiple generators into feature maps to generate the final image;

[0046] Step 44: Input the final image generated in step 43 into the discriminant network to complete the construction of the generative adversarial network ensemble learning model.

[0047] In this embodiment, the generative adversarial network ensemble learning model described in this embodiment proposes a multi-generator generative adversarial network model with ensemble learning ideas, each generator contains three parts: front, middle and back; the whole generator has 9 layers, of which the front part contains 3 convolutional layers, the middle part is 3 residual network modules, and the back part corresponds to 2 transposed convolutions and 1 convolution layer; since each generator has the same goal and there is no difference in influence, the generators in the model are arranged in parallel; in step 43, the noise in the label image refers to the sky area, forest area, river area and open space area marked on the preprocessed image.

[0048] The transmission line images collected by drone aerial photography are generally similar, but there are differences in the image features of different power towers, such as straight towers and tension towers, wine glass-shaped and cat-head-shaped, and the difference in light and dark between the backgrounds of the photos. When building the generative network, each generator adheres to the method of roughly the same details, that is, each generator is set to the same number of convolution layers, different convolution kernel sizes and convolution steps. This setting method prompts each generator to obtain different image information, further ensuring the difference in the generated images between the generators.

[0049] At the same time, due to the similarity between image features, this embodiment further introduces a learning method that allows multiple generators to communicate, which is called collaborative work between generators; this learning method can accelerate the network to fit the real data features and thus reduce the problems that are prone to occur during training; the collaborative work mechanism mainly includes two parts: parameter sharing of each generator and feature map fusion: since the low-dimensional features of images in the same data set are often similar, parameter sharing can not only reduce the amount of parameters, but also shorten the training time of the network; regarding feature map fusion, this embodiment uses a weighted fusion method to fuse multiple feature maps into one image, assigns different weights according to the performance of each generator, and obtains the final feature map by weighting; the model organization method of this collaborative work effectively avoids the disadvantage of limited learning ability of a single generator, and by fusing the images generated by each generator, it greatly improves the amount of information contained in the feature map.

[0050] The loss of the integrated generative network system consists of two parts: the generative adversarial loss of the original GAN ​​and the collaborative work loss of each generator. Its loss function is shown in formula (1);

[0051] G * = arg min(W(P Data ||P g )-βJSD(PG1 ,P G2 ,...,P Gn )) (1)

[0052] Where P g Represents the joint distribution of multiple generative networks; the first item in the brackets is the loss in the original GAN ​​to drive the generated distribution close to the real data distribution, and the second item is used to adjust the generation of more detailed information between the generated networks.

[0053] Specific implementation method 5: This implementation method further limits the method for extracting transmission line information from drone-collected images based on a generative adversarial network described in specific implementation method 4. In this implementation method, the specific method for obtaining the solidified generative adversarial network integrated learning model in step 5 is:

[0054] Step 51: Input the real application data of the power transmission line into the discriminant network for comparison with the final image generated in step 43; the discriminant network outputs the discriminant error;

[0055] Step 52: The discrimination error output in step 51 is transmitted back to the multiple generators built in step 41 to update the parameters of the multiple generators, thereby completing the solidification of the parameters of the generative adversarial network ensemble learning model constructed in step 4.

[0056] Specific implementation method six: Combination Figure 3 This embodiment is described. This embodiment further limits the method for extracting transmission line information from drone-collected images based on a generative adversarial network described in the specific embodiment 4. In this embodiment, the residual network in step 41 is composed of residual blocks;

[0057] The residual block includes a weight layer 1, a weight layer 2 and an adder;

[0058] The input x is used as the input of the weight layer 1, and the output F(x) is obtained after passing through the weight layer 1; where F(x) represents the output of x after the weighted and activated functions of the 2 layers;

[0059] The output F(x) of weight layer 1 is then input to weight layer 2, and after passing through weight layer 2, the output is F(x)+x;

[0060] The output F(x)+x is used as one input of the adder, and the input x is used as another input of the adder after passing through the label function.

[0061] In this embodiment, x is the input, and F(x) represents the output obtained by applying the weighted sum activation function to x in two layers, that is,

[0062] F(x)=W2σ(W1x) (2)

[0063] Where W1 and W2 are weight matrices respectively; σ is the activation function.

[0064] Specific implementation method seven: This implementation method further limits the method for extracting transmission line information from drone-collected images based on a generative adversarial network described in specific implementation method four. In this implementation method, the discriminant network in step four-four includes convolutional layer one, convolutional layer two, convolutional layer three, convolutional layer four, fully connected layer one and fully connected layer two;

[0065] Among them, convolution layer 1 has 64 convolution kernels, convolution layer 2 has 128 convolution kernels, convolution layer 3 has 256 convolution kernels, convolution layer 4 has 512 convolution kernels, fully connected layer 1 has 1024 output dimensions, and fully connected layer 2 has 1 output dimension.

[0066] In this implementation, the convolutional layer 1, the convolutional layer 2, the convolutional layer 3, the convolutional layer 4, the fully connected layer 1 and the fully connected layer 2 are connected in a convolutional series manner.

Claims

1. A method for extracting transmission line information from images collected by drones based on a generative adversarial network, the method comprising the following steps: Step 1: preprocessing the original image to generate a preprocessed image; Step 2: Classify and annotate the preprocessed image generated in step 1 to generate a label image; Step 3: Select multiple original images and their corresponding label images for pairing to form a training set; Step 4: Build a generative adversarial network ensemble learning model; Step 5: Input the training set formed in step 3 into the generative adversarial network ensemble learning model constructed in step 4, and solidify the parameters of the generative adversarial network ensemble learning model constructed in step 2 in combination with the real application data of the transmission line, so as to obtain the solidified generative adversarial network ensemble learning model; Step 6: Input the drone image to be extracted into the generative adversarial network ensemble learning model solidified in step 5 to perform image recognition, and extract the power transmission line information in the drone image to be extracted; It is characterized in that the construction process of generating an ensemble learning model of a confrontation network in step 4 includes: Step 41: On the PyTorch platform, use the residual network to build multiple generators; Step 42: Introduce a collaborative working mechanism between the multiple generators built in step 41, allowing multiple generators to exchange information; Step 43: Use the noise in the label image as the input of multiple generators, and fuse the outputs of multiple generators into feature maps to generate the final image; Step 44: Input the final image generated in step 43 into the discriminant network to complete the construction of the generative adversarial network ensemble learning model.

2. The method for extracting transmission line information from images collected by drones based on a generative adversarial network according to claim 1 is characterized in that: Generating the pre-processed image in step 1 includes: performing pitch adjustment, roll operation or yaw mode image correction on the original image.

3. The method for extracting transmission line information from drone-collected images based on a generative adversarial network according to claim 1 is characterized in that: The label image generated in step 2 is a pre-processed image with tower areas, sky areas, forest areas, river areas and open space areas marked on it.

4. The method for extracting transmission line information from drone-collected images based on a generative adversarial network according to claim 1 is characterized in that: The specific method for obtaining the solidified generative adversarial network ensemble learning model in step 5 is: Step 51: Input the real application data of the power transmission line into the discriminant network for comparison with the final image generated in step 43; the discriminant network outputs the discriminant error; Step 52: The discrimination error output in step 51 is transmitted back to the multiple generators built in step 41 to update the parameters of the multiple generators, thereby completing the solidification of the parameters of the generative adversarial network ensemble learning model constructed in step 4.

5. The method for extracting transmission line information from drone-collected images based on a generative adversarial network according to claim 1 is characterized in that: In step 4, the residual network is composed of residual blocks; The residual block includes a weight layer 1, a weight layer 2 and an adder; The input x is used as the input of the weight layer 1, and the output F(x) is obtained after passing through the weight layer 1; where F(x) represents the output of x after the weighted and activated functions of the 2 layers; The output F(x) of weight layer 1 is then input to weight layer 2, and after passing through weight layer 2, the output is F(x)+x; The output F(x)+x is used as one input of the adder, and the input x is used as another input of the adder after passing through the label function.

6. The method for extracting transmission line information from drone-collected images based on a generative adversarial network according to claim 1, characterized in that: The discriminant network in step 44 includes convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, fully connected layer 1 and fully connected layer 2; Among them, convolution layer 1 has 64 convolution kernels, convolution layer 2 has 128 convolution kernels, convolution layer 3 has 256 convolution kernels, convolution layer 4 has 512 convolution kernels, fully connected layer 1 has 1024 output dimensions, and fully connected layer 2 has 1 output dimension.

Images