A coal mine UAV image enhancement method based on generative adversarial network
Through the method based on the generative adversarial network, the adversarial training generator and discriminator is solved, and the problem of noise and details loss in the image enhancement of coal mine drone is achieved, which is efficiently enhanced to coal mine drone images, and the generated image details are richer , suitable for coal mine inspection scenarios.
Patent Information
- Application Number
- CN202010490445.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-06-02
- Publication Date
- 2024-12-20
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The images captured by coal mine drones are difficult to effectively detect and identify small targets due to severe noise and loss of details. The existing image enhancement methods fail to consider image context information and cannot obtain real and clear images.
Using a method based on a generative adversarial network, by building a network containing generators and discriminators, using clear and blurred images in the training set to engage in adversarial training, the generator generates clear images from blurred images, and the discriminator distinguishes the generated images from clear images, through Error backpropagation updates parameters until the image generated by the generator is judged as a clear image.
It effectively improves the quality of blurred images in coal mine drone inspection scenarios, makes the generated image details richer, and can generate richly detailed enhanced images for brightness, low contrast and image blur problems.
Smart Images
Figure CN111783545B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a coal mine unmanned aerial vehicle image enhancement method based on a generative adversarial network. Background Art
[0002] In recent years, with the rapid development of drone technology, many drone inspection technologies have been applied, especially in the field of coal mines. The manual inspection method is inefficient and labor-intensive, and is limited by the coal mine area, which poses great safety hazards. Drone inspection uses visible light cameras, infrared cameras, lidar ranging sensors, etc. to replace line inspectors to perform inspections and realize functions such as target detection and target recognition. However, due to the complex environmental factors in the coal mine area, the images taken by drones have serious noise and loss of details, making it difficult to effectively detect and identify small targets in the image. Therefore, image enhancement of coal mine drones is of great significance to the widespread application of drone inspection technology.
[0003] Traditional image enhancement methods mainly increase the contrast of images through histogram equalization and Laplace transform, but these methods do not take into account the contextual information in the image and cannot obtain a real and clear image. Summary of the invention
[0004] The present invention provides a coal mine UAV image enhancement method based on a generative adversarial network, which can overcome certain or some defects of the prior art.
[0005] According to the present invention, a coal mine drone image enhancement method based on a generative adversarial network comprises the following steps:
[0006] Step S1, obtaining a clear image taken by a coal mine drone and processing the obtained image, thereby constructing a training set, wherein a single element of the training set is a two-tuple, and the two-tuple includes a clear image and a blurred image corresponding to the clear image;
[0007] Step S2, constructing a generative adversarial network, the constructed generative adversarial network includes a generator and a discriminator;
[0008] Step S3, fixing the parameters of the discriminator, inputting the blurred image in the training set into the generator, and using the discriminator to determine whether the image generated by the generator is clear, thereby training the generator;
[0009] Step S4, fixing the parameters of the generator, inputting the clear images in the training set and the images generated by the generator into the discriminator respectively, to train the discriminator;
[0010] Step S5, repeating steps S3 and S4, and updating the parameters of the generator and the discriminator according to the loss function, until the discriminator identifies the image generated by the generator as a clear image, and the model training is completed;
[0011] In step S6, the blurred image taken by the drone is processed and input into the trained generator, and the corresponding image output by the generator is used as the enhanced clear image.
[0012] Preferably, in step S1, the processing of the acquired image includes the following steps:
[0013] Step S11, according to the acquired image, acquiring a blurred image corresponding to the corresponding clear image by adding noise;
[0014] Step S12, setting the pixel sizes of the corresponding clear image and the corresponding blurred image, thereby forming a single element of the training set.
[0015] Preferably, in step S2, the generator adopts a convolutional neural network, including a linear mapping layer and four convolution layers, the size of the convolution kernel of each convolution layer is 5×5, the linear mapping and the first three convolution layers are respectively connected to batch normalization and ReLU activation functions, and the last convolution layer is connected to a Tanh activation function.
[0016] Preferably, the discriminator adopts a convolutional neural network, including four convolutional layers and one linear mapping layer. The size of the convolution kernel of each convolutional layer is 5×5. The first convolutional layer is connected to the LReLU activation function, and the subsequent three convolutional layers are respectively connected to batch normalization and LReLU activation functions.
[0017] Preferably, in step S3, an error back propagation algorithm is used to fix the parameters of the discriminator and minimize the loss function of the generator.
[0018] Preferably, in step S4, an error back propagation algorithm is used to fix the parameters of the generator and minimize the loss function of the discriminator.
[0019] As a preference, the loss function of the generator is Where G is the generator model, D is the discriminator model, z is random noise, and x is the blurred image.
[0020] As a preference, the loss function of the discriminator is Where G is the generator model, D is the discriminator model, x is the blurred image, and y is the clear image.
[0021] The method of the present invention enables:
[0022] 1. The target image of the inspection can be obtained through the coal mine drone, and the target image can be preprocessed, and then the preprocessed image can be input into the generative adversarial network for processing, so that the image obtained by the coal mine drone can be enhanced through the generative adversarial network. Therefore, the quality of the fuzzy image in the coal mine drone inspection scene can be effectively improved, and the generated image details are richer;
[0023] 2. By establishing a training set to train the generative adversarial network, the parameters of the generator and the discriminator can be better determined, so that the new images obtained by the coal mine drone inspection can be enhanced;
[0024] 3. By introducing the mean square error penalty and gradient penalty function in the training process of the generative adversarial network, the model after the final training is able to generate enhanced images with richer details when processing new images, addressing problems such as brightness, low contrast and image blur in the captured images. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 This is a schematic diagram of the process of the coal mine drone image enhancement method in Example 1;
[0026] Figure 2 A schematic diagram of a model of a generative adversarial network in Example 1;
[0027] Figure 3 This is a schematic diagram of the model of the generator in Example 1;
[0028] Figure 4 Schematic diagram of the model of the discriminator in Example 1;
[0029] Figure 5 This is a rendering of the coal mine drone image enhancement method in Example 1. DETAILED DESCRIPTION
[0030] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.
[0031] Example 1
[0032] Combination Figure 1 and 2 As shown, this embodiment provides a coal mine drone image enhancement method based on a generative adversarial network, which includes the following steps:
[0033] Step S1, obtaining a clear image taken by a coal mine drone and processing the obtained image, thereby constructing a training set, wherein a single element of the training set is a two-tuple, and the two-tuple includes a clear image and a blurred image corresponding to the clear image;
[0034] Step S2, constructing a generative adversarial network, the constructed generative adversarial network includes a generator and a discriminator;
[0035] Step S3, fixing the parameters of the discriminator, inputting the blurred image in the training set into the generator, and using the discriminator to determine whether the image generated by the generator is clear, thereby training the generator;
[0036] Step S4, fixing the parameters of the generator, inputting the clear images in the training set and the images generated by the generator into the discriminator respectively, to realize the training of the discriminator;
[0037] Step S5, repeating steps S3 and S4, and updating the parameters of the generator and the discriminator according to the loss function, until the discriminator identifies the image generated by the generator as a clear image, and the model training is completed;
[0038] In step S6, the blurred image taken by the drone is processed and input into the trained generator, and the corresponding image output by the generator is used as the enhanced clear image.
[0039] Through the method in this embodiment, the target image for inspection can be obtained through the coal mine drone, and the target image can be preprocessed, and then the preprocessed image can be input into the generative adversarial network for processing, so that the image obtained by the coal mine drone can be enhanced better through the generative adversarial network, so it can effectively improve the quality of blurred images in the coal mine drone inspection scenario, making the generated image details richer.
[0040] The generative adversarial network includes a generator and a discriminator. The generator is responsible for generating new images from random noise images and blurred images, and the discriminator is responsible for distinguishing between generated images and clear images. By establishing a training set to train the generative adversarial network, the parameters of the generator and the discriminator can be better determined, so that the new images obtained by the coal mine drone inspection can be enhanced.
[0041] In this embodiment, the training set includes 10,000 clear pictures taken by coal mine drones and blurred pictures obtained by adding noise to the 10,000 clear pictures.
[0042] In step S1 of this embodiment, the processing of the acquired image includes the following steps:
[0043] Step S11, according to the acquired image, acquiring a blurred image corresponding to the corresponding clear image by adding noise;
[0044] Step S12: setting the pixel sizes of the corresponding clear image and the corresponding blurred image to form a single element of the training set.
[0045] In this embodiment, through the processing of step S11, the clear image that has been acquired can be added with random noise to obtain a blurred image corresponding to the corresponding clear image, so that the training set can be better obtained; through the processing of step S12, all images can be processed into the same pixel size, so that subsequent image processing can be better facilitated.
[0046] In this embodiment, in step S12, the pixel size of the image is processed to 64×64.
[0047] Combination Figure 3 As shown, in step S2 of this embodiment, the generator adopts a convolutional neural network, including a linear mapping layer and four convolution layers. The size of the convolution kernel of each convolution layer is 5×5. The linear mapping and the first three convolution layers are respectively connected to batch normalization and ReLU activation functions, and the last convolution layer is connected to a Tanh activation function.
[0048] In the generator of this embodiment, the linear mapping layer adopts the Linear function, the convolution layer adopts the Deconv function, and the batch normalization function is the BN function.
[0049] Combination Figure 4 As shown in the figure, the discriminator adopts a convolutional neural network, including four convolutional layers and one linear mapping layer. The size of the convolution kernel of each convolutional layer is 5×5. The first convolutional layer is connected to the LReLU activation function, and the following three convolutional layers are respectively connected to batch normalization and LReLU activation function.
[0050] In the discriminator of this embodiment, the linear mapping layer adopts the Linear function, the convolution layer adopts the Conv function, and the batch normalization function is the BN function.
[0051] In step S3 of this embodiment, an error back propagation algorithm is used to fix the parameters of the discriminator and minimize the loss function of the generator, thereby enabling better training of the generator.
[0052] In step S4 of this embodiment, an error back propagation algorithm is used to fix the parameters of the generator and minimize the loss function of the discriminator, thereby enabling better training of the discriminator.
[0053] In this embodiment, the loss function of the generator is Where G is the generator model, D is the discriminator model, z is random noise, and x is the blurred image.
[0054] In this embodiment, the loss function of the discriminator is Where G is the generator model, D is the discriminator model, x is the blurred image, and y is the clear image.
[0055] In this embodiment, a mean square error penalty is introduced into the loss function of the generator to make the generated image closer to the corresponding clear image. The error penalty function is p G =||G(z|x)-y|| 2 . Among them, G is the generator model, z is random noise, x is the blurred image, and y is the clear image.
[0056] In this embodiment, a gradient penalty is introduced into the loss function of the discriminator to stabilize the training process of the generative adversarial network. The gradient penalty function is Where D is the discriminator model, x is the blurred image, To generate the image, is the gradient value of the discriminator model, and α is a random value between [0,1].
[0057] In this embodiment, by introducing the mean square error penalty and gradient penalty function in the training process of the generative adversarial network, the model after the final training is able to generate an enhanced image with richer details when processing a newly collected image, in response to problems such as low brightness, low contrast and image blur in the captured image.
[0058] In step S6 of this embodiment, the processing of the blurred image taken by the drone is mainly to crop the image size to a predetermined pixel size, which is 64×64 pixel size in this embodiment.
[0059] More specifically, the parameters of the generative adversarial network of this embodiment are set as follows: the batch size during training is 32, the convolution kernel size of the generator and the discriminator is 5×5, and the step size is 2. The Adam optimizer is used for the training process of the generator and the discriminator, the learning rate of the generator is 0.0001, the learning rate of the discriminator is 0.0005, and the dimension of the noise is 100.
[0060] Combination Figure 5 As shown in the figure, it is a rendering of the image enhancement method of this embodiment. The left picture is the original blurred image taken by the drone, and the right picture is the clear image after the generative adversarial network image enhancement. It can be seen from the figure that the image enhancement method based on the generative adversarial network of the present invention has a good effect in terms of image clarity and authenticity.
[0061] The present invention and its embodiments are described schematically above, and the description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by it and designs a structural method and an embodiment similar to the technical solution without creativity without departing from the purpose of the invention, they shall all fall within the protection scope of the present invention.
Claims
1. A coal mine drone image enhancement method based on a generative adversarial network, comprising the following steps: Step S1, obtaining a clear image taken by a coal mine drone and processing the obtained image, thereby constructing a training set, wherein a single element of the training set is a two-tuple, and the two-tuple includes a clear image and a blurred image corresponding to the clear image; Step S2, constructing a generative adversarial network, the constructed generative adversarial network includes a generator and a discriminator; Step S3, fixing the parameters of the discriminator, inputting the blurred image in the training set into the generator, and using the discriminator to determine whether the image generated by the generator is clear, thereby training the generator; Step S4, fixing the parameters of the generator, inputting the clear images in the training set and the images generated by the generator into the discriminator respectively, to train the discriminator; Step S5, repeating steps S3 and S4, and updating the parameters of the generator and the discriminator according to the loss function, until the discriminator identifies the image generated by the generator as a clear image, and the model training is completed; Step S6, after processing, the blurred image taken by the drone is input into the trained generator, and the corresponding image output by the generator is used as the enhanced clear image; In step S3, the error back propagation algorithm is used to fix the parameters of the discriminator, minimize the loss function of the generator, and introduce the mean square error penalty into the loss function of the generator. The error penalty function is p G =||G(z|x)-y|| 2 Among them, G is the generator model, z is random noise, x is the blurred image, and y is the clear image; In step S4, the error back propagation algorithm is used to fix the parameters of the generator, minimize the loss function of the discriminator, and introduce a gradient penalty into the loss function of the discriminator. The gradient penalty function is Where D is the discriminator model, x is the blurred image, To generate the image, is the gradient value of the discriminator model, and α is a random value between [0,1].
2. According to the coal mine drone image enhancement method based on generative adversarial network described in claim 1, Features: In step S1, the processing of the acquired image includes the following steps: Step S11, according to the acquired image, acquiring a blurred image corresponding to the corresponding clear image by adding noise; Step S12, setting the pixel sizes of the corresponding clear image and the corresponding blurred image, thereby forming a single element of the training set.
3. According to the coal mine drone image enhancement method based on generative adversarial network in claim 1, Features: In step S2, the generator uses a convolutional neural network, including a linear mapping layer and four convolution layers. The size of the convolution kernel of each convolution layer is 5×5. The linear mapping and the first three convolution layers are connected to batch normalization and ReLU activation functions respectively, and the last convolution layer is connected to the Tanh activation function.
4. According to the coal mine drone image enhancement method based on generative adversarial network in claim 1, Features: The discriminator uses a convolutional neural network, which includes four convolutional layers and one linear mapping layer. The size of the convolution kernel of each convolutional layer is 5×5. The first convolutional layer is connected to the LReLU activation function, and the following three convolutional layers are connected to batch normalization and LReLU activation functions respectively.
5. According to the coal mine drone image enhancement method based on generative adversarial network in claim 1, Features: The loss function of the generator is Where G is the generator model, D is the discriminator model, z is random noise, and x is the blurred image.
6. According to the coal mine drone image enhancement method based on generative adversarial network in claim 1, Features: The loss function of the discriminator is Where G is the generator model, D is the discriminator model, x is the blurred image, and y is the clear image.