Image rain removing method and device for unmanned aerial vehicle power grid inspection

Abstract

The invention discloses an image rain removal method and device for unmanned aerial vehicle power grid inspection, and belongs to the technical field of unmanned aerial vehicle aerial image rain removal. The image rain removal method provided by the invention comprises the following steps: acquiring an aerial image acquired by an unmanned aerial vehicle, and performing step-by-step information mining operation on the aerial image to obtain a de-noised sub-image; the at least one level of information mining operation process comprises a spatial dimension reduction operation and a spatial dimension expansion operation which are performed in sequence; and based on the de-noised sub-image, constructing a target image after rain stripe removal, thereby completing image rain removal operation. In the feature information mining process of the aerial image, the spatial dimension reduction operation is firstly carried out, the content of rain stripe noise is greatly compressed, meanwhile, the influence on other objects in the image is relatively small, then the spatial dimension expansion operation is carried out, the sparsity and the dispersion degree of residual rain stripe information are improved, rain stripes are efficiently removed, and meanwhile the feature information mining efficiency is improved. And other image features are well reserved.

Description

Technical Field

[0001] The present invention belongs to the technical field of rain removal for unmanned aerial vehicle (UAV) aerial images, and in particular relates to an image rain removal method and device for unmanned aerial vehicle (UAV) power grid inspection. Background Art

[0002] Compound wing UAVs combine the features of fixed-wing UAVs and rotary-wing UAVs. They can cruise at high speeds like fixed-wing UAVs and take off and land vertically like rotary-wing UAVs. In the field of power grid inspection, compound wing UAVs are often needed to capture high-definition orthographic projection images of target areas. However, when UAVs operate on rainy days, the images will contain a large number of rain streaks. On the other hand, since compound wing UAVs fly at a high altitude and high speed, many inspection objects cover a small area in the image. When the existing algorithms are applied to this scenario, they cannot achieve a good rain removal effect. Summary of the invention

[0003] In view of this, the present invention provides an image deraining method and device for unmanned aerial vehicle power grid inspection to solve the above technical problems.

[0004] The technical solution is as follows: An image deraining method for UAV power grid inspection includes the following steps:

[0005] Acquire an aerial image collected by a drone, perform information mining operations on the aerial image step by step, and obtain a denoised sub-image; in at least one level of the information mining operation process, a spatial dimension reduction operation and a spatial dimension expansion operation are successively performed;

[0006] A target image after rain streaks are removed is constructed based on the denoising sub-image, thereby completing the image deraining operation.

[0007] Furthermore, multiple feature mining mechanisms arranged in series are used to perform information mining operations on the aerial images step by step.

[0008] Furthermore, the feature mining mechanism is internally provided with a space reduction unit, a space expansion unit and a calibration unit: the internal operation process of the feature mining mechanism includes:

[0009] Using a spatial reduction unit to perform a spatial dimension reduction operation on the received sub-graph to generate a first adapted sub-graph;

[0010] Using a spatial expansion unit to perform a spatial dimension expansion operation on the first adapted subgraph to generate a second adapted subgraph;

[0011] The calibration unit takes the first adaptation subgraph and the second adaptation subgraph as inputs and generates a calibration subgraph through calculation;

[0012] The proofreading subgraph is merged with the second adaptation subgraph to generate an optimized subgraph.

[0013] Furthermore, the denoised subgraph is an optimized subgraph generated by the last operation of the feature mining mechanism.

[0014] Furthermore, the internal operation process of the space reduction unit includes:

[0015] Using a maximum pooling layer and a first activation function to sequentially process the sub-images received by the spatial reduction unit to generate a first reduced sub-image;

[0016] Processing the first reduced sub-graph in sequence using a first convolutional layer and a second activation function to generate a second reduced sub-graph;

[0017] Using an average pooling layer and a third activation function to sequentially process the sub-images received by the spatial reduction unit to generate a third reduced sub-image;

[0018] Subtracting the third reduced sub-image from the second reduced sub-image to generate a fourth reduced sub-image;

[0019] Processing the fourth reduced sub-graph in sequence using the second convolutional layer and the fourth activation function to generate a fifth reduced sub-graph;

[0020] Performing a Hadamard product on the first reduced subgraph and the fifth reduced subgraph to obtain the first adapted subgraph.

[0021] After adopting the above scheme, a feature mismatch self-adjustment mechanism is constructed inside the spatial reduction unit. Through the cooperation of the maximum pooling layer, the average pooling layer, the first convolution layer, the second convolution layer and four activation functions, during the spatial dimension reduction operation, targeted and selective information screening can be performed according to the characteristics of the image. Compared with the existing dimensionality reduction operations, it has better flexibility and more accurate denoising capabilities.

[0022] Furthermore, the spatial expansion unit includes a deconvolution layer and a fifth activation function which are arranged in sequence.

[0023] Furthermore, the internal calculation process of the calibration unit includes:

[0024] Performing full-channel average pooling processing and full-channel maximum pooling processing on the first adaptation subgraph respectively, and generating a first matrix and a second matrix correspondingly;

[0025] Performing a first deconvolution process on the first matrix to generate a third matrix, and performing a second deconvolution process on the second matrix to generate a fourth matrix;

[0026] Performing full-channel maximum pooling and full-channel average pooling processing on the second adapted subgraph respectively, and generating a fifth matrix and a sixth matrix correspondingly;

[0027] Subtracting the fifth matrix from the fourth matrix to generate a seventh matrix, and adding the third matrix to the sixth matrix to generate an eighth matrix;

[0028] splicing the fifth matrix, the sixth matrix, the seventh matrix and the eighth matrix together to generate a combined matrix;

[0029] A first convolution calculation and a first activation calculation are sequentially performed on the combined matrix to obtain the calibration subgraph.

[0030] After adopting the above scheme, the calibration unit takes the first adapted sub-graph and the second adapted sub-graph as input, maps the relationship before and after the spatial dimension expansion through the internal decomposition and transformation mechanism, and fuses the generated calibration sub-graph with the second adapted sub-graph, so that the algorithm can better learn various complex image feature patterns and adaptively expand them to the area originally covered by rain streaks.

[0031] Furthermore, the internal operation process of the calibration unit also includes: performing Hadamard product of the third matrix and the fourth matrix to generate a ninth matrix, and the ninth matrix generates an additional subgraph after being activated by a sigmoid function.

[0032] Furthermore, the process of constructing a target image based on the denoised sub-image includes:

[0033] The additional sub-graphs generated in all the proofreading units are concatenated to generate a first transition sub-graph;

[0034] The first transition subgraph is sequentially subjected to a second convolution calculation and a second activation calculation to generate a second transition subgraph;

[0035] The denoised sub-image is merged with the second transition sub-image to generate a third transition sub-image;

[0036] The third transition sub-image is sequentially subjected to a third convolution calculation and a third activation calculation to generate the target image.

[0037] After adopting the above scheme, the second transition sub-image generated by the additional sub-image is fused with the denoising sub-image to generate the target image. Since the information of the additional sub-image comes from the first adaptation sub-image before the spatial expansion operation, this can greatly reduce the probability of image distortion and mutual interference in the target image caused by multiple superimposed spatial dimension expansion operations, and improve the stability of the quality of the generated target image.

[0038] The present invention also provides an image deraining device for unmanned aerial vehicle power grid inspection, including a computer program / instruction. When the computer program / instruction is executed, the method as described above is implemented.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] In the process of mining feature information of aerial images, the present invention first performs a spatial dimension reduction operation. Since the coverage area of ​​a single rain streak in the image is much smaller than that of other objects, the spatial dimension reduction operation can greatly compress the content of rain streak noise, while having relatively little impact on other objects in the image. Then, a spatial dimension expansion operation is performed to further improve the sparsity and discreteness of the residual rain streak information by fitting and expanding the useful information. While efficiently removing the rain streaks, other image features are well preserved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The figure is a flow chart of the image deraining method for UAV power grid inspection of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0043] This embodiment provides an image deraining method for UAV power grid inspection. Figure 1 As shown, the following steps are included:

[0044] An aerial image collected by a drone on a rainy day is obtained, and the aerial image contains rain streaks. The aerial image is subjected to information mining operations step by step to obtain a denoised subgraph. In this embodiment, a conventional convolution layer is first used to perform convolution processing on the aerial image to generate a shallow information subgraph to implement primary information mining operations, and then four feature mining mechanisms arranged in series are used to perform information mining operations on the shallow information subgraph step by step, and one feature mining mechanism implements primary information mining operations.

[0045] Specifically, the feature mining mechanism is internally provided with a space reduction unit, a space expansion unit and a calibration unit: the internal operation process of the feature mining mechanism includes:

[0046] The spatial dimension reduction operation is performed on the received subgraph by the spatial reduction unit to generate a first adapted subgraph; the spatial dimension expansion operation is performed on the first adapted subgraph by the spatial expansion unit to generate a second adapted subgraph, and the size of the second adapted subgraph is equal to the size of the subgraph received by the spatial reduction unit; the calibration unit takes the first adapted subgraph and the second adapted subgraph as inputs and generates a calibration subgraph after calculation; the calibration subgraph is fused with the second adapted subgraph to generate an optimized subgraph, and the optimized subgraph is used as the output of the feature mining mechanism. For all feature mining mechanisms, the size of the output optimized subgraph is the same.

[0047] In this embodiment, the calibration subgraph is first broadcasted in the channel direction to make the size of the calibration subgraph equal to that of the second adaptation subgraph, and then the calibration subgraph and the second adaptation subgraph are fused by multiplying the corresponding elements. The shallow information subgraph is used as the input of the first feature mining mechanism, and the subsequent feature mining mechanisms all use the optimized subgraph generated by the previous feature mining mechanism as input. The above-mentioned denoising subgraph is the optimized subgraph generated by the last feature mining mechanism.

[0048] The space reduction unit can be implemented by using some existing dimensionality reduction algorithms. In some implementations, the subgraph of the input feature mining mechanism is directly used as the input of the space reduction unit inside the feature mining mechanism. The internal operation process of the space reduction unit includes:

[0049] The sub-image received by the space reduction unit is processed in sequence by using the maximum pooling layer and the first activation function to generate a first reduced sub-image; the first reduced sub-image is processed in sequence by using the first convolution layer and the second activation function to generate a second reduced sub-image; the sub-image received by the space reduction unit is processed in sequence by using the average pooling layer and the third activation function to generate a third reduced sub-image; the third reduced sub-image is subtracted from the second reduced sub-image to generate a fourth reduced sub-image; the fourth reduced sub-image is processed in sequence by using the second convolution layer and the fourth activation function to generate a fifth reduced sub-image; the first reduced sub-image is Hadamard-multiplied with the fifth reduced sub-image to obtain a first adapted sub-image.

[0050] The spatial expansion unit can be implemented by using some existing dimensionality increase algorithms. In some implementations, the spatial expansion unit includes a deconvolution layer and a fifth activation function arranged in sequence.

[0051] The pooling window sizes of the above-mentioned maximum pooling layer and average pooling layer are equal, and the step sizes are also equal. The pooling window sizes can be 2*2, 3*3 or 4*4, etc., and the sliding step sizes can be 2 or 3 or 4, etc. During the pooling operation, the pooling window slides along the spatial direction of the sub-image. The first activation function, the second activation function, the third activation function, the fourth activation function and the fifth activation function can all independently select existing nonlinear functions such as ReLU, logistics or Tanh to implement. The step sizes of the first convolutional layer and the second convolutional layer are both 1, and their convolution kernel sizes can usually be 3*3 or 5*5, etc.

[0052] In some implementations, the internal calculation process of the calibration unit includes:

[0053] The first adapted sub-graph is subjected to full-channel average pooling and full-channel maximum pooling, respectively, to generate a first matrix and a second matrix. Full-channel pooling is an existing technology, which can also be called global pooling. Full-channel pooling can compress the channel size of the sub-graph to 1, while the height and width remain unchanged; full-channel average pooling is to calculate the average value of all eigenvalues ​​of the entire channel, and full-channel maximum pooling is to calculate the maximum value of all eigenvalues ​​of the entire channel. The first matrix is ​​subjected to a first deconvolution process to generate a third matrix, and the second matrix is ​​subjected to a second deconvolution process to generate a fourth matrix. The second adapted sub-graph is subjected to full-channel maximum pooling and full-channel average pooling, respectively, to generate a fifth matrix and a sixth matrix; the sizes of the third matrix, the fourth matrix, the fifth matrix, and the sixth matrix are all equal. The fifth matrix is ​​subtracted from the fourth matrix to generate a seventh matrix, and the third matrix is ​​added to the sixth matrix to generate an eighth matrix. The fifth matrix, the sixth matrix, the seventh matrix, and the eighth matrix are concatenated to generate a combined matrix; since the combined matrix is ​​concatenated from four matrices, the channel size of the combined matrix is ​​4. The first convolution calculation and the first activation calculation are sequentially performed on the combined matrix to obtain a calibration subgraph, and the channel size of the calibration subgraph is 1. As an example, the step size of the first convolution calculation may be 1, and the first activation calculation may be completed using a sigmoid function.

[0054] In some implementations, the process of constructing a target image based on a denoised sub-image includes: the denoised sub-image is sequentially subjected to a fourth convolution calculation and a fourth activation calculation to generate a target image, and the template image channel size is 3. As an example, the step size of the fourth convolution calculation can be 1, and the fourth activation calculation can be performed using a ReLU function.

[0055] In some other embodiments, the internal calculation process of the calibration unit further includes: performing Hadamard product of the third matrix and the fourth matrix to generate a ninth matrix, and the ninth matrix generates an additional sub-image after being activated by a sigmoid function. At this time, the process of constructing the target image based on the denoised sub-image includes:

[0056] The additional sub-graphs generated in all the calibration units are spliced ​​together to generate a first transition sub-graph. In this embodiment, the channel size of the first transition sub-graph is 4. The first transition sub-graph is sequentially subjected to the second convolution calculation and the second activation calculation to generate a second transition sub-graph. The channel size of the second transition sub-graph is 1. The denoising sub-graph is fused with the second transition sub-graph to generate a third transition sub-graph. Specifically, the second transition sub-graph is first broadcasted in the channel direction to make the size of the second transition sub-graph equal to the size of the denoising sub-graph. Then, the second transition sub-graph and the denoising sub-graph are fused by multiplying the corresponding elements to generate the third transition sub-graph. The third transition sub-graph is sequentially subjected to the third convolution calculation and the third activation calculation to generate a target image. The channel size of the target image is 3. As an example, the step size of the second convolution calculation and the third convolution calculation can both be 1, and the second activation calculation and the third activation calculation can both be completed using the ReLU function.

[0057] Finally, it should be noted that the above description is only a preferred embodiment of the present invention. Under the guidance of the present invention, ordinary technicians in this field can make various similar expressions without violating the purpose and claims of the present invention, and such changes all fall within the scope of protection of the present invention.

Claims

1. An image deraining method for UAV power grid inspection, characterized in that: The following steps are involved: Acquire an aerial image collected by a drone, perform information mining operations on the aerial image step by step, and obtain a denoised sub-image; in at least one level of the information mining operation process, a spatial dimension reduction operation and a spatial dimension expansion operation are successively performed; A target image is constructed based on the denoising sub-image, thereby completing the image deraining operation.

2. The image deraining method for UAV power grid inspection according to claim 1 is characterized in that: A plurality of feature mining mechanisms arranged in series are used to perform information mining operations on the aerial images step by step.

3. The image deraining method for UAV power grid inspection according to claim 2 is characterized in that: The feature mining mechanism is internally provided with a space reduction unit, a space expansion unit and a calibration unit. The internal operation process of the feature mining mechanism includes: Using a spatial reduction unit to perform a spatial dimension reduction operation on the received sub-graph to generate a first adapted sub-graph; Using a spatial expansion unit to perform a spatial dimension expansion operation on the first adapted subgraph to generate a second adapted subgraph; The calibration unit takes the first adaptation subgraph and the second adaptation subgraph as inputs and generates a calibration subgraph through calculation; The proofreading subgraph is merged with the second adaptation subgraph to generate an optimized subgraph.

4. The image deraining method for UAV power grid inspection according to claim 3 is characterized in that: The denoised subgraph is the last optimized subgraph generated by the feature mining mechanism.

5. The image deraining method for UAV power grid inspection according to claim 3 is characterized in that: The internal operation process of the space reduction unit includes: Using a maximum pooling layer and a first activation function to sequentially process the sub-images received by the spatial reduction unit to generate a first reduced sub-image; Processing the first reduced sub-graph in sequence using a first convolutional layer and a second activation function to generate a second reduced sub-graph; Using an average pooling layer and a third activation function to sequentially process the sub-images received by the spatial reduction unit to generate a third reduced sub-image; Subtracting the third reduced sub-image from the second reduced sub-image to generate a fourth reduced sub-image; Processing the fourth reduced sub-graph in sequence using the second convolutional layer and the fourth activation function to generate a fifth reduced sub-graph; Performing a Hadamard product on the first reduced subgraph and the fifth reduced subgraph to obtain the first adapted subgraph.

6. The image deraining method for UAV power grid inspection according to claim 3 is characterized in that: The spatial expansion unit includes a deconvolution layer and a fifth activation function which are arranged in sequence.

7. The image deraining method for UAV power grid inspection according to claim 4 is characterized in that: The internal calculation process of the calibration unit includes: Performing full-channel average pooling processing and full-channel maximum pooling processing on the first adaptation subgraph respectively, and generating a first matrix and a second matrix correspondingly; Performing a first deconvolution process on the first matrix to generate a third matrix, and performing a second deconvolution process on the second matrix to generate a fourth matrix; Performing full-channel maximum pooling and full-channel average pooling processing on the second adapted subgraph respectively, and generating a fifth matrix and a sixth matrix correspondingly; Subtracting the fifth matrix from the fourth matrix to generate a seventh matrix, and adding the third matrix to the sixth matrix to generate an eighth matrix; splicing the fifth matrix, the sixth matrix, the seventh matrix and the eighth matrix together to generate a combined matrix; A first convolution calculation and a first activation calculation are sequentially performed on the combined matrix to obtain the calibration subgraph.

8. The image deraining method for UAV power grid inspection according to claim 7 is characterized in that: The internal calculation process of the calibration unit also includes: performing Hadamard product on the third matrix and the fourth matrix to generate a ninth matrix, and the ninth matrix generates an additional subgraph after being activated by a sigmoid function.

9. The image deraining method for UAV power grid inspection according to claim 8 is characterized in that: The process of constructing a target image based on the denoised sub-image includes: The additional sub-graphs generated in all the proofreading units are concatenated to generate a first transition sub-graph; The first transition subgraph is sequentially subjected to a second convolution calculation and a second activation calculation to generate a second transition subgraph; The denoised sub-image is merged with the second transition sub-image to generate a third transition sub-image; The third transition sub-image is sequentially subjected to a third convolution calculation and a third activation calculation to generate the target image.

10. Image deraining equipment for drone power grid inspection, characterized in that: The method comprises a computer program / instruction, and when the computer program / instruction is executed, the method according to any one of claims 1 to 9 is implemented.

Images