Image deraining method and device for unmanned aerial vehicle power grid inspection

Abstract

The application discloses an image rain removing method and device for unmanned aerial vehicle power grid inspection, and belongs to the technical field of unmanned aerial vehicle aerial image rain removing.The image rain removing method provided by the application comprises the following steps: acquiring an aerial image collected by an unmanned aerial vehicle, performing step-by-step information mining operation on the aerial image to obtain a denoising sub-image; in at least one information mining operation process, a spatial dimension reduction operation and a spatial dimension expansion operation are performed in sequence; and a target image with rain streaks removed is constructed based on the denoising sub-image, so that the image rain removing operation is completed.In the feature information mining process of the aerial image, the spatial dimension reduction operation is performed first, the content of rain streak noise is greatly compressed, and the influence on other objects in the image is relatively small, then the spatial dimension expansion operation is performed, the sparseness and the discrete degree of residual rain streak information are improved, the rain streaks are efficiently removed, and other image features are well reserved.

Description

Technical Field

[0001] This invention belongs to the field of drone aerial image deraining technology, specifically relating to a method and device for deraining images used in drone power grid inspection. Background Technology

[0002] Compound-wing UAVs combine the characteristics of fixed-wing and rotary-wing UAVs, enabling them to cruise at high speeds like fixed-wing UAVs and take off and land vertically like rotary-wing UAVs. In power grid inspection, compound-wing UAVs are often used to capture high-resolution orthophotos of target areas. However, when operating in rainy weather, the images contain numerous rain streaks. Furthermore, due to the high altitude and speed of compound-wing UAVs, many inspection targets cover a small area in the image, and existing algorithms are not yet effective at removing rain in this scenario. Summary of the Invention

[0003] In view of this, the present invention provides an image deraining method and device for UAV power grid inspection to solve the above-mentioned technical problems.

[0004] The technical solution is as follows: an image de-raining method for UAV power grid inspection, comprising the following steps:

[0005] Aerial images captured by a drone are acquired, and information mining operations are performed on the aerial images step by step to obtain denoised sub-images; in at least one level of information mining operations, spatial dimension reduction and spatial dimension expansion operations are performed sequentially.

[0006] Based on the denoised sub-image, a target image with rain patterns removed is constructed, thereby completing the image deraining operation.

[0007] Furthermore, multiple feature mining mechanisms arranged in series are used to perform step-by-step information mining operations on the aerial images.

[0008] Furthermore, the feature mining mechanism internally includes a space reduction unit, a space expansion unit, and a correction unit; the internal calculation process of the feature mining mechanism includes:

[0009] The spatial dimension of the received subgraph is reduced using the spatial reduction unit to generate the first adapted subgraph;

[0010] The first adapt subgraph is expanded in spatial dimension using spatial expansion units to generate a second adapt subgraph.

[0011] The calibration unit takes the first adaptation subgraph and the second adaptation subgraph as input and generates a calibration subgraph through calculation;

[0012] The calibration subgraph is merged with the second adaptation subgraph to generate an optimized subgraph.

[0013] Furthermore, the denoised subgraph is the optimized subgraph generated by the last feature mining operation.

[0014] Furthermore, the internal computational process of the space reduction unit includes:

[0015] The subgraph received by the space reduction unit is processed sequentially using the max pooling layer and the first activation function to generate the first reduced subgraph;

[0016] The first reduced subgraph is processed sequentially using the first convolutional layer and the second activation function to generate the second reduced subgraph.

[0017] The subgraph received by the spatial reduction unit is processed sequentially using the average pooling layer and the third activation function to generate the third reduced subgraph;

[0018] Subtract the third reduced subgraph from the second reduced subgraph to generate the fourth reduced subgraph;

[0019] The fourth reduced subgraph is processed sequentially using the second convolutional layer and the fourth activation function to generate the fifth reduced subgraph.

[0020] The first reduced subgraph and the fifth reduced subgraph are multiplied by Hadamard to obtain the first adapted subgraph.

[0021] After adopting the above scheme, a feature mismatch self-adjustment mechanism is constructed inside the spatial reduction unit. Through the cooperation of max pooling layer, average pooling layer, first convolutional layer, second convolutional layer and four activation functions, the spatial dimension reduction operation can perform targeted and selective information filtering according to the features of the image. Compared with the existing dimensionality reduction operation, it has better flexibility and more accurate noise reduction capability.

[0022] Furthermore, the spatial expansion unit includes a deconvolution layer and a fifth activation function arranged sequentially.

[0023] Furthermore, the internal calculation process of the calibration unit includes:

[0024] The first adaptation subgraph is subjected to full-channel average pooling and full-channel max pooling respectively, resulting in a first matrix and a second matrix.

[0025] Perform a first deconvolution on the first matrix to generate a third matrix, and perform a second deconvolution on the second matrix to generate a fourth matrix;

[0026] The second adaptation subgraph is subjected to full-channel max pooling and full-channel average pooling respectively, resulting in the fifth and sixth matrices.

[0027] Subtracting the fifth matrix from the fourth matrix produces the seventh matrix; adding the third matrix to the sixth matrix produces the eighth matrix.

[0028] The fifth, sixth, seventh, and eighth matrices are concatenated to generate a combined matrix;

[0029] The combined matrix is ​​subjected to a first convolution calculation and a first activation calculation in sequence to obtain the collation primer diagram.

[0030] After adopting the above scheme, the correction unit takes the first and second adaptation subgraphs as inputs, maps the relationship before and after the spatial dimension expansion through the internal decomposition and transformation mechanism, and merges the generated correction subgraph with the second adaptation subgraph, so that the algorithm can better learn various complex image feature patterns and adapt them to the area originally covered by rain patterns.

[0031] Furthermore, the internal operation process of the calibration unit also includes: performing a Hadamard product between the third matrix and the fourth matrix to generate a ninth matrix, and generating an additional subgraph after the ninth matrix is ​​activated by the sigmoid function.

[0032] Furthermore, the process of constructing the target image based on the denoised sub-image includes:

[0033] The additional subgraphs generated in all the calibration units are stitched together to produce the first transition subgraph;

[0034] The first transition subgraph is sequentially processed by the second convolution calculation and the second activation calculation to generate the second transition subgraph.

[0035] The denoised subgraph is merged with the second transition subgraph to generate a third transition subgraph;

[0036] The third transition subgraph is sequentially processed through a third convolution calculation and a third activation calculation to generate the target image.

[0037] After adopting the above scheme, the second transition sub-image generated by the additional sub-image is fused with the denoised sub-image to generate the target image. Since the information of the additional sub-image comes from the first adaptation sub-image before the spatial expansion operation, this can greatly reduce the probability of image distortion and mutual interference caused by multiple superimposed spatial dimension expansion operations, and improve the stability of the generated target image quality.

[0038] The present invention also provides an image de-raining device for UAV power grid inspection, comprising a computer program / instructions, which, when executed, implements the method described above.

[0039] Compared with the prior art, the beneficial effects of the present invention are:

[0040] In the process of feature information mining of aerial images, this invention first performs a spatial dimension reduction operation. Since the coverage area of ​​a single rain streak in the image is much smaller than that of other objects, the spatial dimension reduction operation can significantly compress the noise content of the rain streak while having a relatively small impact on other objects in the image. Then, a spatial dimension expansion operation is performed. By fitting and expanding the useful information, the sparsity and dispersion of the residual rain streak information are further improved, which efficiently removes the rain streak while well preserving other image features. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the image deraining method for UAV power grid inspection according to the present invention. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the accompanying drawings.

[0043] This embodiment provides an image deraining method for UAV power grid inspection, such as... Figure 1 As shown, it includes the following steps:

[0044] Aerial images captured by a drone in rainy weather are acquired, containing rain streaks. The aerial images undergo progressive information mining operations to obtain denoised sub-images. In this embodiment, a conventional convolutional layer is first used to convolve the aerial images, generating shallow information sub-images for primary information mining. Then, four cascaded feature mining mechanisms are used to perform progressive information mining operations on the shallow information sub-images, with each feature mining mechanism performing a first-level information mining operation.

[0045] Specifically, the feature mining mechanism includes a space reduction unit, a space expansion unit, and a correction unit. The internal computation process of the feature mining mechanism includes:

[0046] The received subgraph is reduced in spatial dimension using a spatial reduction unit to generate a first adapted subgraph. The first adapted subgraph is then expanded in spatial dimension using a spatial expansion unit to generate a second adapted subgraph, the size of which is equal to the size of the subgraph received by the spatial reduction unit. A correction unit takes the first and second adapted subgraphs as input and calculates to generate a correction subgraph. The correction subgraph is then merged with the second adapted subgraph to generate an optimized subgraph, which serves as the output of this feature mining mechanism. The size of the optimized subgraph is the same for all feature mining mechanisms.

[0047] In this embodiment, the calibration sub-image is first broadcast along the channel direction to make its size equal to that of the second adaptation sub-image. Then, the calibration sub-image and the second adaptation sub-image are fused by multiplying their corresponding elements. The shallow information sub-image serves as the input to the first feature mining mechanism. For subsequent feature mining mechanisms, the optimized sub-image generated by the previous feature mining mechanism is used as the input. The aforementioned denoised sub-image is the optimized sub-image generated by the last feature mining mechanism.

[0048] The spatial reduction unit can be implemented using some existing dimensionality reduction algorithms. In some implementations, the subgraph of the input feature mining mechanism is directly used as the input of the spatial reduction unit inside the feature mining mechanism. The internal operation process of the spatial reduction unit includes:

[0049] The first reduced subgraph is generated by processing the subgraph received by the spatial reduction unit sequentially using a max pooling layer and a first activation function. The second reduced subgraph is generated by processing the first reduced subgraph sequentially using a first convolutional layer and a second activation function. The third reduced subgraph is generated by processing the subgraph received by the spatial reduction unit sequentially using an average pooling layer and a third activation function. The second reduced subgraph is subtracted from the third reduced subgraph to generate a fourth reduced subgraph. The fourth reduced subgraph is generated by processing the second convolutional layer and a fourth activation function. The fifth reduced subgraph is generated by performing a Hadamard product on the first and fifth reduced subgraphs to obtain the first adapted subgraph.

[0050] The spatial expansion unit can be implemented using some existing dimensionality-upgrading algorithms. In some implementations, the spatial expansion unit includes a deconvolution layer and a fifth activation function arranged sequentially.

[0051] The max pooling and average pooling layers mentioned above have equal pooling window sizes and strides. The pooling window sizes can be 2x2, 3x3, or 4x4, and the stride can be 2, 3, or 4. During pooling, the pooling window slides along the spatial direction of the subgraph. The first, second, third, fourth, and fifth activation functions can each be implemented independently using existing nonlinear functions such as ReLU, logistic, or Tanh. The stride of the first and second convolutional layers is 1, and their kernel sizes are typically 3x3 or 5x5.

[0052] In some implementations, the internal calculation process of the calibration unit includes:

[0053] The first fitting subimage is subjected to full-channel average pooling and full-channel max pooling, respectively, generating the first and second matrices. Full-channel pooling is an existing technique, also known as global pooling. Full-channel pooling compresses the channel size of the subimage to 1, while keeping the height and width unchanged. Full-channel average pooling calculates the average of all feature values ​​in the entire channel, while full-channel max pooling calculates the maximum of all feature values ​​in the entire channel. The first matrix is ​​subjected to a first deconvolution, generating the third matrix, and the second matrix is ​​subjected to a second deconvolution, generating the fourth matrix. The second fitting subimage is subjected to full-channel max pooling and full-channel average pooling, respectively, generating the fifth and sixth matrices; the third, fourth, fifth, and sixth matrices are all of equal size. The fifth matrix is ​​subtracted from the fourth matrix to generate the seventh matrix, and the third and sixth matrices are added to generate the eighth matrix. The fifth, sixth, seventh, and eighth matrices are concatenated to generate the combined matrix; since the combined matrix is ​​composed of four matrices, its channel size is 4. The combined matrix is ​​subjected to the first convolution and the first activation calculations sequentially to obtain the corrected guide map, which has a channel size of 1. As an example, the stride of the first convolution calculation can be 1, and the first activation calculation can be performed using the sigmoid function.

[0054] In some implementations, the process of constructing a target image based on a denoised subimage includes: the denoised subimage sequentially undergoes a fourth convolution calculation and a fourth activation calculation to generate the target image, where the template image has a channel size of 3. As an example, the stride of the fourth convolution calculation can be 1, and the fourth activation calculation can be performed using the ReLU function.

[0055] In other implementations, the internal calculation process of the correction unit further includes: performing a Hadamard product between the third and fourth matrices to generate a ninth matrix; and activating the ninth matrix using a sigmoid function to generate an additional sub-image. The process of constructing the target image based on the denoised sub-image then includes:

[0056] The additional sub-images generated in all the calibration units are stitched together to produce a first transition sub-image, which in this embodiment has a channel size of 4. The first transition sub-image undergoes a second convolution calculation and a second activation calculation to produce a second transition sub-image, which has a channel size of 1. The denoised sub-image is then fused with the second transition sub-image to produce a third transition sub-image. Specifically, the second transition sub-image is first broadcast in the channel direction to make its size equal to that of the denoised sub-image. Then, the second transition sub-image and the denoised sub-image are fused by element-wise multiplication to produce the third transition sub-image. The third transition sub-image undergoes a third convolution calculation and a third activation calculation to produce the target image, which has 3 channels. As an example, the stride of both the second and third convolution calculations can be 1, and both the second and third activation calculations can be performed using the ReLU function.

[0057] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention. Those skilled in the art, under the guidance of the present invention, can make various similar representations without departing from the spirit and claims of the present invention, and such modifications all fall within the protection scope of the present invention.

Claims

1. A method for de-raining images used in UAV power grid inspection, characterized in that, Includes the following steps: Aerial images captured by a drone are acquired, and information mining operations are performed on the aerial images step by step to obtain denoised sub-images; in at least one level of information mining operations, spatial dimension reduction and spatial dimension expansion operations are performed sequentially. The target image is constructed based on the denoised sub-image, thereby completing the image deraining operation; Multiple feature mining mechanisms arranged in series are used to perform step-by-step information mining operations on the aerial images; The feature mining mechanism internally includes a space reduction unit, a space expansion unit, and a correction unit. The internal computation process of the feature mining mechanism includes: The spatial dimension of the received subgraph is reduced using the spatial reduction unit to generate the first adapted subgraph; The first adapt subgraph is expanded in spatial dimension using spatial expansion units to generate a second adapt subgraph. The calibration unit takes the first adaptation subgraph and the second adaptation subgraph as input and generates a calibration subgraph through calculation; The calibration subgraph is merged with the second adaptation subgraph to generate an optimized subgraph; The internal computation process of the space reduction unit includes: The subgraph received by the space reduction unit is processed sequentially using the max pooling layer and the first activation function to generate the first reduced subgraph; The first reduced subgraph is processed sequentially using the first convolutional layer and the second activation function to generate the second reduced subgraph. The subgraph received by the spatial reduction unit is processed sequentially using the average pooling layer and the third activation function to generate the third reduced subgraph; Subtract the third reduced subgraph from the second reduced subgraph to generate the fourth reduced subgraph; The fourth reduced subgraph is processed sequentially using the second convolutional layer and the fourth activation function to generate the fifth reduced subgraph. The first reduced subgraph and the fifth reduced subgraph are multiplied by Hadamard to obtain the first adapted subgraph.

2. The image deraining method for UAV power grid inspection according to claim 1, characterized in that, The denoised subgraph is the optimized subgraph generated by the last feature mining mechanism.

3. The image deraining method for UAV power grid inspection according to claim 1, characterized in that, The spatial expansion unit includes a deconvolution layer and a fifth activation function arranged sequentially.

4. The image deraining method for UAV power grid inspection according to claim 2, characterized in that, The internal calculation process of the calibration unit includes: The first adaptation subgraph is subjected to full-channel average pooling and full-channel max pooling respectively, resulting in a first matrix and a second matrix. Perform a first deconvolution on the first matrix to generate a third matrix, and perform a second deconvolution on the second matrix to generate a fourth matrix; The second adaptation subgraph is subjected to full-channel max pooling and full-channel average pooling respectively, resulting in the fifth and sixth matrices. Subtracting the fifth matrix from the fourth matrix produces the seventh matrix; adding the third matrix to the sixth matrix produces the eighth matrix. The fifth, sixth, seventh, and eighth matrices are concatenated to generate a combined matrix; The combined matrix is ​​subjected to a first convolution calculation and a first activation calculation in sequence to obtain the collation primer diagram.

5. The image deraining method for UAV power grid inspection according to claim 4, characterized in that, The internal operation process of the calibration unit also includes: performing a Hadamard product between the third matrix and the fourth matrix to generate a ninth matrix, and generating an additional subgraph after the ninth matrix is ​​activated by the sigmoid function.

6. The image deraining method for UAV power grid inspection according to claim 5, characterized in that, The process of constructing the target image based on the denoised sub-image includes: The additional subgraphs generated in all the calibration units are stitched together to produce the first transition subgraph; The first transition subgraph is sequentially processed by the second convolution calculation and the second activation calculation to generate the second transition subgraph. The denoised subgraph is merged with the second transition subgraph to generate a third transition subgraph; The third transition subgraph is sequentially processed through a third convolution calculation and a third activation calculation to generate the target image.

7. An image de-raining device for UAV power grid inspection, characterized in that, Includes a computer program / instruction that, when executed, implements the method as described in any one of claims 1-6.

Images