A method and device for removing rain and fog from power UAV inspection images

Abstract

The present invention provides a method and device for removing rain and fog from inspection images of electric power drones, the method comprising: constructing a m subtensors of different spatial scales; among them, m is an integer and m is greater than or equal to 2, is an integer, and; uses the low-rank tensor of the inspection image and the spatial neighborhood similarity to remove rain and fog from the sub-tensor to obtain an initial rain and fog-free image; performs decision-average fusion on the initial rain and fog-free image at each spatial scale to generate a final restored image. The present invention can better utilize contextual spatial structural information to improve the performance of removing rain and fog, and can ensure that even under adverse weather conditions, the electric power drone can obtain high-quality inspection images, and improve the safety and reliability of intelligent operation and inspection of power facilities; it can be widely used in fields such as electric power drone inspection, and has broad application prospects and market value.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for removing rain and fog from inspection images of electric power drones. Background Art

[0002] Power drones have been widely used in power overhead line inspections due to their ease of operation and high efficiency. Especially in areas with complex geographical conditions, such as mountainous areas, drones can overcome terrain obstacles and achieve effective monitoring of transmission lines. However, in harsh natural environments, such as rain, snow, clouds and fog, the quality of inspection images obtained by drones will be affected, resulting in the loss or blurring of some spatial textures and detail information in the images.

[0003] In this case, image restoration technology becomes particularly important. It can restore the clarity of disturbed images to a certain extent and improve detection accuracy. The application of this technology helps ensure that even in adverse weather conditions (such as rain and fog), power drones can provide high-quality inspection data, thereby ensuring the safe operation and maintenance of power facilities.

[0004] In summary, there is an urgent need for a method and device for removing rain and fog from power UAV inspection images to solve the problems existing in the prior art. Summary of the invention

[0005] The purpose of the present invention is to provide a method for removing rain and fog from inspection images of electric power drones, aiming to remove the interference of rain and fog on inspection images and meet the requirements for high-quality imaging of electric power inspection images. The specific technical scheme is as follows:

[0006] A method for removing rain and fog from power UAV inspection images, comprising the following steps:

[0007] Step S1: construct the collected inspection image m Subtensors of different spatial scales ;in, m is an integer and m Greater than or equal to 2, is an integer and ;

[0008] Step S2: Use the low-rank tensor of the inspection image and the spatial neighborhood similarity to obtain the sub-tensor Remove rain and fog to obtain the initial rain and fog-free image ;

[0009] Step S3: Initial rain and fog removal images at each spatial scale Perform decision averaging fusion to generate the final restored image .

[0010] In the above technical solution, preferably, in step S1, the inspection image Y Using subtensors It is expressed as:

[0011] (1),

[0012] in, Represents the subtensor in the inspection image The number of Indicates n The first spatial scale Subtensor .

[0013] The preferred solution in the above technical solution is: set a sliding window with a step size of 1 and take the sub-tensor The spatial scale is ,but ;in, M , N are the two spatial dimensions of the inspection image, Q is the number of channels of the inspection image, Represents a sub-tensor The spatial dimension of Satisfy and less than M and N .

[0014] In the above technical solution, preferably, in step S2, the low-rank tensor of the inspection image and the spatial neighborhood similarity are used to obtain the sub-tensor The solution formula for removing rain and fog is:

[0015] (8)

[0016] in, Indicates n The center of the spatial scale is The low-rank subtensor of Indicates n The center of the spatial scale is The sparse subtensor of Indicates n The center of the spatial scale is The sub-tensor of is the spatial neighborhood similarity constraint, l is a parameter used to balance the weights between low-rank sub-tensors and sparse sub-tensors. is the nuclear norm of the subtensor, is the 1-norm of the subtensor, is the Fourier norm of the matrix, is the preset weight value.

[0017] In the above technical solution, preferably, the spatial neighborhood similarity constraint term Specifically:

[0018] (9)

[0019] in, Indicates n The center of the spatial scale is The sub-tensor of Indicates n The center of the spatial scale is The sub-tensor of Indicates n The center of the spatial scale is The sub-tensor of Indicates n The center of the spatial scale is The sub-tensor of Indicates n The center of the spatial scale is A sub-tensor of .

[0020] In the above technical solution, preferably, formula (8) is converted into formula (10) and formula (11) for iterative solution:

[0021] (10)

[0022] (11)

[0023] When the iteration end condition is met, the solution is obtained in the first n Low-rank subtensors at different spatial scales , then the initial rain and fog removal image It is expressed as:

[0024] (12)

[0025] in, Indicates n At the spatial scale k +1 iteration centered at The low-rank subtensor of Indicates n At the spatial scale k The center of the iteration is The sparse subtensor of Represents the subtensor in the inspection image The number of Indicates n The first spatial scale A low-rank subtensor.

[0026] The preferred technical solution above is: when the The low-rank subtensor obtained after the solution is ;

[0027] In formula (8) , ;in, Q is the number of channels of the inspection image, Represents a sub-tensor The spatial dimension of Preset value for Gaussian noise intensity.

[0028] In the above technical solution, preferably, in step S3, the initial rain-free fog image at each spatial scale is calculated according to formula (13): Perform decision averaging fusion to generate the final restored image :

[0029] (13).

[0030] The present invention also provides a device for removing rain and fog from inspection images of electric power drones, comprising a memory and a processor. When the processor runs computer instructions stored in the memory, it executes the method for removing rain and fog from inspection images of electric power drones.

[0031] The present invention also provides another device for removing rain and fog from inspection images of electric power drones, comprising an image acquisition unit and a data calculation unit; the image acquisition unit is used to photograph electric power equipment to obtain inspection images; the data calculation unit processes the inspection images according to the method for removing rain and fog from inspection images of electric power drones.

[0032] The application of the technical solution of the present invention has the following beneficial effects:

[0033] The rain and fog removal method of the present invention increases the spatial neighborhood similarity constraint term, mines the similarity information in the sub-tensor space field, improves the smoothness of image restoration, further eliminates or minimizes the influence of noise, and improves image quality; the rain and fog removal method of the present invention can better utilize the contextual spatial structure information to improve the performance of removing rain and fog, and can ensure that even under adverse weather conditions (such as rain and fog interference), the electric power drone can obtain high-quality inspection images, and improve the safety and reliability of intelligent operation and inspection of power facilities; it can be widely used in fields such as electric power drone inspection, and has broad application prospects and market value.

[0034] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0036] Figure 1 This is a flow chart of the method for removing rain and fog from electric power UAV inspection images of the present invention. DETAILED DESCRIPTION

[0037] In order to facilitate the understanding of the present invention, the present invention will be described more fully below, and preferred embodiments of the present invention are provided. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0039] Example:

[0040] See also Figure 1 This embodiment provides a method for removing rain and fog from an electric power UAV inspection image, comprising the following steps:

[0041] Step S1: construct the collected inspection image m Subtensors of different spatial scales ;in, m is an integer and m Greater than or equal to 2, is an integer and ;

[0042] Preferably, the power equipment is photographed by a power drone to obtain an original visible light image Y (i.e., an inspection image) in jpg format. The inspection image can be regarded as a three-dimensional tensor, and its data scale size is expressed as ,in, M , N are the two spatial dimensions of the inspection image, Q is the number of channels of the inspection image, specifically the three RGB channels, that is Q =3.

[0043] Due to the complexity of the background of the power inspection image, a single spatial scale is difficult to fully mine the spatial structure information in the image. Therefore, this embodiment adopts m sub-tensors of different spatial scales to fully mine the spatial structure information in the inspection image; specifically, Subtensors of spatial scales The construction method is:

[0044] Designing a seed tensor , and its spatial scale is , Represents a sub-tensor The spatial dimension of Must be less than M and N ; Design a sliding window with a step size of 1, each inspection image can be divided into Subtensor , then the inspection image Y Using subtensors It can be expressed as:

[0045] (1)

[0046] in, , Indicates n The first spatial scale Subtensor .

[0047] Furthermore, inspection images Y use m The subtensors of different spatial scales are expressed as:

[0048] (2)

[0049] Step S2: Use the low-rank tensor of the inspection image and the spatial neighborhood similarity to obtain the sub-tensor Remove rain and fog to obtain the initial rain and fog-free image ;

[0050] Prior art "C. Lu, J. Feng, Y. Chen, W. Liu, Z. Lin, and S. Yan,''Tensor robust principal component analysis: Exact recovery of corrupted low-rank tensors via convex optimization,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2016, pp. 5249–5257. " proposed a model of low-rank tensor recovery algorithm for image restoration. The model assumes an input image Y Can be decomposed into a low-rank tensor L, sparse tensor S and the noise tensor N , expressed as:

[0051] (3)

[0052] At the same time, the prior art "C. Lu, J. Feng, Y. Chen, W. Liu, Z. Lin, and S. Yan,''Tensor robust principal component analysis: Exact recovery of corrupted low-rank tensors via convex optimization,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2016, pp. 5249–5257. " also discloses a target function for solving, which is expressed as:

[0053] (4)

[0054] in, l is a tensor used to balance low rank L and sparse tensors S The weight parameter between is the nuclear norm of the tensor, is the 1-norm of the tensor, is the Fourier norm of the matrix, is the preset value of Gaussian noise intensity, usually .

[0055] Since the spatial neighborhood pixels of the inspection image are similar, a linear combination of several basis vectors can be used to represent each pixel in the background. Therefore, the overall area of ​​the inspection image usually has a low-rank characteristic. Compared with the overall area of ​​the image, rain and fog targets are usually small-area targets. Rain and fog targets have sparse characteristics. Therefore, formula (4) can be used to remove rain and fog from the inspection image.

[0056] The characteristic of the low-rank tensor recovery algorithm is to use the low-rank characteristics of the image itself to remove rain and fog from the image. However, due to the complexity of the inspection image, only using the low-rank features of the image may not be able to effectively remove the rain and fog targets in the image. Therefore, based on the existing technology, this embodiment further innovates the use of the low-rank tensor of the inspection image and the spatial neighborhood similarity to remove the sub-tensor To improve the image de-raining effect, this embodiment constructs a spatial neighborhood similarity constraint term. Adding it to formula (4) yields formula (5):

[0057] (5)

[0058] Specifically, in this embodiment It is expressed as:

[0059] (6)

[0060] in, Indicates the center is The sub-tensor of Indicates the center is The sub-tensor of Indicates the center is The sub-tensor of Indicates the center is A sub-tensor of .

[0061] Transforming formula (5) into an objective function with penalty term is:

[0062] (7)

[0063] in, is the preset weight value.

[0064] Formula (7) is the solution formula for low-rank tensor decomposition of the entire inspection image. According to formula (1), the sub-tensor It can be converted into the following formula for solution:

[0065] (8)

[0066] in, Indicates n The center of the spatial scale is The low-rank subtensor of Indicates n The center of the spatial scale is The sparse subtensor of Indicates n The center of the spatial scale is The sub-tensor of is the spatial neighborhood similarity constraint; in formula (8), l is a parameter used to balance the weights between low-rank sub-tensors and sparse sub-tensors. is the nuclear norm of the subtensor, is the 1-norm of the subtensor, is the Fourier norm of the matrix, is the preset weight value; further, in formula (8) , .

[0067] Furthermore, the spatial neighborhood similarity constraint term By mining the similarity information in the spatial domain, the smoothness of image restoration can be improved, the influence of noise can be further eliminated or minimized, and the image quality can be improved. The spatial neighborhood similarity constraint term in formula (8) Specifically:

[0068] (9)

[0069] in, Indicates n The center of the spatial scale is The sub-tensor of Indicates n The center of the spatial scale is The sub-tensor of Indicates n The center of the spatial scale is The sub-tensor of Indicates n The center of the spatial scale is The sub-tensor of Indicates n The center of the spatial scale is A sub-tensor of .

[0070] Furthermore, formula (8) can be converted into formula (10) and formula (11) for iterative solution:

[0071] (10)

[0072] (11)

[0073] in, Indicates n At the spatial scale k +1 iteration centered at The low-rank subtensor of Indicates n At the spatial scale k The center of the iteration is A sparse sub-tensor of .

[0074] When satisfied The low-rank subtensor obtained after the solution is According to formula (1), we can get n The overall low-rank tensor recovered at different spatial scales (i.e. the initial rain and fog removal image) is expressed as:

[0075] (12)

[0076] Preferably, in this embodiment, a successive iteration algorithm is used to iteratively solve formula (10) and formula (11). For the successive iteration algorithm, please refer to the prior art "X. Yuan and J. Yang, ''Sparse and low-rank matrix decomposition via alternating direction methods,' Pacific J. Optim., vol. 9, no. 1, pp. 167–180, 2013.", which is not described in detail in this embodiment.

[0077] Step S3: Initial rain and fog removal images at each spatial scale Perform decision average fusion to generate the final restored image.

[0078] According to formula (13), the initial de-rained and fogged images at different spatial scales are Fuse them together to get the fused restored image:

[0079] (13)

[0080] in, This is the restored image after fusion with rain and fog removed.

[0081] This embodiment also provides a device for removing rain and fog from inspection images of electric power drones, comprising a memory and a processor. When the processor runs computer instructions stored in the memory, the method for removing rain and fog from inspection images of electric power drones is executed.

[0082] This embodiment provides another device for removing rain and fog from electric power drone inspection images, including an image acquisition unit and a data calculation unit; the image acquisition unit is used to photograph electric power equipment to obtain inspection images; the data calculation unit processes the inspection images according to the method for removing rain and fog from electric power drone inspection images.

[0083] The rain and fog removal method of this embodiment can better utilize contextual spatial structure information to improve the performance of removing rain and fog, and can ensure that even under adverse weather conditions (such as rain and fog interference), the power drone can obtain high-quality inspection images, thereby improving the safety and reliability of intelligent operation and inspection of power facilities; it can be widely used in fields such as power drone inspection, and has broad application prospects and market value.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for removing rain and fog from power UAV inspection images, characterized in that: The following steps are involved: Step S1: construct the collected inspection image m Subtensors of different spatial scales ;in, m is an integer and m Greater than or equal to 2, is an integer and ; Step S2: Use the low-rank tensor of the inspection image and the spatial neighborhood similarity to obtain the sub-tensor Remove rain and fog to obtain the initial rain and fog-free image ; Step S3: Initial rain and fog removal images at each spatial scale Perform decision averaging fusion to generate the final restored image ; In step S2, the low-rank tensor of the inspection image and the spatial neighborhood similarity are used to obtain the sub-tensor The solution formula for removing rain and fog is: （8） in, Indicates n The center of the spatial scale is The low-rank subtensor of Indicates n The center of the spatial scale is The sparse subtensor of Indicates n The center of the spatial scale is The sub-tensor of is the spatial neighborhood similarity constraint, λ is a parameter used to balance the weights between low-rank sub-tensors and sparse sub-tensors. is the nuclear norm of the subtensor, is the 1-norm of the subtensor, is the Fourier norm of the matrix, is the preset weight value; The spatial neighborhood similarity constraint Specifically: （9） in, Indicates n The center of the spatial scale is The sub-tensor of Indicates n The center of the spatial scale is The sub-tensor of Indicates n The center of the spatial scale is The sub-tensor of Indicates n The center of the spatial scale is A sub-tensor of .

2. The method for removing rain and fog from power drone inspection images according to claim 1 is characterized in that: Inspection image in step S1 Y Using subtensors It is expressed as: （1） in, Represents the subtensor in the inspection image The number of Indicates n The first spatial scale Subtensor .

3. The method for removing rain and fog from power drone inspection images according to claim 2 is characterized in that: Set a sliding window with a step size of 1 and take the sub-tensor The spatial scale is ,but ;in, M , N are the two spatial dimensions of the inspection image, Q is the number of channels of the inspection image, Represents a sub-tensor The spatial dimension of Satisfy and less than M and N .

4. The method for removing rain and fog from power drone inspection images according to claim 1 is characterized in that: Convert formula (8) into formula (10) and formula (11) and solve them iteratively: （10） （11） When the iteration end condition is met, the solution is obtained in the first n Low-rank subtensors at different spatial scales , then the initial rain and fog removal image It is expressed as: （12） in, Indicates n At the spatial scale k +1 iteration centered at The low-rank subtensor of Indicates n At the spatial scale k The center of the iteration is The sparse subtensor of Represents the subtensor in the inspection image The number of Indicates n The first spatial scale A low-rank subtensor.

5. The method for removing rain and fog from power drone inspection images according to claim 4 is characterized in that: When satisfied The low-rank subtensor obtained after the solution is ; In formula (8) , ;in, Q is the number of channels of the inspection image, Represents a sub-tensor The spatial dimension of Preset value for Gaussian noise intensity.

6. The method for removing rain and fog from power drone inspection images according to claim 1 is characterized in that: In step S3, the initial rain-free and fog-free images at each spatial scale are calculated according to formula (13): Perform decision averaging fusion to generate the final restored image : （13）。 7. A rain and fog removal device for power drone inspection images, characterized in that: It comprises a memory and a processor, and when the processor runs the computer instructions stored in the memory, it executes the method for removing rain and fog from the inspection image of an electric power drone as described in any one of claims 1 to 6.

8. A rain and fog removal device for power drone inspection images, characterized in that: It includes an image acquisition unit and a data calculation unit; the image acquisition unit is used to photograph the power equipment to obtain an inspection image; the data calculation unit processes the inspection image according to the power UAV inspection image rain and fog removal method as described in any one of claims 1-6.

Images