Control Method and System for Unmanned Aerial Vehicle (UAV) Power Line Inspection Based on Image and Data Fusion

Abstract

This invention relates to a control method and system for unmanned aerial vehicle (UAV) power system inspection based on image and data fusion, belonging to the field of UAV control. The method includes: first, the UAV flies at high altitude to acquire survey image information, and preprocesses it to extract features and location information of power facilities. Based on the location information of the power facilities, the inspection area and the power facilities to be inspected are determined. Using the image information of the power facilities in the inspection area, the take-off and landing positions of the UAV are determined. The images of the take-off and landing positions and the images of the power facilities to be inspected are fused with data to determine the UAV's flight path. Finally, the UAV begins its inspection, acquiring image information of the inspected power facilities, and then analyzing whether the power facilities are faulty. This invention controls the UAV's flight path through image and data fusion, enabling automatic inspection of power systems without manual operation. It can handle various terrain environments, improve the efficiency of power system inspection, and reduce labor costs.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) control, specifically relating to a control method and system for UAV power line inspection based on image and data fusion. Background Technology

[0002] With the rapid development of China's economy, the demand for electricity is gradually increasing. High-voltage and ultra-high-voltage overhead power lines are the main methods for long-distance power transmission and distribution. Therefore, the safe operation of ultra-high-voltage, high-capacity power lines is a strong power guarantee for my country's economic construction. Reducing power transmission faults and power outages are key issues that need to be addressed. Manual inspections are affected by terrain, weather, and personnel factors, resulting in low efficiency, low data accuracy, and difficulty in ensuring personnel safety. Therefore, drone inspections are gradually being used in power grid safety monitoring. However, for conventional drone inspections, the system often sets up a cruise route and captures, transmits, and analyzes images at each cruise point, which is inefficient and yields inaccurate inspection results.

[0003] Therefore, how to provide a control method for power line inspection using unmanned aerial vehicles (UAVs) to address the problems of conventional UAV power line inspections, which rely on pre-set navigation routes and image capture and analysis at each fixed navigation point—a process that is labor-intensive, inefficient, and yields inaccurate results—is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to overcome the deficiencies of the prior art and provide a control method and system for UAV power system inspection based on image and data fusion. By using the method of UAV image and data fusion, the flight path of the UAV is controlled, and automatic inspection of the power system by UAV can be achieved without manual operation. It can cope with various terrain environments, improve the efficiency of power system inspection, and reduce labor costs.

[0005] To achieve the above objectives, the technical solution of the present invention is: a control method for unmanned aerial vehicle (UAV) power line inspection based on image and data fusion, comprising:

[0006] Preprocess the survey images acquired by the drone to determine the area to be inspected and the power facilities to be inspected.

[0007] Based on the image information of the power facilities to be inspected in the survey image information, determine the take-off and landing position of the drone;

[0008] Image and data fusion is performed on images of the drone's take-off and landing locations and survey images to determine the drone's flight path;

[0009] Based on the take-off and landing locations and flight paths of drones, drones are used to inspect and obtain inspection image information in order to analyze whether there are faults in power facilities.

[0010] In one embodiment of the present invention, the survey image information is obtained by using a drone equipped with a binocular camera to fly over the area to be inspected and capture images of the geographical environment of the area.

[0011] In one embodiment of the present invention, the method of preprocessing the survey image information acquired by the UAV to determine the inspection area and the power facilities to be inspected is as follows: after processing the survey image acquired by the UAV including image denoising, image enhancement and image registration, a fused image is synthesized according to the fusion rules, and the power facilities to be inspected are extracted based on the fused image to determine the location of the power facilities to be inspected, and then the inspection area is determined.

[0012] In one embodiment of the present invention, the method for determining the take-off and landing position of the UAV based on the image information of the power facility to be inspected in the survey image information is as follows: after the UAV carrying a binocular camera acquires the image information of the power facility to be inspected, the image information of the power facility to be inspected in the area to be inspected is processed and calculated, the influence range of the power facility to be inspected is determined based on the image information of the power facility to be inspected, the flight path range is determined based on the influence range of the power facility to be inspected and the location of the power facility to be inspected, and finally the take-off and landing position of the UAV is determined.

[0013] In one embodiment of the present invention, the method of fusing images and survey images of the UAV take-off and landing positions to determine the UAV trajectory is as follows: using discrete wavelet transform, each survey image is decomposed into three sub-images, and then the three sub-images from the survey image are fused at each level to obtain the fused image of that level. After obtaining the fused images of all corresponding levels, an inverse transform is performed to obtain the fusion result. Based on the fusion result of the UAV take-off and landing positions and survey images, combined with the UAV flight dynamics constraints and limitations, the optimal UAV trajectory is planned.

[0014] In one embodiment of the present invention, the method for analyzing whether a power facility is faulty is as follows: preprocessing the inspection image information, classifying power facility damage, and qualitatively judging power facility damage through a pre-created damage detection model to obtain the damage detection result of the power facility to be inspected.

[0015] In one embodiment of the present invention, the qualitative judgment of damage to power facilities includes one or more of the following: line breakage, tower deformation, and insulation damage.

[0016] In one embodiment of the present invention, the specific implementation method for preprocessing the survey image information acquired by the UAV to determine the area to be inspected and the power facilities to be inspected is as follows:

[0017] (1) Image denoising and image enhancement processing are performed on the survey image information, as follows:

[0018] The mathematical model for a noisy image z:X→R is:

[0019] z(x)=y(x)+η(x),x∈X

[0020] Where: x belongs to the image domain The 2D spatial coordinates, y is the original image, and η is the coordinates in σ. 2 It is zero-mean Gaussian noise with zero variance, i.e. Z is represented by a block of fixed size N1×N1 taken from z. x Where x is the coordinate of the top-left corner of the block, i.e., Z. x Located at x of z;

[0021] When performing block estimation, the reference image is processed using a sliding window method. x R ∈X, grouping is achieved by block matching within the noisy image z. Blocks with a distance less than a fixed threshold are considered similar and grouped together. If the real image is available, the block distance calculation expression is:

[0022]

[0023] in: The block size represents the hard threshold block size used for block estimation, and ||·||2 represents the L2 norm. and Y x x located in y R At x, but since only the noisy image z is available, the block distance can only be determined by... and Z x The calculation is as follows:

[0024]

[0025] when Z x When there is no overlap, the block distance is a non-centered chi-square random variable with mean E and variance var as follows:

[0026]

[0027]

[0028] As shown in the above formula, only blocks with an ideal distance greater than the threshold will be matched for similarity, while others will be ignored. Therefore, pre-filtering is introduced. By applying two normalized 2D linear transforms, the coefficients are hard-thresholded to obtain the required distance:

[0029]

[0030] In the formula, Υ′ is the threshold λ 2D The hard threshold operator for σ, Represents a normalized 2D linear transformation;

[0031] The result of block matching is a set, expressed in coordinates as follows:

[0032]

[0033] In the formula, It is the maximum distance during block matching;

[0034] Collaborative filtering is obtained through a threshold in the 3D transform domain. Normalized 3D linear transformation Make the real image group It exhibits sparsity, and the hard threshold effectively attenuates noise, resulting in block estimation. 3D array:

[0035]

[0036] Where γ is the threshold λ 3D The hard threshold operator for σ, These are the coordinates after block matching. This refers to a fixed-size block located at Z. Place, Depend on Estimated value of each stack of blocks Composition, to obtain a block estimate

[0037] Given a global preliminary estimate of a real image Based on this value, groups are performed, and Wiener filtering is applied to the grouped results to obtain the normalized 2D linear distance. Replace it with the normalized distance calculated within the initial estimate, and obtain The approximate value; therefore, the coordinates of the block matching are:

[0038]

[0039] in, It is the hard threshold size for Wiener filtering. It is the maximum distance during block matching, block and x located in y R and at x;

[0040] Initial estimation group within the estimated value Depend on Stacked together These are the block matching coordinates after Wiener filtering.

[0041] yes At that time, a block of fixed size is located at Z. Place, A fixed-size block located at Z. Place, Decide Normalized linear 3D transform coefficients of the initial estimation group Energy definition, empirical Wiener contraction coefficient for:

[0042]

[0043] yes At that time, the initial estimation group located within the estimated value will normalize the linear 3D transform coefficients of the noisy data. With Wiener's shrinkage coefficient Element-wise multiplication, inverse normalization transform A set of estimates was obtained:

[0044]

[0045] At the matching position Block estimate at the location Also in the initial estimation group In the middle, a block estimate is obtained.

[0046] Two estimates are obtained during block estimation. and These are all partial representations of the real image, not complete representations; the block estimation regions have a probability of overlapping. In order to calculate the final estimate of the real image, the pixel positions of the overlapping parts of the corresponding block regions are weighted and averaged, i.e., the aggregation operation.

[0047] Block estimation is statistically correlated and biased, and the variance of each pixel is different. The weights in weighted aggregation are inversely proportional to the variance of the block estimation; that is, blocks with less noise have larger weights. and The weight allocations are as follows:

[0048]

[0049]

[0050] in, A fixed-size block located at Z. Place, This refers to a fixed-size block located at Z. Place, and They are respectively and When noise is independent and The sample variance;

[0051] Global preliminary estimate From block estimates The weighted average is obtained as follows:

[0052]

[0053] in, It is located at x m Characteristic function of the block at ∈X, block estimation and In x m Zero-padding is performed outside of X;

[0054] Global final estimate The results were calculated based on the weighted average draw, which was initially estimated globally. and Each by and replace;

[0055] (2) Segmenting the target and selecting registration feature points: After performing image denoising and image enhancement on the survey image information, the target power facilities and the area to be inspected are segmented. The corner points, edges and other representative points of the power facilities in the image are automatically detected. The extracted points should be able to effectively describe the structural information of the power facilities. The selected features should be the corresponding points of the same physical marker on the two images.

[0056] (3) Image registration using feature points: By calculating the similarity between feature points of power facilities, the extracted power facility outlines are compared with the positions and characteristics of feature points in different images to find corresponding matching points. Feature points are regarded as linear or nonlinear transformations between two datasets. Image space coordinate transformation parameters, such as translation and rotation, are calculated using these feature points. These parameters describe how to transform one image into the position and orientation of another image. Finally, the error between the two transformed datasets is minimized according to a predetermined criterion.

[0057] (4) Image fusion creation: The useful information of the two registered power facility images is fused and expressed as a two-dimensional or three-dimensional image in the same coordinate system;

[0058] (5) Parameter extraction: Extract and measure the characteristic parameters of relevant power facilities from the fused images for qualitative and quantitative analysis. Determine the specific inspection area and the power facilities to be inspected.

[0059] In one embodiment of the present invention, before the drone inspection, the power facilities to be inspected need to be classified and their minimum safe zones determined. The power facilities to be inspected are divided into two main categories based on their height: low-profile and high-pole types, denoted as h. obs h is the height of the power facility. d h is the altitude for drone inspection operations. l As the dividing standard, when h obs ≤h l At that time, it was a low-profile type; h obs >h l At that time, it was a tall pole type, h l Inspection operation altitude h of drones d And so on.

[0060] This invention also provides a control system for unmanned aerial vehicle (UAV) power line inspection based on image and data fusion, comprising:

[0061] The first module preprocesses the survey images acquired by the UAV to determine the area to be inspected and the power facilities to be inspected.

[0062] The second module determines the take-off and landing location of the drone based on the image information of the power facilities to be inspected in the survey image information;

[0063] The third module fuses images and data from the UAV take-off and landing locations and survey images to determine the UAV's flight path.

[0064] The fourth module, based on the drone's take-off and landing location and flight path, involves the drone conducting inspections and obtaining inspection image information to analyze whether power facilities are faulty.

[0065] Compared with the prior art, the present invention has the following advantages: The present invention controls the flight path of the UAV by using the method of fusion of UAV images and data, and can realize automatic inspection of the power system of the UAV without manual operation. It can cope with various terrain environments, improve the efficiency of power system inspection, and reduce labor costs. Attached Figure Description

[0066] Figure 1 This is a flowchart of the control method of the present invention.

[0067] Figure 2 This is the framework of the binocular vision system of the present invention.

[0068] Figure 3 This is a model of a collection of low-rise power facilities.

[0069] Figure 4 This is a model of a collection of high-pole power facilities. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0071] To address the low efficiency of drone-based power line inspection in existing technologies, this invention provides a control method for drone-based power line inspection based on image and data fusion, comprising:

[0072] Preprocess the survey images acquired by the drone to determine the area to be inspected and the power facilities to be inspected.

[0073] Based on the image information of the power facilities to be inspected in the survey image information, determine the take-off and landing position of the drone;

[0074] Image and data fusion is performed on images of the drone's take-off and landing locations and survey images to determine the drone's flight path;

[0075] Based on the take-off and landing locations and flight paths of drones, drones are used to inspect and obtain inspection image information in order to analyze whether there are faults in power facilities.

[0076] This invention also provides a control system for unmanned aerial vehicle (UAV) power line inspection based on image and data fusion, comprising:

[0077] The first module preprocesses the survey images acquired by the UAV to determine the area to be inspected and the power facilities to be inspected.

[0078] The second module determines the take-off and landing location of the drone based on the image information of the power facilities to be inspected in the survey image information;

[0079] The third module fuses images and data from the UAV take-off and landing locations and survey images to determine the UAV's flight path.

[0080] The fourth module, based on the drone's take-off and landing location and flight path, involves the drone conducting inspections and obtaining inspection image information to analyze whether power facilities are faulty.

[0081] The following are specific implementation examples of the present invention.

[0082] This invention discloses a control method for unmanned aerial vehicle (UAV) power line inspection based on image and data fusion. See [link to relevant documentation]. Figure 1 The control method includes:

[0083] 101. Unmanned aerial vehicles (UAVs) fly at high altitudes to acquire survey image information;

[0084] 102. Process the surveyed images to determine the areas to be inspected and the power facilities that need to be inspected;

[0085] 103. Process the image information of the power facilities in the area to be inspected and determine the take-off and landing positions;

[0086] 104. Images of take-off and landing locations and images and data of the power facilities to be inspected are fused together to determine the UAV's flight path;

[0087] 105. The drone begins its inspection, acquiring inspection image information.

[0088] 106. The images and data of power facilities obtained during inspections are fused together to analyze whether the power facilities are faulty.

[0089] Step 101 involves the UAV flying at high altitude to acquire survey image information. Specifically, this refers to using a UAV equipped with a binocular camera to fly over the area requiring inspection. Binocular cameras are not only inexpensive, small, and lightweight, but also unaffected by the external environment, which has led to their widespread adoption in UAVs. Binocular stereo vision is a non-contact method based on the principle of human parallax, using a computer within the UAV to acquire the target's distance and orientation. Imaging equipment is used to observe the target from two or more locations, obtaining at least 20 two-dimensional images. The parallax of the object is calculated by matching pixels from images at different viewpoints, thereby perceiving the three-dimensional geometric features of the power facility area.

[0090] Step 102, which processes the surveyed images, must adhere to the fundamental principle that each corresponding point on two or more images must be accurately aligned. Its main steps are summarized as follows:

[0091] (1) Perform noise reduction, enhancement and other processing on the acquired image data, and unify the data format, image size and resolution.

[0092] To ensure flight safety, UAVs need to acquire real-time images of the airspace ahead via imaging systems during flight. These images are then processed and analyzed to determine the status of the airspace. However, images are affected to varying degrees by the imaging system, storage medium, and environmental factors during acquisition, storage, and transmission, resulting in issues such as noise pollution, limited dynamic range, low contrast, and blurred edges. These problems severely impact the UAV system's recognition and judgment of image content. To ensure flight and mission safety, it is necessary to reduce errors in image analysis and improve image quality. Therefore, image denoising and image enhancement become crucial.

[0093] 1) Block estimation

[0094] The mathematical model for a noisy image z:X→R is:

[0095] z(x)=y(x)+η(x),x∈X

[0096] Where: x belongs to the image domain 2D spatial coordinates, and η is in σ 2 It is zero-mean Gaussian noise with zero variance. We represent Z using a block of fixed size N1×N1 taken from z. x , where x is the coordinate of the top-left corner of the block. In other words, we call Z... x It is located at x in z.

[0097] When performing block estimation, the reference image is processed using a sliding window method. (located at current coordinate x) R ∈X). Grouping is achieved by block matching within the noisy image z, where only blocks whose distance (i.e., dissimilarity) relative to a reference distance is less than a fixed threshold are considered similar and grouped together. Ideally, if the real image were available, the block distance calculation expression would be:

[0098]

[0099] in: The block size represents the hard threshold block size used for block estimation, and ||·||2 represents the L2 norm. and Y x x located in y R And at x∈X. However, only the noisy image z is available, and the block distance can only be obtained from the noise blocks. and Z x The calculation is as follows:

[0100]

[0101] when Z x When there is no overlap, the block distance is a non-centered chi-square random variable with mean E and variance var as follows:

[0102]

[0103]

[0104] As shown in the formula above, only blocks with an ideal distance greater than the threshold will be matched for similarity, while others will be ignored. Pre-filtering can avoid this problem by hard-thresholding the coefficients through two normalized 2D linear transformations to obtain the desired distance.

[0105]

[0106] In the formula, Υ′ is the hard threshold λ 2DThe operator of σ Represents a normalized 2D linear transformation.

[0107] The result of block matching is a set, which can be expressed in coordinates as follows:

[0108]

[0109] In the formula, It is the maximum distance during block matching.

[0110] Collaborative filtering is obtained through a threshold in the 3D transform domain. Normalized 3D linear transformation is selected Make the real image group It exhibits sparsity. Hard thresholding effectively attenuates noise, resulting in block estimation. 3D array.

[0111]

[0112] Where γ is the threshold λ 3D The hard threshold operator for σ. Depend on Estimated value of each stack of blocks composition.

[0113] Given a preliminary estimate of a real image Based on this value, the data is grouped, and Wiener filtering is applied to the grouped results to improve noise reduction. According to the established assumption, the noise in the initial estimate has been significantly attenuated, and the normalized 2D linear distance is then... Replace it with the normalized distance calculated within the initial estimate, and obtain The approximate value. Therefore, the coordinates of the block match are:

[0114]

[0115] in, It is the hard threshold size for Wiener filtering. It is the maximum distance during block matching, block and x located in y R and x∈X.

[0116] Preliminary estimates and noise groups, initial estimate groups within the estimated values. Depend on Stacked together Decide Normalized linear 3D transform coefficients of the initial estimation group Energy definition, empirical Wiener contraction coefficient for:

[0117]

[0118] Normalized linear 3D transform coefficients of noise data With Wiener's shrinkage coefficient Element-wise multiplication, inverse normalization transform A set of estimates was obtained:

[0119]

[0120] At the matching position Block estimate at the location It is also included in this group's estimates.

[0121] 2) Aggregation achieves global estimation

[0122] Two estimates are obtained during block estimation. and Both of these estimates are partial representations of the real image, not complete ones. The estimated block regions have a certain probability of overlap; in regions where many other similar blocks are near a given block, there are numerous overlapping areas, leading to a probabilistically incomplete signal representation. To calculate the final estimate of the real image, a weighted average of the pixel positions in the overlapping portions of the corresponding block regions is performed—a process known as aggregation.

[0123] Block estimation is statistically correlated, biased, and the variance varies for each pixel. The weights in weighted aggregation are inversely proportional to the variance of the block estimation; that is, blocks with less noise have larger weights. Group estimation and The weight allocations are as follows:

[0124]

[0125]

[0126] in, and for and When the noise in the middle is independent, and The sample variance.

[0127] Global preliminary estimate From block estimates The weighted average is obtained.

[0128]

[0129] in, It is located at x m Characteristic function of the block at ∈X, block estimation Zero padding was applied outside this area.

[0130] Global final estimate The results were calculated based on the weighted average draw, which was initially estimated globally. and Each by and replace.

[0131] (2) Segmenting the target and selecting registration feature points: After performing image denoising and image enhancement on the survey image information, the target power facilities and the area to be inspected are segmented. The corner points, edges and other representative points of the power facilities in the image are automatically detected. The extracted points should be able to effectively describe the structural information of the power facilities. The selected features should be the corresponding points of the same physical marker on the two images.

[0132] (3) Image registration using feature points: By calculating the similarity between feature points of power facilities, the extracted power facility outlines are compared with the positions and characteristics of feature points in different images to find corresponding matching points. Feature points are regarded as linear or nonlinear transformations between two datasets. Image space coordinate transformation parameters, such as translation and rotation, are calculated using these feature points. These parameters describe how to transform one image into the position and orientation of another image. Finally, the error between the two transformed datasets is minimized according to a predetermined criterion.

[0133] (4) Image fusion creation: The useful information of the two registered power facility images is fused and expressed as a two-dimensional or three-dimensional image in the same coordinate system;

[0134] (5) Parameter extraction: Extract and measure the characteristic parameters of relevant power facilities from the fused images for qualitative and quantitative analysis. Determine the specific inspection area and the power facilities to be inspected.

[0135] Step 103 involves processing the image information of the power facilities in the area to be inspected and determining the take-off and landing positions. After acquiring relevant information about the power facilities from the binocular camera, this information needs to be processed and calculated so that the UAV can monitor the airspace situation in real time. Binocular stereo matching is a crucial step in the target extraction process. The accuracy of stereo matching directly affects the accuracy of power facility extraction, which is significantly related to subsequent route planning and determines the flight safety of the UAV.

[0136] like Figure 2As shown, to ensure the ranging accuracy requirements of the binocular ranging system, a series of image processing techniques are required. First, the binocular vision system calibration involves calibrating the intrinsic and extrinsic parameters of the monocular ZED camera separately, and then calibrating the relative position parameters of the binocular cameras within the entire structure using the extrinsic parameters of the left and right cameras. Ensuring the accuracy of the binocular camera intrinsic and extrinsic parameters, as well as the accuracy of the system parameters in the binocular ranging system, ensures the accuracy of the ranging results when these parameters are substituted into the distance calculation formula. After obtaining the intrinsic and extrinsic parameters of the binocular cameras, the next step is stereo calibration. Based on the obtained intrinsic and extrinsic parameters of the calibrated cameras, the image images are first distorted, and then the two image images are adjusted to ensure that their optical axes are parallel, their image planes are coplanar, and their epipolar lines are aligned. Stereo calibration is performed before stereo matching to ensure that matching points are on the same horizontal line, thus reducing the matching search range to a one-dimensional horizontal line. This reduces matching time and improves efficiency. Then, stereo matching is performed, and after stereo matching is completed, contour extraction is performed on the initial disparity map.

[0137] In step 104, the images and data of the take-off and landing positions and the power facilities to be inspected are fused to determine the UAV's flight path. By referring to environmental information such as terrain, weather, and enemy presence, and combining factors such as its own performance, flight requirements, fuel consumption, obstacle avoidance, and mission constraints, one or more optimal paths with high safety and short flight time are planned for the UAV to ensure that it can successfully complete the power inspection mission and return safely. The UAV must have autonomous flight path planning capabilities. UAV flight path planning in power inspection generally refers to planning a smooth, flyable route from the starting base to the target location in a known terrain environment, based on terrain constraints, UAV dynamics constraints, and mission constraints, and that this route is an optimal route that meets the mission constraints. Therefore, flight path planning is essentially an optimization problem. Assuming the UAV's Cartesian coordinate system is o-xyz, and any point P in space... i The coordinates are (x i ,y i ,z i The coordinates of the takeoff base P0 are (x0, y0, z0), and the coordinates of the target point Pd are (x0, y0, z0). d ,y d ,z d The drone's initial state is P0, and its final state is P. d The goal of route planning is to calculate M route nodes {P1, P2, ..., P} that satisfy the aforementioned constraints from the flight area. i ,...,P m-1 ,P m}, P1=P0, P m =P dFinally, the M nodes are connected and optimized into the required flight path. Next, based on the obtained obstacle depth information, combined with UAV flight dynamics constraints and other limitations, analysis and decision-making are performed to plan the optimal flight path. Based on the planned flyable smooth flight path, the UAV's flight state is adjusted to move towards the target.

[0138] In step 105, the UAV begins its inspection, acquiring inspection image information. First, it uses a binocular camera to obtain information about the power facilities. Then, it uses an effective fusion method between this information and the trajectory planning model data. This method analyzes and organizes the data into valid data, which can then be used for subsequent trajectory planning. By adding relevant constraints to the UAV during trajectory planning, a reliable automatic inspection control method is achieved. During the inspection, a minimum safety zone needs to be established around the power facilities. When the UAV enters this zone, it acquires information about the power facilities using a binocular camera. Power facilities come in various shapes; if the minimum safety zone is set to a single shape and the same inspection method is used, unnecessary large-angle flights may occur, reducing the efficiency of the UAV's inspection work. Therefore, power facilities need to be classified and their minimum safety zones determined separately. Power facilities are divided into two main categories based on their height: low-profile (… Figure 3 ) and tall pole type ( Figure 4 ). Let h be the number of digits. obs h is the height of the power facility. d h is the altitude for drone inspection operations. l As the dividing standard. When h obs ≤h l At that time, it was a low-profile type; h obs >h l At that time, it was a tall pole type. h l The classification standard is based on the inspection altitude h of the drone. d As for the value, h is generally taken as the value. l =2h d ~3h d By fusing with image data, drones can adjust their flight paths according to different power facilities.

[0139] In step 106, based on the acquired images of the power facilities, the system analyzes whether the power facilities are faulty. After completing the inspection flight, the UAV returns to base, transmitting continuous images of the power facilities requiring inspection, captured by the binocular camera, to the ground station. The ground station then performs inspections on the power system based on the acquired images. Specifically, a memory or memory card can be installed inside the UAV, allowing images to be transmitted via copying after the UAV returns. The image processing module then uses a pre-created damage detection model to perform damage detection on at least 20 power system images, automatically detecting and classifying line breaks, tower deformation, and insulation damage in the power system, thereby facilitating subsequent maintenance.

[0140] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0144] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A control method for unmanned aerial vehicle (UAV) power inspection based on image and data fusion, characterized in that, The method comprises the following steps: Preprocessing survey image information obtained by the unmanned aerial vehicle, determining a region to be inspected and power facilities to be inspected; Based on the image information of the power facilities to be inspected in the survey image information, determining the take-off and landing position of the unmanned aerial vehicle; Image and data fusion of the take-off and landing position image of the unmanned aerial vehicle and the survey image information to determine the flight path of the unmanned aerial vehicle; Based on the take-off and landing position of the unmanned aerial vehicle and the flight path of the unmanned aerial vehicle, the unmanned aerial vehicle inspects and obtains inspection image information to analyze whether the power facilities are faulty; The survey image information is obtained by flying an unmanned aerial vehicle carrying a binocular camera in the region to be inspected, and shooting the geographical environment image of the region to be inspected; The preprocessing of the survey image information obtained by the unmanned aerial vehicle, the determination of the region to be inspected and the power facilities to be inspected are as follows: After the survey image obtained by the unmanned aerial vehicle is processed through image denoising, image enhancement and image registration, a fused image is synthesized according to the fusion rule, and the position of the power facilities to be inspected is determined based on the fused image, and then the region to be inspected is determined; After the image information of the power facilities to be inspected in the region to be inspected is obtained by the unmanned aerial vehicle carrying a binocular camera, the image information of the power facilities to be inspected in the region to be inspected is processed and calculated, the influence range of the power facilities to be inspected is determined based on the image information of the power facilities to be inspected, the flight path range is determined based on the influence range of the power facilities to be inspected and the position of the power facilities to be inspected, and finally the take-off and landing position of the unmanned aerial vehicle is determined; The method for image and data fusion of the take-off and landing position image of the unmanned aerial vehicle and the survey image information to determine the flight path of the unmanned aerial vehicle is as follows: Discrete wavelet transform is used to decompose each survey image into three sub-images, then the three sub-images from the survey image are fused at each level to obtain the fused image at this level, and after obtaining all the corresponding level fused images, inverse transformation is performed to obtain the fusion result; According to the fusion result of the take-off and landing position image of the unmanned aerial vehicle and the survey image information, the optimal flight path of the unmanned aerial vehicle is planned combined with the flight dynamics constraints and limitations of the unmanned aerial vehicle; Before the unmanned aerial vehicle inspection, the power facilities to be inspected need to be classified and divided respectively and their minimum safety area is determined. The power facilities to be inspected are divided into two categories according to their height: low type and high pole type, denoted as h obs The height of the power facility is denoted as h d The height of the unmanned aerial vehicle inspection operation is denoted as h l The classification standard is that when h obs ≤h l , it is a low type; when h obs >h l , it is a high pole type, and h l is determined by the height of the unmanned aerial vehicle inspection operation h d . 2.The control method of the unmanned aerial vehicle power inspection based on image and data fusion according to claim 1, characterized in that, The method for analyzing whether the power facilities are faulty is as follows: Through the pre-created damage detection model, the inspection image information is preprocessed, the power facility damage is classified, and the power facility damage is qualitatively judged to obtain the damage detection result of the power facilities to be inspected; The qualitative judgment of the power facility damage includes one or more of the following: line breakage, tower deformation, and insulator damage; The specific implementation of preprocessing the survey image information obtained by the unmanned aerial vehicle to determine the region to be inspected and the power facilities to be inspected is as follows: where: x is a 2D spatial coordinate belonging to the image domain y is the original image, and η is a zero-mean Gaussian noise with variance σ 2 Z is represented by a block of fixed size N1xN1 taken from z x where x is the coordinate of the top-left corner of the block, i.e. Z x is located at x in z;​ When performing block estimation, the reference image is processed in a sliding window fashion x R ∈X, the grouping is achieved by block matching within the noisy image z, blocks with a block distance less than a fixed threshold are considered similar and grouped, if the true image is available, the block distance is computed as where: represents the hard threshold block size used by the block estimate, || · ||2represents the L2norm, and block and Y x are located at x R and x in y, but since only the noisy image z is available, the block distance can only be computed as and Z x as described by the following equation: When Z x When not overlapping, the block distance is a non-central chi-square random variable with mean E and variance var given by: (1) Image denoising and image enhancement processing of the survey image information is as follows: where Y' is a threshold value λ 2D a hard threshold operator of σ, denotes a normalized 2D linear transformation; The mathematical model of the noise image z:X→R is as follows: z(x) = y(x) + η(x), x∈X As can be seen from the above formula, only the block with a distance greater than the threshold value is matched, and others are ignored, so a pre-filter is introduced, which is applied to two normalized 2D linear transformations, and the coefficients are hard-thresholded to obtain the required distance: The result obtained after block matching is a set, which is expressed as a coordinate: wherein is the maximum distance when block matching; Thresholding in 3D transform domain for collaborative filtering Selecting normalized 3D linear transform Real image set Sparse, hard thresholding for effective noise attenuation, block estimate 3D array: where γ is a threshold λ 3D σ hard-thresholding operator, is the coordinate after block matching, denotes that a fixed size block is located at Z , is composed of stacked block estimates to obtain a block estimate Given a global preliminary estimate of a real image Grouping is performed on the basis of this value, and the result after grouping is Wiener filtered, and the normalized 2D linear distance is replaced by the normalized distance calculated in the initial estimate, to obtain an approximation of ; and the coordinates of the block matching are: wherein, is a hard threshold size for the Wiener filter, is the maximum distance in block matching, block and are located at x in y R and x; Initial estimate set located within the estimate By Stacked, Is the block matching coordinates after wiener filtering, is when the fixed size block is located at of Z, is the fixed size block located at of Z, determines the energy of the normalized linear 3D transform coefficients of the initial estimate group defines the empirical wiener shrinkage coefficients as: is When the initial estimate set is within the estimate, the normalized linear 3D transform coefficients of the noise data are multiplied by the elements of the Wiener shrinkage coefficients and the normalized inverse transform yields an estimate set: at the matching position block estimate value also in the initial estimate group a block estimate Two estimates are obtained at block estimation and are partial representations of the real image, not complete representations; the block estimation regions have a probability of overlapping, and for the final estimate of the real image, the pixel positions in the overlapping part of the corresponding block regions are processed by weighted averaging, i.e. aggregation operation; The block estimates are statistically correlated, biased, and have different variances for each pixel. The weights in the weighted aggregation are inversely proportional to the variances of the block estimates, i.e. the block estimates with small noise are weighted inversely large, and the estimates and The weight distributions of the two methods are respectively: where is a fixed size block located at Z , is a fixed size block located at Z , and are the sample variances of and respectively when the noise in and are independent. global preliminary estimate from the block estimate by weighted averaging wherein is the characteristic function of the block located at x m ∈X, the block estimate and is zero padded outside of x m ∈X global final estimate is computed from the weighted tie-breaks according to the global preliminary estimate, wherein and are replaced by and respectively; (2) Segmentation target and selection of registration feature points: after image denoising and image enhancement processing of survey image information, the power facilities to be inspected and the area to be inspected are segmented, and the representative feature points of the power facilities to be inspected in the image are selected, and the selected feature points should be the corresponding points of the same physical marker in two images; (3) Image registration using feature points: by calculating the similarity between the feature points of the power facilities to be inspected, the extracted power facility contour is compared with the positions and characteristics of the feature points in different images to find the corresponding matching points. The feature points are regarded as linear or nonlinear transformation between two matching point data sets. The image space coordinate transformation parameters are calculated through these feature points. Finally, the error of the two matching point data sets after transformation is minimized to meet the predetermined criteria; (4) Fusion image creation: the two power facility images after registration are fused in the same coordinate system to express the useful information of each other into two-dimensional or three-dimensional images; (5) Parameter extraction: qualitative and quantitative analysis of the feature parameters of the power facilities are extracted and measured from the fused image to determine the specific inspection area and the power facilities to be inspected.

3. A control system for unmanned aerial vehicle power line inspection based on image and data fusion, characterized in that, The control method for unmanned aerial vehicle power inspection based on image and data fusion according to claim 1 or 2, the system comprises: A first module for pre-processing survey image information obtained by the unmanned aerial vehicle to determine the area to be inspected and the power facilities to be inspected; A second module for determining the unmanned aerial vehicle take-off and landing position based on the power facility image information in the survey image information; A third module for image and data fusion of the unmanned aerial vehicle take-off and landing position image and the survey image information to determine the unmanned aerial vehicle flight path; A fourth module for unmanned aerial vehicle inspection based on the unmanned aerial vehicle take-off and landing position, the unmanned aerial vehicle flight path, and obtaining inspection image information to analyze whether the power facility is faulty.

Images