Intelligent registration and fusion method for close-range non-coaxial spectral image of power line patrol

Abstract

The invention relates to an intelligent registration and fusion method of power line patrol close-range non-coaxial spectral images, which finally realizes high-precision fusion of spectral images through region-of-interest identification, gridding feature point detection, nearest neighbor matching, random consistency sampling, image transformation and batch processing. According to the method, high-precision image registration of non-coaxial close-range images can be effectively achieved, the average comprehensive mutual information value exceeds 0.8 on average, pixel-level spectral image information fusion can be achieved, and high-quality registration and information fusion are basic prerequisites for insulator surface state enhanced imaging and spectral reconstruction imaging. Meanwhile, a large amount of redundant calculation and registration errors caused by complex background image feature point matching can be avoided, the calculation efficiency is greatly improved, and in-situ analysis and real-time imaging under an unmanned aerial vehicle airborne camera are facilitated. The method is high in automation degree and high in adaptability, and the efficiency and accuracy of power line patrol image processing are remarkably improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular to an intelligent registration and fusion method for short-range non-coaxial spectral images of power line inspection. Background Art

[0002] Insulators in high-voltage transmission lines are prone to insulation degradation due to long-term exposure to harsh and complex natural environments, which seriously threatens the safety and reliability of power transmission systems. In view of this, the status monitoring of insulators occupies a vital position in the maintenance and inspection of power lines. Although manual inspections are outstanding in terms of target targeting, they are limited in efficiency and cannot keep up with the rapid expansion of the scale of transmission lines in modern society, nor can they meet the urgent needs of the power grid for intelligent and digital transformation. The rise of drone technology, with its excellent flexibility and efficiency, provides a new solution for the non-contact detection of transmission line insulators and has broad application potential. In the non-contact status detection method of insulators, the multi-band image information obtained by the multi-spectral camera on the drone can provide important data support for the analysis and evaluation of the status of line insulators, such as the visualization analysis of contamination based on non-coaxial multi-spectral cameras.

[0003] Among drone-mounted cameras of various structures, multi-eye non-coaxial imaging devices can realize modular combination of multiple spectral channels and ensure high spatial resolution of acquired images. However, in the application scenario of drone line patrol, non-coaxial imaging of close-focus targets will inevitably produce large visual distortion and nonlinear registration problems. Non-coaxial images must be effectively registered to achieve image information fusion, and the spectral resolution needs to be effectively improved without losing the spatial resolution of the original image. Most of the existing non-coaxial image registration methods are suitable for long-distance target shooting and do not require high registration accuracy. However, for power line patrol scenarios such as contamination, aging, heating and discharge that aim at visual information fusion, pixel-level registration is required to improve information utilization.

[0004] For the registration of non-coaxial spectral images, the most commonly used method is the image registration method based on feature point detection. The extraction of image feature points is the key to this type of method. The commonly used traditional algorithms include SIFT, SURF, ORB, etc. However, most of the existing registration algorithms have large nonlinear errors, and the interference of complex background images will also affect the extraction of feature points, reduce the registration efficiency, and cause the failure of image information fusion, which greatly limits the application effect of UAV spectral imaging technology. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and propose an intelligent registration and fusion method for short-range non-coaxial spectral images of power line inspection, which can avoid a large number of redundant calculations and registration errors caused by matching feature points of complex background images, greatly improve the calculation efficiency, and facilitate in-situ analysis and real-time imaging under drone-mounted cameras.

[0006] The present invention solves the technical problem by adopting the following technical solutions:

[0007] The intelligent registration and fusion method of the short-range non-coaxial spectral images of power line inspection includes the following steps:

[0008] Step 1: construct a region of interest recognition model, use the region of interest recognition model to regionalize the target insulator in the image, calibrate the position information of the region of interest positioning frame, and eliminate the background data outside the frame to obtain the region of interest;

[0009] Step 2: Grid partition the region of interest, use feature point detection algorithm to detect feature points in each partition network of the image, and generate a set of feature point coordinates;

[0010] Step 3: Use the nearest neighbor matching algorithm to pair the feature points of the reference image and the axis image to be registered, and output the pixel position information of the feature pairing;

[0011] Step 4: Use random consistency sampling algorithm to process pixel position information to obtain image transformation parameters;

[0012] Step 5: Use the image transformation module to perform projection transformation on the image to be registered to obtain an image registered with the reference image;

[0013] Step 6: Repeat the above steps to perform the same processing on other registered images to obtain all the registered images;

[0014] Step 7: The grayscale value set of each pixel point of each registered image is used to form a spectral image vector matrix to complete the spectral image fusion.

[0015] Moreover, the method for constructing the region of interest recognition model in step 1 is:

[0016] Step 1.1, establish three-dimensional digital white models of various commonly used line insulators and annotate the labels;

[0017] Step 1.2, covering the surface of the three-dimensional digital white mold with uniform or uneven random color spots to form a virtual three-dimensional insulator model;

[0018] Step 1.3, superimposing the virtual 3D insulator model on the common background image of the transmission line corridor;

[0019] Step 1.4, randomly encode the rotation, translation, overlap, and scaling posture parameters of the virtual three-dimensional insulator model superimposed on the two-dimensional image of the common background of the transmission line corridor, use the random encoding to generate the two-dimensional plane image data of the posture insulator, and select the rectangular area containing the insulator. In principle, the number of pixels in the non-insulator area in the selection range shall not exceed 30% of the total number of pixels in the selection;

[0020] Step 1.5: Use the Yolo series algorithm to train and test the two-dimensional plane image data of different white model postures, and repeat steps 1.2 to 1.4 until the correct judgment rate of the training model reaches more than 95%.

[0021] Moreover, the label in step 1.1 includes the insulator model, type and material.

[0022] Moreover, the grid partitioning in step 2 equally divides the area within the frame into n×m rectangular areas, and the ratio R of the number of pixels in each area to the total number of pixels is maintained in the range of 1 / 25≤R≤1 / 9.

[0023] Moreover, the feature point detection algorithm in step 2 preferably adopts the SuperPoint algorithm, constructs a feature point detection channel for n×m rectangular areas, completes feature point detection, and generates a feature point coordinate set {[x, y]}.

[0024] Moreover, the specific implementation method of step 3 is as follows: use the nearest neighbor matching algorithm Nearest Neighbor Search and perform calculation and estimation. When the number of pixels in the rectangular area Ns exceeds 1.8×105, start the FLANN algorithm to speed up the nearest neighbor search and use the KD tree as the index type; the initial value of the leaf point number (checks) is set to the number of pixels in the rectangular area Ns×10 -4 Round up.

[0025] Moreover, the vectors in the spectral image vector matrix (Axy) in step 7 are Axy=[g1,g2,…gi,…,gk-1,gk], where gi is the grayscale value at the i-th feature point (x, y) of the spectral image to be registered.

[0026] The advantages and positive effects of the present invention are:

[0027] The present invention finally realizes high-precision fusion of spectral images through region of interest identification, grid feature point detection, nearest neighbor matching, random consistency sampling, image transformation and batch processing. The present invention can effectively realize high-precision image registration of non-coaxial close-range images, with an average comprehensive mutual information value exceeding 0.8 on average, and can achieve pixel-level spectral image information fusion. High-quality registration and information fusion are the basic prerequisites for insulator surface state enhanced imaging and spectral reconstruction imaging. At the same time, it can avoid a large number of redundant calculations and registration errors caused by complex background image feature point matching, greatly improve computing efficiency, and facilitate in-situ analysis and real-time imaging under drone airborne cameras. The present invention has a high degree of automation and strong adaptability, and significantly improves the efficiency and accuracy of power line inspection image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the present invention;

[0029] Figure 2 It is a schematic diagram of the spectral image registration and fusion results of the non-coaxial spectral insulator of the present invention. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below with reference to the accompanying drawings.

[0031] Intelligent registration and fusion methods for short-range non-coaxial spectral images of power line inspection, such as Figure 1 As shown, the following steps are included:

[0032] Step 1: construct a region of interest recognition model, use the region of interest recognition model to regionalize the target insulator in the image, calibrate the position information of the region of interest positioning box, and eliminate the background data outside the box to obtain the region of interest.

[0033] The construction method of the region of interest recognition model is:

[0034] Step 1.1: Create three-dimensional digital white models of various commonly used line insulators and annotate the labels (insulator model, type, material, etc.).

[0035] Step 1.2: Cover the surface of the three-dimensional digital white mold with uniform or non-uniform random color spots to form a virtual three-dimensional insulator model.

[0036] Step 1.3: Superimpose the virtual 3D insulator model on common background images of the transmission line corridor, such as urban roads, forests, mountains and mining areas.

[0037] Step 1.4, randomly encode the rotation, translation, overlap, and scaling posture parameters of the three-dimensional insulator model superimposed on the two-dimensional image of the common background of the transmission line corridor, use random encoding to generate two-dimensional plane image data of the posture insulator, and select a rectangular area containing the insulator. In principle, the number of pixels in the non-insulator area in the selection range shall not exceed 30% of the total number of pixels in the selection.

[0038] Step 1.5: Use the Yolo series algorithm to train and test the two-dimensional plane image data of different white model postures, and repeat steps 1.2 to 1.4 until the correct judgment rate of the training model reaches more than 95%.

[0039] Step 2: Grid partition the region of interest, use feature point detection algorithm to detect feature points in each partition network of the image, and generate a set of feature point coordinates.

[0040] The grid partitioning method divides the area in the frame into n×m rectangular areas in an equal manner, and the ratio R of the number of pixels in each area to the total number of pixels is maintained in the range of 1 / 25≤R≤1 / 9.

[0041] The feature point detection algorithm preferably adopts the SuperPoint algorithm, constructs a feature point detection channel for n×m rectangular areas, completes feature point detection, and generates a feature point coordinate set {[x, y]}.

[0042] Step 3: Use the nearest neighbor matching algorithm to pair the feature points of the reference image and the axis image to be registered, and output the pixel position information of the feature pairing.

[0043] The nearest neighbor matching algorithm Nearest Neighbor Search is used and the calculation is estimated. When the number of pixels in the rectangular area Ns exceeds 1.8×105, the FLANN algorithm is started to speed up the nearest neighbor search speed, and the KD tree is used as the index type; the initial value of the leaf point number (checks) is set to the number of pixels in the rectangular area Ns×10 -4 Round up.

[0044] Step 4: Use random consistency sampling algorithm to process pixel position information to obtain image transformation parameters.

[0045] Step 5: Use the image transformation module to perform a projection transformation on the image to be registered to obtain an image registered with the reference image.

[0046] Step 6: Repeat the above steps to perform the same processing on other registered images to obtain all the registered images.

[0047] Step 7: The grayscale value set of each pixel point of each registered image is used to form a spectral image vector matrix to complete the spectral image fusion.

[0048] The vectors in the spectral image vector matrix (Axy) are Axy = [g1, g2, ...gi, ..., gk-1, gk], where gi is the grayscale value at the i-th feature point (x, y) of the spectral image to be registered.

[0049] According to the above-mentioned intelligent registration and fusion method of the close-range non-coaxial spectral images of power line inspection, the effect of the present invention has been verified by calculating a group of 6 non-coaxial spectral images with 3 million pixels.

[0050] Constructing an insulator region of interest identification model

[0051] Use SolidWorks or other digital 3D modeling software to create a 3D digital white model of the line insulator.

[0052] ① The types of insulators include suspension insulators, post insulators, porcelain insulators, glass insulators and composite insulators used in 10kV to 750kV lines;

[0053] ② For each established three-dimensional digital model of the insulator, annotate the model, shape, material and other information as training labels to build an insulator shape library.

[0054] Cover the generated insulator 3D white model with random color spots:

[0055] ① Collect the surface material patterns of insulator surface pollution, aging, powdering, water mist, etc. to generate random color spots;

[0056] ② Cover the surface of the three-dimensional digital model with uniform or uneven random color spots to simulate stains or aging in the actual environment.

[0057] Overlay a 2D background image on the 3D white model of the insulator:

[0058] The insulator model is superimposed on typical transmission line corridor backgrounds, such as urban roads, forests, mountains, mining areas, etc.

[0059] Attitude parameter encoding:

[0060] ① Randomly generate a set of coding instructions that can rotate, translate, overlap, scale, and perform other operations on the insulator model, and construct random coding posture parameters.

[0061] ② Generate two-dimensional plane image data of the posture insulator, and select the rectangular area containing the insulator, and keep the pixel ratio of the non-insulator area at 19%.

[0062] Train the Yolo model:

[0063] ③ Use the Yolo series algorithms to train and test two-dimensional plane image data of different postures.

[0064] ④ Repeat steps 2 to 4 above until the model’s correct judgment rate reaches 97%

[0065] Registration and fusion of multiple spectral images

[0066] Image preprocessing:

[0067] The trained region of interest recognition model is used to process the input power line inspection image to identify and locate the target insulator.

[0068] Grid partitioning:

[0069] The identified region of interest (rectangular region) is divided into equal grids of 3×4, and the number of pixels in each rectangular region accounts for 1 / 12 of the total number of pixels.

[0070] Feature point detection within the frame:

[0071] The SuperPoint algorithm is used to detect feature points in each grid area and generate a feature point coordinate set {[x, y]}.

[0072] Pairing of feature points of the registered image with the reference image:

[0073] ① Use the nearest neighbor matching algorithm to pair the feature points of the reference image and the image to be registered, and output the pixel position information of the feature pairing.

[0074] ② Select the number of spectral image pixels to be 3×10 6 At this time, the number of pixels in the area Ns is 1.93×10 5 , use the FLANN algorithm to accelerate the nearest neighbor search, use the KD tree as the index type, and set the number of leaf points (checks) to 20.

[0075] Feature point image transformation:

[0076] ①Use the random consistency sampling algorithm to process feature matching information and obtain image transformation parameters.

[0077] ② Use the image transformation module to project the image to be registered to obtain the image registered with the reference image.

[0078] Multiple spectral image registration:

[0079] Repeat steps 1 to 5 for other images to be registered to complete the registration of all images.

[0080] Spectral image fusion:

[0081] The pixel grayscale value set of each registered image is used to form a spectral image vector matrix (Axy), where Axy = [g1, g2, …gi, …, g6], gi is the grayscale value at the i-th feature point (x, y) of the spectral image to be registered, and the fusion of the spectral images is completed.

[0082] Figure 2 The following is an example of a spectral reconstruction image obtained by registering and fusing a set of 6 non-coaxial spectral insulator spectral images after the above steps. Figure 2 The figure shows a reconstructed image (the last one) obtained after a group of six 3-megapixel non-coaxial spectral 750V line composite insulator images (the first six) are registered and fused using the method in the embodiment of the present invention. Different colors in the reconstructed image represent the distribution of different degrees of spectral reflectivity.

[0083] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation manner. Any other implementation manners derived by those skilled in the art based on the technical solution of the present invention also fall within the scope of protection of the present invention.

Claims

1. An intelligent registration and fusion method for short-range non-coaxial spectral images of power line inspection, characterized in that: The following steps are involved: Step 1: construct a region of interest recognition model, use the region of interest recognition model to regionalize the target insulator in the image, calibrate the position information of the region of interest positioning frame, and eliminate the background data outside the frame to obtain the region of interest; Step 2: Grid partition the region of interest, use feature point detection algorithm to detect feature points in each partition network of the image, and generate a set of feature point coordinates; Step 3: Use the nearest neighbor matching algorithm to pair the feature points of the reference image and the axis image to be registered, and output the pixel position information of the feature pairing; Step 4: Use random consistency sampling algorithm to process pixel position information to obtain image transformation parameters; Step 5: Use the image transformation module to perform projection transformation on the image to be registered to obtain an image registered with the reference image; Step 6: Repeat the above steps to perform the same processing on other registered images to obtain all the registered images; Step 7: The grayscale value set of each pixel point of each registered image is used to form a spectral image vector matrix to complete the spectral image fusion.

2. The intelligent registration and fusion method for power line inspection short-range non-coaxial spectral images according to claim 1 is characterized by: The method for constructing the region of interest recognition model in step 1 is: Step 1.1, establish three-dimensional digital white models of various commonly used line insulators and annotate the labels; Step 1.2, covering the surface of the three-dimensional digital white mold with uniform or uneven random color spots to form a virtual three-dimensional insulator model; Step 1.3, superimposing the virtual 3D insulator model on the common background image of the transmission line corridor; Step 1.4, randomly encode the rotation, translation, overlap, and scaling posture parameters of the virtual three-dimensional insulator model superimposed on the two-dimensional image of the common background of the transmission line corridor, use the random encoding to generate the two-dimensional plane image data of the posture insulator, and select the rectangular area containing the insulator. In principle, the number of pixels in the non-insulator area in the selection range shall not exceed 30% of the total number of pixels in the selection; Step 1.5: Use the Yolo series algorithm to train and test the two-dimensional plane image data of different white model postures, and repeat steps 1.2 to 1.4 until the correct judgment rate of the training model reaches more than 95%.

3. The intelligent registration and fusion method for power line inspection short-range non-coaxial spectral images according to claim 2 is characterized by: The label in step 1.1 includes the insulator model, type and material.

4. The intelligent registration and fusion method for power line inspection short-range non-coaxial spectral images according to claim 1 is characterized by: In the step 2, the grid partitioning is performed by equally dividing the area within the frame into n×m rectangular areas, and the ratio R of the number of pixels in each area to the total number of pixels is maintained in the range of 1 / 25≤R≤1 / 9.

5. The intelligent registration and fusion method for power line inspection short-range non-coaxial spectral images according to claim 1 is characterized by: The feature point detection algorithm in step 2 preferably adopts the SuperPoint algorithm to construct a feature point detection channel for n×m rectangular areas, complete feature point detection, and generate a feature point coordinate set {[x, y]}.

6. The intelligent registration and fusion method for power line inspection short-range non-coaxial spectral images according to claim 1 is characterized by: The specific implementation method of step 3 is as follows: use the nearest neighbor matching algorithm Nearest Neighbor Search and perform calculation and estimation. When the number of pixels in the rectangular area Ns exceeds 1.8×105, start the FLANN algorithm to speed up the nearest neighbor search speed, and use the KD tree as the index type; the initial value of the leaf point number (checks) is set to the number of pixels in the rectangular area Ns×10 -4 Round up.

7. The intelligent registration and fusion method for power line inspection short-range non-coaxial spectral images according to claim 1 is characterized by: The vectors in the spectral image vector matrix (Axy) in step 7 are Axy=[g1,g2,…gi,…,gk-1,gk], where gi is the grayscale value at the i-th feature point (x, y) of the spectral image to be registered.

Images