Multispectral channel image space alignment fusion method and device for avoiding image misregistration

Abstract

The application relates to the technical field of image fusion, in particular to a multispectral channel image space alignment fusion method and device capable of avoiding image misplacement, which comprises the following steps: collecting detection images of composite insulators on a power transmission line in different spectral channels, and extracting edge contours corresponding to the composite insulators in each detection image; extracting corner points of the edge contours, obtaining response values of the corner points; obtaining symmetrical corner points corresponding to each corner point; calculating the saliency of each corner point; obtaining each matching point corresponding to each corner point, determining the defect degree of each matching point; determining the matching degree of each corner point; obtaining the feature response degree of each corner point, screening all the corner points in each detection image, extracting feature points of each detection image, and performing image fusion. The application can improve the feature point matching precision, improve the multispectral channel image space alignment fusion efficiency and precision, and improve the image fusion effect.

Description

Technical Field

[0001] This application relates to the field of image fusion technology, specifically to a method and apparatus for spatial alignment fusion of multispectral channel images to avoid image misalignment. Background Technology

[0002] Multispectral channel image fusion technology, by integrating image information captured from different spectral bands, can significantly improve the comprehensiveness and accuracy of scene analysis, and has important application value in the field of UAV inspection of composite insulators for ultra-high voltage transmission lines. However, due to differences in the physical characteristics of imaging equipment in different spectral bands, sensor installation position deviations, asynchronous imaging times, and target movement in dynamic scenes, spatial misalignment exists between images from different channels, resulting in ghosting, blurred edges, or distortion of details after direct image fusion.

[0003] Traditional methods mainly achieve image alignment through feature point extraction and matching. However, in multispectral scenarios, the response differences of corner points in images of different bands are significant, resulting in low consistency of feature points selected from images of different bands. This leads to a high mismatch rate of feature points, resulting in poor fusion effect of multispectral images, which in turn affects the defect detection of composite insulators. Summary of the Invention

[0004] To address the aforementioned technical problems, a method and apparatus for spatial alignment and fusion of multispectral channel images that avoids image misalignment are provided to resolve the existing issues.

[0005] The solution to the technical problem addressed in this application is to provide a method and apparatus for spatial alignment and fusion of multispectral channel images to avoid image misalignment, including the following steps:

[0006] In a first aspect, embodiments of this application provide a multispectral channel image spatial alignment and fusion method to avoid image misalignment, the method comprising the following steps:

[0007] Images of composite insulators on transmission lines are acquired in different spectral channels, and the edge contours of the composite insulators in each image are extracted.

[0008] Extract the corner points of the edge contour and obtain the response value of each corner point; analyze the symmetry relationship between each corner point and the other corner points on the edge contour and obtain the corresponding symmetrical corner points;

[0009] The saliency of any corner point is calculated by considering the difference in response values ​​between any corner point and its symmetrical corner point in each image to be detected, as well as the extreme variations of the difference.

[0010] The corner points at the same position corresponding to any corner point in different images to be detected are recorded as the matching points corresponding to any corner point;

[0011] The distribution of gradient directions of all pixels in the neighborhood of the matching point is analyzed to obtain the main direction. The defect degree of the matching point is determined by the difference and dispersion of pixel values ​​of adjacent pixels in the main direction in the neighborhood of the matching point.

[0012] Based on the difference in saliency between any corner point and all its matching points, and in conjunction with the defect degree, the matching degree of any corner point is determined.

[0013] Based on saliency and matching degree, the feature response of any corner point is obtained. All corner points in each image to be detected are filtered, feature points of each image to be detected are extracted, and image fusion is performed.

[0014] Preferably, the image to be detected includes a visible light image, an ultraviolet image, and an infrared image, and the visible light image is grayscale processed.

[0015] Preferably, obtaining the symmetrical corner points corresponding to each corner point includes:

[0016] Obtain the center point of the smallest bounding rectangle of the edge contour, and denote it as the contour center;

[0017] A rectangle is constructed with the center of the outline as the center point and any corner point as the vertex. The corner points that are adjacent to the other vertices of the rectangle are denoted as the symmetrical corner points of any corner point.

[0018] Preferably, calculating the salience of any corner point includes:

[0019] The difference between the response values ​​of any corner point and its symmetrical corner points is denoted as the relative difference.

[0020] The sum of the relative differences between any given corner point and all its symmetrical corner points is denoted as the relative difference quantity.

[0021] Calculate the ratio of the response value at any corner point to the relative difference.

[0022] Calculate the range of the relative differences between any given corner point and all its symmetrical corner points;

[0023] The significance is the product of the range and the ratio.

[0024] Preferably, obtaining the main direction includes: constructing a histogram of the gradient directions of all pixels in the neighborhood of the matching point in the image to be detected, and recording the gradient direction corresponding to the largest magnitude as the main direction.

[0025] Preferably, determining the defect degree of the matching point includes:

[0026] In the neighborhood of the matching point, select multiple adjacent pixels along the main direction, calculate the difference in pixel value between the two selected adjacent pixels, and record it as the pixel difference;

[0027] Calculate the degree of dispersion of the pixel difference between all selected adjacent pixels;

[0028] For the matching point, the product of the sum of the pixel differences between all two adjacent pixels and the degree of dispersion is used as the defect degree of the matching point.

[0029] Preferably, determining the matching degree of any corner point includes:

[0030] Calculate the sum of the ratios of saliency between any corner point and all its corresponding matching points, and perform a negative mapping on the sum;

[0031] The ratio of the product of the defect degrees of all the matching points corresponding to any corner point to the result of the negative mapping of any corner point is taken as the matching degree of any corner point.

[0032] Preferably, the feature response is the product of the saliency and the matching degree.

[0033] Preferably, the step of extracting feature points from each image to be detected and performing image fusion includes:

[0034] Cluster the feature responsivity of all corner points in each image to be detected to obtain multiple clusters;

[0035] Calculate the average feature responsivity of all corner points within each cluster, and select all corner points within the cluster with the largest average value as feature points of each image to be detected;

[0036] All images to be detected are matched and aligned using the feature points, and then image fusion is performed on all images to be detected.

[0037] Secondly, embodiments of this application also provide a multispectral channel image spatial alignment and fusion device for avoiding image misalignment, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any of the above-described multispectral channel image spatial alignment and fusion methods for avoiding image misalignment.

[0038] This application has at least the following beneficial effects:

[0039] This application uses the ORB algorithm to extract corner points of the corresponding edge contours of composite insulators in images under test from different spectral channels, obtains the response values ​​of each corner point, analyzes the symmetry between other corner points and each corner point, obtains the corresponding symmetrical corner points of each corner point, and calculates the saliency of any corner point based on the difference in response values ​​between each corner point and its corresponding symmetrical corner points. Its beneficial effect lies in considering the symmetry between each corner point and its symmetrical corner points to reflect the prominence of the corner point in a local area and the shape and texture information of the composite insulator represented by that corner point in the image. Extreme changes in the difference reflect the possibility of the corner point's symmetry being broken. The saliency not only contains the texture features of the composite insulator surface but also the feature information of possible defects, reflecting the impact of the corner point on subsequent image registration and the accuracy of defect identification of the composite insulator. The application also analyzes the corner points at corresponding positions in different images under test, obtains the matching points corresponding to each corner point, and obtains the main direction in the neighborhood of the matching point by analyzing the gradient direction changes of the pixels in the neighborhood of the matching point. The difference and dispersion of pixel values ​​of adjacent pixels along the main direction in the neighborhood of the matching point are considered to calculate the defect degree of the matching point. This has the advantage of considering the drastic changes in pixel values ​​within the local range of the matching point, thus reflecting the possibility that the matching point is located in a complex defect area of ​​the image. The matching degree of any corner point is then determined, which has the advantage of considering the matching situation between each corner point and the matching point, thus explaining the impact of the corner point on the registration accuracy of subsequent images. The feature response of each corner point is obtained, and all corner points in each image to be detected are screened to extract the feature points of each image to be detected. Image fusion is then performed on all images to be detected. This has the advantage of improving the accuracy of feature point matching by extracting the feature points of each image to be detected through the feature response of each corner point, thereby improving the efficiency and accuracy of multispectral channel image spatial alignment fusion. The resulting image fusion effect is better, preserving the high-resolution texture details of the visible light image and highlighting the discharge abnormal area of ​​the ultraviolet image, while also fusing the temperature rise hotspot information of the infrared image, thus improving the accuracy of defect detection of composite insulators. Attached Figure Description

[0040] The following section provides a more detailed description of the multispectral channel image spatial alignment and fusion method for avoiding image misalignment in this application, with reference to the accompanying drawings.

[0041] Figure 1 A flowchart illustrating the steps of the multispectral channel image spatial alignment and fusion method for avoiding image misalignment provided in this application embodiment;

[0042] Figure 2 A flowchart illustrating the steps of a method for obtaining the saliency of each corner point in each image to be detected, as provided in an embodiment of this application.

[0043] Figure 3 A flowchart illustrating the steps of feature point extraction provided in this application embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of the multispectral channel image spatial alignment and fusion method and apparatus for avoiding image misalignment proposed in this application, in conjunction with the accompanying drawings and implementation examples, is provided. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit the scope of this application.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0046] Please see Figure 1 The diagram illustrates a flowchart of a multispectral channel image spatial alignment and fusion method for avoiding image misalignment according to an embodiment of this application. The method includes the following steps:

[0047] Step 1: Acquire images of composite insulators on transmission lines in different spectral channels and extract the edge contours of the composite insulators in each image.

[0048] Composite insulators for power transmission lines are key components in circuit systems, and their defect detection is crucial for ensuring power grid safety. The development of multi-source sensors has changed the past simple inspection method that mainly relied on human eyes. However, the instruments and equipment for manual inspection are now highly integrated and intelligent. Visual imaging equipment can also obtain fault and defect characteristics in real time through image fusion, and automatically locate the fault location in a timely manner. The full spectrum information composed of infrared, visible light, and ultraviolet light can reflect the defect information of composite insulators, improve the efficiency of visual inspection work, and reduce the cost of manual inspection.

[0049] Visible light is perceptible to the human eye and aligns with human visual characteristics, but it struggles to penetrate clouds, fog, snow, and haze in harsh weather conditions and has insufficient imaging capabilities at night, thus limiting its operating hours. Infrared radiation, on the other hand, has strong penetrating power and a strong ability to capture the heating characteristics of composite insulators, allowing it to operate at night. However, its imaging resolution and contrast are low, and compared to visible light, it does not conform to human visual habits. Ultraviolet images can detect arc discharge wavelengths and can effectively identify insulation defects in composite insulators in power equipment. To more accurately identify defects in composite insulator images, the images from the multispectral imaging channels are aligned and fused.

[0050] Based on the above analysis, by integrating sensors, an ultraviolet-sensitive CCD or CMOS sensor, a near-infrared or far-infrared thermal imaging sensor, and a standard visible light sensor can be integrated into the same device carried by a UAV. In this way, images of the composite insulator of ultra-high voltage transmission lines under test can be acquired in different spectral imaging channels. That is, ultraviolet images can be obtained through the ultraviolet-sensitive CCD or CMOS sensor, infrared images can be obtained through the near-infrared or far-infrared thermal imaging sensor, and visible light images can be obtained through the standard visible light sensor. The images under test include visible light images, ultraviolet images, and infrared images.

[0051] Secondly, since visible light images contain RGB channels, their pixel values ​​represent color and brightness, while ultraviolet and infrared images are single-channel images. In ultraviolet images, pixel values ​​represent ultraviolet photon counts or radiation intensity, and in infrared images, pixel values ​​represent temperature or radiation energy. Therefore, after grayscale processing of the visible light image, the composite insulator region of the transmission line is extracted from the visible light, ultraviolet, and infrared images respectively. Specifically:

[0052] Convert the visible light image to grayscale;

[0053] In this embodiment, a weighted grayscale algorithm is used to process the visible light image into grayscale. The weighted grayscale algorithm is a well-known technology and will not be described in detail here.

[0054] Edge detection is performed on each image to be detected to extract the edge contour corresponding to the composite insulator;

[0055] In this embodiment, the Canny edge detection algorithm is used for edge detection. The Canny edge detection algorithm is a well-known technology and will not be described in detail here. As other implementation methods, implementers can use other methods of the prior art, such as the Sobel operator, etc. This embodiment does not impose any special restrictions on this.

[0056] Thus, visible light, ultraviolet, and infrared images of the composite insulators of the transmission line were obtained.

[0057] Step 2: Extract the corner points of the edge contour and obtain the response value of each corner point; analyze the symmetry relationship between each corner point and the other corner points on the edge contour and obtain the corresponding symmetrical corner points; calculate the significance of any corner point by the difference in response value between any corner point and its symmetrical corner point in each image to be detected and the extreme changes in the difference.

[0058] Based on the above analysis, the research on visible light, infrared light, and ultraviolet light imaging fusion technology and the identification of its typical defects is a core issue in the field of intelligent inspection. Image fusion requires image registration. Traditional algorithms extract key points from the images, use key point detection algorithms to find corresponding feature points in different images, and calculate the transformation relationship between different images based on the accurate positions of these points, thus achieving image spatial alignment. During this process, the images to be fused must be strictly aligned. If the registration algorithm has errors in feature point detection, matching, or geometric transformation model selection, it will lead to misalignment in subsequent fusion. Therefore, the accuracy of feature point detection directly affects the efficiency and accuracy of image spatial alignment fusion.

[0059] The ORB algorithm (Oriented Fast and Rotated Brief) is a commonly used feature point extraction algorithm. After extracting corner points, the ORB algorithm selects the most important corner points as feature points based on the response values ​​of the corner points in the FAST algorithm. However, the response values ​​of the corner points only consider the brightness changes of the surrounding pixels, which may lead to mismatches of useless corner points, thus affecting the accuracy and efficiency of image registration. Therefore, to improve the accuracy of image registration, corner points of composite insulators are extracted by analyzing their own characteristics, such as the texture information contained in the corner points. Specifically:

[0060] The ORB algorithm is used to extract the corner points of the edge contour and obtain the response value of each corner point;

[0061] It should be noted that the process of obtaining response values ​​using the ORB algorithm is a well-known technique and will not be elaborated upon here.

[0062] Furthermore, since composite insulators typically have a circular or elliptical appearance and their structure consists of layers of circular structures, the extracted corner points often exhibit a certain degree of symmetrical similarity. However, when defects such as surface damage, aging, or corrosion exist in localized areas of the composite insulator surface, the symmetry of the corner points is disrupted. In such cases, directly analyzing the texture information contained in the corner points based on their symmetry features for corner point matching may mistakenly identify the corner points corresponding to the defects as useless corner points, thus affecting matching accuracy and efficiency. Therefore, when there are local defects in the composite insulator, the texture information is disrupted. Since defects are usually sporadic and random, the texture information difference between corner points at defective locations and other symmetrical corner points is significant, while the difference between corner points not at defective locations and symmetrical corner points is smaller. By analyzing the extreme distribution range of the responsivity between symmetrical corner points and the differences in their responsivity, the saliency is calculated to indicate the saliency of the texture information contained in the corresponding corner points, thereby evaluating the impact of the corresponding corner points on the accuracy of subsequent image matching. The flowchart of the method for obtaining the saliency of each corner point in each image to be detected provided in this application embodiment is shown below. Figure 2 As shown, it specifically includes:

[0063] Obtain the center point of the smallest bounding rectangle of the edge contour, and denote it as the contour center;

[0064] A rectangle is constructed with the center of the outline as the center point and any corner point as the vertex. The corner points that are adjacent to the other vertices of the rectangle are recorded as the symmetrical corner points corresponding to any corner point.

[0065] The difference between the response values ​​of any corner point and its symmetrical corner points is denoted as the relative difference.

[0066] In this embodiment, the absolute value of the difference between the response values ​​of any corner point and its symmetrical corner points is calculated and denoted as the relative difference.

[0067] Calculate the sum of the relative differences between any corner point and all its symmetrical corner points, and denot it as the relative difference quantity;

[0068] Calculate the ratio of the response value at any corner point to the relative difference.

[0069] Calculate the range of the relative differences between any given corner point and all its symmetrical corner points;

[0070] The product of the range and the ratio is taken as the significance of any corner point;

[0071] In this embodiment, taking the saliency of the q-th corner point in a visible light image as an example, the calculation formula is as follows:

[0072]

[0073] Among them, F q X represents the saliency of the q-th corner point in a visible light image. q Let q be the response value of the q-th corner point in the visible light image. C represents the response value of the i-th symmetrical corner point corresponding to the q-th corner point in the visible light image. q ε is the range corresponding to the q-th corner point in the visible light image, and ε is a preset value greater than 0 to avoid the denominator being 0. In this embodiment, ε is 0.01. As for other implementation methods, the implementer can set it according to the actual situation. n is the number of all symmetrical corner points corresponding to the q-th corner point in the visible light image, where n is 3. Since the rectangle has 4 vertices, the number of symmetrical corner points corresponding to the q-th corner point is 3.

[0074] It should be noted that the greater the responsivity, the richer the information contained in the corner point, and the stronger the distinguishability and prominence of the corner point in the local area. The smaller the relative difference, the smaller the difference between the corner point and the symmetrical corner point, and the more likely the corner point is to represent the shape and texture information of the composite insulator in the image, rather than other interference points that are mistakenly extracted due to interference. The larger the range, the more likely the corner point is to have a defect. The greater the saliency, the more likely the corner point is to have a defect and the richer the texture information of the composite insulator it contains. The greater the influence of the corner point on the accuracy of subsequent image registration and defect identification of the composite insulator, the more it should be used for image matching. The saliency contains not only the texture feature information of the composite insulator surface but also the feature information of possible defects.

[0075] Thus, the salience of any corner point is obtained.

[0076] Step 3: Mark the corner points at the same positions as any corner point in different images to be detected as matching points corresponding to any corner point; analyze the distribution of gradient directions of all pixels in the neighborhood of the matching point to obtain the main direction; determine the defect degree of the matching point by using the differences and dispersion of pixel values ​​of adjacent pixels in the main direction in the neighborhood of the matching point; determine the matching degree of the any corner point by combining the difference in saliency between the any corner point and all its matching points with the defect degree.

[0077] Secondly, since visible light images, infrared images, and ultraviolet images reflect different properties of composite insulators, the corner point with a larger response value in the visible light image may not necessarily have a larger response value in the corresponding corner point in the infrared or ultraviolet image. That is, the response value of the same corner point may differ between different images, which may lead to corner points not matching or low matching accuracy between different images.

[0078] When defects such as damage occur on the surface of composite insulators, local discharges such as corona discharge and arc discharge will appear on the surface of the insulator in ultraviolet images. This usually produces ultraviolet radiation, and local ultraviolet spots or halos will appear near the surface of the insulator, representing the discharge area. That is, the radiation intensity of the discharge area is significantly different from the radiation intensity of the normal area, and the radiation intensity decreases continuously from the defect point outward. Secondly, the heat sources of composite insulators are mainly polarization loss, partial discharge, and leakage current. Usually, the ends of the air gap and carbonized channels will become "hot spots" due to the presence of partial discharge, which will deviate significantly from the surrounding parts. In infrared images, this will appear as a local temperature that is higher, and the temperature will decrease continuously from the center of the area outward.

[0079] Based on the above analysis, the corner points of the visible light images, infrared images, and ultraviolet images generated when there are surface defects in composite insulators are correlated. The matching degree is calculated by matching the corresponding corner points between different images to be detected.

[0080] First, we analyze the differences in saliency between corner points at the same location in different images to be detected, specifically:

[0081] The corner points at the corresponding positions of any corner point in each of the remaining images to be detected are denoted as the matching points corresponding to any corner point.

[0082] Calculate the ratio of the saliency between any corner point and each of its corresponding matching points, and record it as the relative ratio.

[0083] Calculate the sum of the relative ratios between any corner point and all its corresponding matching points, and perform a negative mapping on the sum.

[0084] In this embodiment, the negative mapping process is as follows: the absolute value of the difference between the preset value and the sum is taken as the result of the negative mapping; wherein, the preset value is 2, and in other implementations, the implementer can set it according to the actual situation; secondly, in order to avoid the denominator being 0 when calculating the ratio, a preset value greater than 0 is added to the denominator. Therefore, the preset value greater than 0 is 0.1, and in other implementations, the implementer can set it according to the actual situation.

[0085] It should be noted that if any corner point matches the matching point more closely, the ratio is closer to 1, and the sum is closer to 2, the result after negative mapping will be smaller. Taking the i-th corner point in the visible light image as an example, the matching points of the corner point in the ultraviolet and infrared images are obtained, and the saliency ratio between the i-th corner point in the visible light image and the corresponding matching point in the ultraviolet image is calculated. The saliency ratio between the i-th corner point in the visible light image and the corresponding matching point in the infrared image is also calculated.

[0086] Secondly, since the greater the difference in pixel value variation within the local area of ​​the matching point in the image to be detected, the more drastic and uneven the local changes at the matching point are, and the more complex and significant the changes in the local structure around the matching point are, the more likely the matching point is located in a complex defect region of the image. Therefore, the defect degree is calculated by analyzing the changes in pixel values ​​in the local area of ​​the matching point.

[0087] For the image to be detected where the matching point is located, a histogram is constructed for the gradient directions of all pixels in the neighborhood of the matching point, and the gradient direction corresponding to the maximum magnitude is recorded as the main direction.

[0088] In this embodiment, a histogram is constructed for the gradient directions of all pixels in the 7×7 neighborhood of the matching point. As another implementation method, the implementer can set it according to the actual situation. Furthermore, the construction of the histogram is a well-known technology and will not be described in detail here.

[0089] In the neighborhood of the matching point, select multiple adjacent pixels along the main direction, calculate the difference in pixel value between the two selected adjacent pixels, and record it as the pixel difference;

[0090] In this embodiment, five consecutive adjacent pixels are selected in the main direction within the 7×7 neighborhood of the matching point. As another implementation method, the implementer can set it according to the actual situation. Next, the absolute value of the difference between the pixel values ​​of the two selected adjacent pixels is calculated and recorded as the pixel difference.

[0091] Calculate the degree of dispersion of pixel differences between all selected adjacent pixels;

[0092] In this embodiment, the degree of dispersion is measured by calculating the information entropy of the pixel difference between all selected adjacent pixels. The calculation of information entropy is a well-known technique and will not be described in detail here. As other implementation methods, implementers can use other methods of the prior art, such as variance, standard deviation, etc. This embodiment does not impose any special restrictions on this.

[0093] For the matching point, calculate the sum of the pixel differences between all two adjacent pixels selected, and multiply the sum by the degree of dispersion as the defect degree of the matching point;

[0094] It should be noted that the greater the defect degree, the more random the pixel value of the pixels in the neighborhood of the matching point changes along the main direction, and the more likely the matching point is to be a corner point representing a defect.

[0095] Furthermore, based on the defect degree and the result of the negative mapping, the matching degree is determined, specifically as follows:

[0096] Calculate the product of the defect degrees of all the matching points corresponding to any given corner point;

[0097] The ratio between the product value of any corner point and the result of the negative mapping is used as the matching degree of any corner point in each image to be detected.

[0098] It should be noted that the greater the matching degree, the greater the influence of the corner point on subsequent image matching, and the more likely the corner point should be subject to subsequent feature point matching.

[0099] At this point, the matching degree of each corner point in each image to be detected is obtained.

[0100] Step 4: Based on saliency and matching degree, obtain the feature response of any corner point, filter all corner points in each image to be detected, extract feature points of each image to be detected, and perform image fusion.

[0101] Furthermore, based on the matching degree and the saliency, the feature response degree of each corner point is determined, specifically as follows:

[0102] The product of the saliency and the matching degree is used as the feature response of each corner point in each image to be detected;

[0103] It should be noted that the greater the feature responsivity, the more important the corner point is, the greater its impact on image registration, and the more likely the corner point should be selected for image matching.

[0104] Cluster the feature responsivity of all corner points in each image to be detected to obtain multiple clusters;

[0105] In this embodiment, the k-means clustering algorithm is used to obtain two clusters. The k-means clustering algorithm is a well-known technology and will not be described in detail here. As other implementation methods, implementers can use other methods of existing technology, such as hierarchical clustering algorithms. This embodiment does not impose any special restrictions on this.

[0106] Calculate the average feature responsivity of all corner points within each cluster, and select all corner points within the cluster with the largest average value as the feature points extracted from each image to be detected;

[0107] Feature matching is performed on all feature points in all images to be detected. After registration and alignment of all images to be detected, image fusion is performed on all images to be detected.

[0108] Affine transformation is used to convert all images to be detected into the same coordinate system, and a weighted fusion algorithm is used to fuse all images to be detected.

[0109] In this embodiment, the KNN (K-Nearest Neighbors) algorithm is used for feature matching. The feature matching algorithm based on KNN and the image fusion process through weighted fusion algorithm are well-known technologies and will not be described in detail here. As other implementation methods, implementers can use other methods of existing technology, such as the FLANN feature matching algorithm, etc. This embodiment does not impose any special restrictions on this.

[0110] The flowchart of the feature point extraction steps provided in this application embodiment is as follows: Figure 3 As shown.

[0111] It should be noted that the fused image of the composite insulator of the ultra-high voltage transmission line retains the high-resolution texture details of the visible light image, highlights the discharge abnormal area of ​​the ultraviolet image, and also incorporates the temperature rise hotspot information of the infrared image. When using the fused image to detect defects in the composite insulator, the accuracy of defect identification can be improved.

[0112] Based on the same inventive concept as the above methods, this application also provides a multispectral channel image spatial alignment and fusion device for avoiding image misalignment, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-described multispectral channel image spatial alignment and fusion methods for avoiding image misalignment.

[0113] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0114] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this application, without departing from the content of the technical solution of this application, shall fall within the protection scope of the technical solution of this application.

Claims

1. A multi-spectral channel image spatial alignment fusion method for avoiding image misregistration, characterized in that, The method includes the following steps: Images of composite insulators on transmission lines are acquired in different spectral channels, and the edge contours of the composite insulators in each image are extracted. Extract the corner points of the edge contour and obtain the response value of each corner point; analyze the symmetry relationship between each corner point and the other corner points on the edge contour and obtain the corresponding symmetrical corner points; The saliency of any corner point is calculated by considering the difference in response values ​​between any corner point and its symmetrical corner point in each image to be detected, as well as the extreme variations of the difference. The corner points at the same position corresponding to any corner point in different images to be detected are recorded as the matching points corresponding to any corner point; The distribution of gradient directions of all pixels in the neighborhood of the matching point is analyzed to obtain the main direction. The defect degree of the matching point is determined by the difference and dispersion of pixel values ​​of adjacent pixels in the main direction in the neighborhood of the matching point. Based on the difference in saliency between any corner point and all its matching points, and in conjunction with the defect degree, the matching degree of any corner point is determined. Based on saliency and matching degree, the feature response of any corner point is obtained. All corner points in each image to be detected are filtered, feature points of each image to be detected are extracted, and image fusion is performed.

2. The multi-spectral channel image spatial alignment fusion method for avoiding image misregistration of claim 1, wherein, The images to be detected include visible light images, ultraviolet images, and infrared images, and the visible light images are processed into grayscale.

3. The multispectral channel image spatial alignment and fusion method for avoiding image misalignment as described in claim 1, characterized in that, The step of obtaining the corresponding symmetrical corner points for each corner point includes: Obtain the center point of the smallest bounding rectangle of the edge contour, and denote it as the contour center; A rectangle is constructed with the center of the outline as the center point and any corner point as the vertex. The corner points that are adjacent to the other vertices of the rectangle are denoted as the symmetrical corner points of any corner point.

4. The multi-spectral channel image spatial alignment fusion method for avoiding image misregistration of claim 1, wherein, The calculation of the salience of any corner point includes: The difference between the response values ​​of any corner point and its symmetrical corner points is denoted as the relative difference. The sum of the relative differences between any given corner point and all its symmetrical corner points is denoted as the relative difference quantity. Calculate the ratio of the response value at any corner point to the relative difference. Calculate the range of the relative differences between any given corner point and all its symmetrical corner points; The significance is the product of the range and the ratio.

5. The multi-spectral channel image spatial alignment fusion method for avoiding image misregistration of claim 1, wherein, The step of obtaining the main direction includes: constructing a histogram of the gradient directions of all pixels in the neighborhood of the matching point in the image to be detected, and recording the gradient direction corresponding to the largest magnitude as the main direction.

6. The multi-spectral channel image spatial alignment fusion method for avoiding image misregistration of claim 1, wherein, Determining the defect degree of the matching point includes: In the neighborhood of the matching point, select multiple adjacent pixels along the main direction, calculate the difference in pixel value between the two selected adjacent pixels, and record it as the pixel difference; Calculate the degree of dispersion of the pixel difference between all selected adjacent pixels; For the matching point, the product of the sum of the pixel differences between all two adjacent pixels and the degree of dispersion is used as the defect degree of the matching point.

7. The multi-spectral channel image spatial alignment fusion method to avoid image misregistration of claim 1, wherein, Determining the matching degree of any corner point includes: Calculate the sum of the ratios of saliency between any corner point and all its corresponding matching points, and perform a negative mapping on the sum; The ratio of the product of the defect degrees of all the matching points corresponding to any corner point to the result of the negative mapping of any corner point is taken as the matching degree of any corner point.

8. The multispectral channel image spatial alignment and fusion method for avoiding image misalignment as described in claim 1, characterized in that, The feature response is the product of the saliency and the matching degree.

9. The multi-spectral channel image spatial registration fusion method to avoid image misregistration of claim 1, wherein, The step of extracting feature points from each image to be detected and performing image fusion includes: Cluster the feature responsivity of all corner points in each image to be detected to obtain multiple clusters; Calculate the average feature responsivity of all corner points within each cluster, and select all corner points within the cluster with the largest average value as feature points of each image to be detected; All images to be detected are matched and aligned using the feature points, and then image fusion is performed on all images to be detected.

10. A multispectral channel image spatial alignment fusion device to avoid image misregistration, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multispectral channel image spatial alignment and fusion method for avoiding image misalignment as described in any one of claims 1-9.

Images