Highlight Region Removal Method Based on Image Fusion

Abstract

A method for removing highlight regions based on image fusion includes the following steps: Detect feature points of a reference image and an auxiliary image using the SIFT algorithm to generate feature description vectors, and perform feature matching on the reference image and the auxiliary image using the nearest neighbor method; Obtain a perspective transformation matrix through the obtained feature matching points, and perform perspective transformation on the auxiliary image to adjust its perspective and size to be consistent with the reference image; According to the position of the highlight region detected in the reference image, use the corresponding region position of the perspectively transformed auxiliary image to perform texture filling and repair on it, thereby removing the highlights in the reference image. The present invention uses an improved OTSU two-dimensional threshold segmentation method to quickly detect highlight interference in the image, reduce the influence of noise, and make the detection of highlight regions in the reference image more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent power grid inspection, and particularly relates to a method for removing high-light regions based on image fusion. Background Art

[0002] With the continuous development of communication technology and artificial intelligence technology, the rapid construction of intelligent power grids has been driven. Intelligent inspection has gradually become an important auxiliary operation and maintenance means in the current unattended substation mode. As an important protection device in the power operation system, the relay secondary equipment protection pressure plate has a large number, and the traditional manual inspection has a large and complicated workload, and misdetection or missed detection often occurs. Therefore, using intelligent robots for inspection has also become a research hotspot. At present, the application of image processing-related technologies in the power system mainly focuses on primary equipment. In actual production, it is not easy to find the operation state switching of secondary equipment, which needs attention and real-time monitoring.

[0003] By observing the collected pressure plate images, it can be seen that since most protection cabinet doors use glass cabinet doors, there is local high-light interference in the pressure plate images under uneven illumination. In severe cases, the high-light interference will cause serious loss of texture information and make it impossible to identify, resulting in misdetection and missed detection of the pressure plate state.

[0004] There are mainly three reasons for the glass cabinet door reflection in the substation:

[0005] (1): The reflection formed by natural light passing through the window and shining on the glass cabinet door;

[0006] (2); The reflection formed by the indoor lighting on the glass cabinet door;

[0007] (3): The reflection formed by the camera flash on the cabinet door when the light is dim.

[0008] Detection of high-light regions based on the two-dimensional OTSU algorithm:

[0009] Among various algorithms for image threshold segmentation, the maximum inter-class variance method (OTSU) is widely used because of its simple calculation and stable performance. The traditional OTSU algorithm uses a bimodal histogram to statistically calculate the average gray value of the pixels in the image, and divides the image into two categories: the foreground region and the background region. When the variance of the two categories is the largest, the threshold is obtained. Since the pixels in the same region of an image have strong consistency and correlation in terms of position and gray level, the traditional OTSU algorithm only considers the gray-level information provided by the histogram and ignores the spatial position information of the image. Summary of the Invention

[0010] The present invention proposes a method for identifying the operating state of protection pressure plates based on image fusion, which is used in the intelligent inspection of secondary relay protection pressure plates in substations. This method uses a threshold segmentation method to detect the high-brightness area, and on this basis, performs image restoration to effectively remove the light and shadow interference existing in the relay protection pressure plate image, so as to better assist the intelligent inspection robot to identify the operating state of the pressure plate in the image and improve its anti-interference ability.

[0011] The technical solution adopted by the present invention is as follows:

[0012] The method for identifying the operating state of protection pressure plates based on image fusion includes the following steps:

[0013] Step 1: Detection of the high-brightness area of the protection pressure plate image:

[0014] Step 1.1: The intelligent robot inspects the relay protection pressure plate and takes pictures of the protection pressure plate image.

[0015] Step 1.2: Since the entire area of the protection pressure plate needs to be obtained during shooting, the taken protection pressure plate image is input into the computer image processing system to screen the protection pressure plate image, and the protection pressure plate images that do not capture the complete area are deleted to avoid affecting the subsequent detection results. The screened protection pressure plate images are compressed to make the computer process the images faster.

[0016] The computer image processing system is implemented by configuring plugins such as OpenCV and Python in the Microsoft Visual Studio development platform.

[0017] Step 1.3: Gray-scale processing is performed on the protection pressure plate image to expand the difference between the high-brightness area and the background area.

[0018] Step 1.4: Using the two-dimensional OTSU algorithm, the optimal threshold is obtained, and the gray-scale image of the protection pressure plate is segmented according to the optimal threshold, so as to quickly detect the high-brightness area in the image.

[0019] Step 2: Removal of the high-brightness area based on image fusion:

[0020] Step 2.1: Feature point detection:

[0021] The SIFT algorithm is used to detect feature points of the reference image and the auxiliary image, generate feature description vectors, and perform feature matching on the reference image and the auxiliary image using the nearest neighbor method.

[0022] Step 2.2: Perspective transformation based on the improved RANSAC algorithm:

[0023] Through the obtained feature matching points, the perspective transformation matrix is obtained, and perspective transformation is used to adjust the perspective and size of the auxiliary image to make it consistent with the referenceFigure 1 To.

[0024] Step 2.3, Image repair:

[0025] According to the position of the highlight area detected in the reference image, use the corresponding area position of the auxiliary image after perspective transformation to fill and repair its texture, and then remove the highlight in the reference image.

[0026] Step 3, Identification of the state of the protection pressure plate:

[0027] Step 3.1: Extraction of connected regions:

[0028] Perform color region screening, binarization, and morphological processing on the reference image after removing the highlight, and extract the connected regions in an 8-connected manner.

[0029] Step 2: Screening of the effective pressure plate area:

[0030] Based on the morphological characteristics, perform area, size, and shape analysis, and accurately extract the effective pressure plate area from the connected regions.

[0031] Step 3: Identification of the insertion / removal state of the pressure plate:

[0032] Identify the effective pressure plate area selected, and after identifying the insertion / removal state of the effective pressure plate, use the centroid coordinates to sort the effective pressure plates in the order from left to right and from top to bottom, and finally obtain a state sequence containing only 0 and 1.

[0033] The specific steps of Step 2.2 include the following steps:

[0034] Step 1: For the effective feature points extracted by the SIFT algorithm, use the nearest neighbor method for initial matching. The initially selected Euclidean distance threshold is 0.6. Divide the image into 4 regions equally, and judge whether the number of pairs of feature matching points in the currently divided 4 regions is greater than 4. If so, proceed to the next step; otherwise, add 0.1 to the Euclidean distance threshold and re-match.

[0035] Step 2: Select the 4 pairs of matching points with the smallest Euclidean distance from each of the 4 regions, a total of 16 pairs. Combine these 16 pairs of matching points in groups of 4, and sort them from 1, 2,..., N according to the ascending order of the sum of the Euclidean distances of the 4 pairs of matching points after combination, and select the first 50 groups.

[0036] Step 3: First, take the 4 pairs of matching points with the serial number 1 in the order of the serial number to calculate the transformation matrix H. Use the matrix H to check all pairs of matching points in the image, and judge whether the proportion of the number of inliers in the total number of pairs of matching points is greater than 50%. If it is greater than 50%, the currently calculated matrix H is the best transformation matrix; otherwise, select the next group of 4 pairs of matching points in the order of the serial number to calculate the transformation matrix H.

[0037] A method for identifying the operating state of protection pressure plates based on image fusion has the following technical effects:

[0038] 1) It can be widely applied to the intelligent inspection of relay protection pressure plates in substations, better assist the inspection robot in verifying the switching state of the pressure plates, improve the accuracy of identifying the switching state of relay protection pressure plates, reduce the labor intensity of inspection personnel, reduce misoperations in power grid operations, avoid economic losses, and ensure the safe and stable operation of the power grid.

[0039] 2) The main function of the present invention is to better assist the intelligent inspection robot in verifying the switching state of the pressure plates, improve the accuracy of identifying the switching state of relay protection pressure plates, reduce the labor intensity of inspection personnel, reduce misoperations in power grid operations, avoid economic losses, and ensure the safe and stable operation of the power grid.

[0040] 3) The detection of the high-light area of the reference image by the computer image processing technology of the present invention mainly includes: graying the collected reference image, and using the improved OTSU two-dimensional threshold segmentation method to quickly detect the high-light interference in the image, reduce the influence of noise, and make the detection of the high-light area in the reference image more accurate.

[0041] 4) The present invention uses the SIFT algorithm to detect the feature points of the reference image and the auxiliary image and matches them with the nearest neighbor method, introduces the improved RANSAC algorithm to remove the wrong matching points and obtain the optimal perspective transformation matrix, and uses the auxiliary image perspective transformation to the main image to repair the high-light area therein. Based on the image repair, the operating state is judged by the inclination angle of the pressure plate edge detection. The present invention can help better assist the intelligent inspection robot in identifying the operating state of the pressure plate in the image and improve its anti-interference ability. Description of the Drawings

[0042] Figure 1 is a two-dimensional histogram.

[0043] Figure 2 is a diagram of the perspective transformation process.

[0044] Figure 3 is a flow chart for detecting the high-light area of the protection pressure plate image.

[0045] Figure 4 is a flow chart for removing high-light based on image fusion.

[0046] Figure 5 is a diagram of the perspective transformation steps based on the improved RANSAC algorithm.

[0047] Figure 6 is a flow chart for identifying the state of the protection pressure plate.

[0048] Figure 7 is the overall structure diagram of the method of the present invention. Specific implementation mode

[0049] A method for identifying the operating state of a protection pressure plate based on image fusion, including:

[0050] I. Highlight area detection based on the OTSU optimization algorithm:

[0051] 1. Analysis of the characteristics of the pressure plate image:

[0052] The present invention uses a threshold segmentation method to detect the highlight area, and on this basis, image repair is carried out to lay a foundation for identifying the operating state of the pressure plate.

[0053] 2. Highlight area detection based on the two-dimensional OTSU algorithm:

[0054] In order to better segment the foreground and background and improve the anti-noise ability of the algorithm, the present invention increases the dimension of the traditional one-dimensional OTSU algorithm to two dimensions. The specific steps are as follows:

[0055] Step 1: Given an image I, assuming that the gray level of the image I(x, y) is L levels, then the neighborhood average gray level of the image I is also L levels.

[0056] Step 2: Let f(x, y) be the gray value of the pixel point (x, y), and g(x, y) be the average gray value within the K×K neighborhood of the central pixel point (x, y). Let f(x, y) = i and g(x, y) = j, and then a binary tuple (i, j) is formed.

[0057] Step 3: Let the number of occurrences of the binary tuple (i, j) be f ij , and calculate the probability density P ij corresponding to the binary tuple. P ij = f ij / N, where i, j = 1, 2,..., L, and N is the total number of image pixel points.

[0058] Step 4: Arbitrarily select a threshold vector (s, t) to divide the two-dimensional histogram of the image into 4 regions. Regions B and C represent the foreground and background of the image, and regions A and D represent noise points, as Figure 1 shown.

[0059] Step 5: Let the probabilities of the background and foreground appearances be ω 1 , ω 2 , and the corresponding mean vectors be μ 1 , μ 2 . The mean vector corresponding to the entire image is μ, and the formula is as follows:

[0060]

[0061] In the formula: ω 1is the probability of the background appearance, P ij is the probability density of the appearance of the binary tuple (i, j).

[0062]

[0063] In the formula: ω 2 is the probability of the foreground appearance, P ij is the probability density of the appearance of the binary tuple (i, j).

[0064]

[0065] In the formula: μ 1 is the mean vector corresponding to the background.

[0066]

[0067] In the formula: μ 2 is the mean vector corresponding to the foreground.

[0068]

[0069] In the formula: μ is the mean vector corresponding to the entire image.

[0070] Step 6: Use the discrete measure matrix S (s,t) to obtain the discrete measure tr(S (s,t) ) of the image. The formula is as follows:

[0071] S (s,t) = ω 1 (μ 1 - μ)(μ 1 - μ) T + ω 2 (μ 2 - μ)(μ 2 - μ) T (6)

[0072] In the formula: S (s,t) is the discrete measure matrix of the image.

[0073] tr(S (s,t) ) = ω 1 [(μ 1i - μ i ) 2 +(μ 1j - μ j ) 2 + ω 2 [(μ 2i - μ i ) 2 +(μ 2j - μ j ) 2(7)

[0074] where: tr(S (s,t) ) is the discrete measure of the image.

[0075] Step 7: The greater the discrete measure, the greater the between-class variance. The one corresponding to the maximum discrete measure is the optimal threshold (s * , t * ).

[0076] (s * , t * ) = arg max{tr(S (s,t) )} (8)

[0077] where: (s * , t * ) is the optimal threshold of the image.

[0078] After obtaining the optimal threshold through the above steps, use this threshold to perform binary processing on the grayscale image with 0 to 255 brightness levels, separating the foreground area and the background area. At this time, the foreground area is the highlight area.

[0079] II. Removal of highlight area based on image fusion:

[0080] Image Fusion is to synthesize two or more images collected about the same target into a new image, effectively improving the utilization rate of image information, so that the fused image has a more comprehensive and clear description of the target.

[0081] In view of the problems of over-reliance on the preset threshold and the existence of wrong matching point pairs when using the nearest neighbor method for feature matching after generating the feature description vector by the SIFT algorithm, the present invention introduces the RANSAC algorithm to further eliminate the wrong matching point pairs and complete the multi-view image feature matching and obtain the optimal perspective transformation matrix. At the same time, to avoid the drawbacks of the random selection of the traditional RANSAC algorithm and reduce unnecessary iteration times and time consumption, the present invention improves the RANSAC algorithm to make the effects of image perspective transformation and highlight removal better.

[0082] 1. Feature point detection:

[0083] Currently, the most classic in feature point description is the SIFT (Scale-invariant feature transform) algorithm, that is, the scale-invariant feature transform. This algorithm is widely used for feature point detection and generating feature description vectors because of its characteristics of being invariant to rotation, scale scaling, and brightness changes. The specific steps are as follows:

[0084] Step 1: Given the input image I(x, y), continuously downsample this image to obtain a series of images of different sizes. Sort these images from large to small and from bottom to top to form a pyramid model. Then, convolve each layer of the image with the two-dimensional Gaussian function G(x, y, σ) with continuously varying scales and the image I(x, y) to obtain the scale space L(x, y, σ).

[0085] L(x,y,σ) = G(x,y,σ) * I(x,y) (9)

[0086]

[0087] where: * represents the convolution operation, and σ is the scale.

[0088] Step 2: Let each layer of multiple images in the scale space be collectively referred to as a group. Subtract adjacent layers of images in the same group to obtain the difference-of-Gaussians image. Compare each pixel point of the difference-of-Gaussians image of each layer in the same group, except for the top and bottom layers, with 8 pixels in its own layer and 9×2 pixels in the upper and lower adjacent layers, a total of 26 pixels. When the pixel value of this point is the maximum or minimum, this pixel point is an extreme point. The formula for the difference-of-Gaussians function is as follows:

[0089] D(x,y,σ) = L(x,y,kσ) - L(x,y,σ) (11)

[0090] where k is a fixed coefficient.

[0091] Step 3: Since the detected extreme points are extreme points in the discrete space and not the true feature points in the continuous space, it is necessary to perform curve fitting on the difference-of-Gaussians function in the scale space to recalculate the coordinates of the extreme points, that is, expand the difference-of-Gaussians function in the scale space using the Taylor formula:

[0092]

[0093] where D(X) is the difference-of-Gaussians function, and X = (x, y, σ) T 。

[0094] Take the derivative and set the equation equal to zero to obtain the offset of the extreme point:

[0095]

[0096] The value of the corresponding extreme point equation is:

[0097]

[0098] where is the value of the extreme point equation corresponding to the offset.

[0099] The new coordinates of the generated extreme points are obtained by adding the offset to the original extreme point coordinates. The pixel values of the generated new coordinates are compared with the set contrast threshold, and the extreme points with low contrast are removed. At this time, the remaining extreme points are the feature points.

[0100] Step 4: To make the descriptor rotation-invariant, it is necessary to assign a direction to each feature point. For this purpose, the gradient method is used to obtain the gradient magnitude and direction of the pixel points in the neighborhood of the feature point in the image. The formulas for the gradient magnitude m(x, y) and direction θ(x, y) are as follows:

[0101]

[0102] θ(x,y)=tan -1 ((L(x,y + 1)-L(x,y - 1)) / (L(x + 1,y)-L(x - 1,y))) (16)

[0103] Among them, the scale used for L is the scale where each feature point is located.

[0104] Then, statistical analysis is performed using a two-dimensional histogram, and the direction with the highest amplitude in the histogram is used as the main direction of the feature point. To enhance the robustness of the matching, the directions where the amplitude is greater than 80% of the main direction amplitude are retained as the secondary directions of the feature point.

[0105] Step 5: A window of 16×16 pixels is taken with the feature point as the center, and the window is divided into 4×4 sub-domains. The gradient accumulation amplitude in 8 directions on each sub-domain is statistically analyzed using a gradient direction histogram. At this time, each sub-domain can be represented by an 8-dimensional feature description vector. Finally, each feature point has a 4×4×8 = 128-dimensional feature vector to describe it. To make it illumination-invariant, normalization processing is performed on this 128-dimensional feature vector and a threshold is taken to limit the gradient amplitude, which can effectively reduce the influence of uneven illumination on the matching result.

[0106] When the SIFT feature vectors of the reference image and the auxiliary image are generated, the nearest neighbor method is used to match them, that is, a ratio threshold is set. If the ratio of the nearest Euclidean distance to the second nearest Euclidean distance of the two feature point description vectors is less than this ratio threshold, it is considered that the two feature points are correctly matched.

[0107] 2. Perspective transformation based on the improved RANSAC algorithm:

[0108] Since it is necessary to keep the perspective and size of the auxiliary image consistent with those of the reference image during the high-light removal image fusion,

[0109] Therefore, it is necessary to adjust the auxiliary image using perspective transformation. The perspective transformation formula is as follows:

[0110]

[0111] Among them, (x', y') are the coordinate values of the feature matching points of the reference image, (u, v) are the coordinate values of the corresponding feature matching points of the auxiliary image, S is the transformation coefficient between the images, and H is a 3×3 transformation matrix, that is:

[0112]

[0113] Among them, Rotation, scaling, and distortion transformations can be performed on the image, and T 2 = [a 13 a 23 T Translation transformation can be performed on the image, and T 3 = [a 31 a 32 can generate a perspective transformation of the image, as Figure 2 shown.

[0114] Since the coordinates of the feature matching points of the reference image and the auxiliary image have been obtained using the SIFT algorithm, only 4 pairs of feature matching points need to be randomly selected to obtain the transformation matrix H, and the perspective transformation can be performed on the auxiliary image using the transformation matrix. The formula for the pixel coordinates (x, y) of the transformed auxiliary image is as follows:

[0115]

[0116] Among them, (x, y) are the pixel coordinates of the transformed auxiliary image.

[0117] Since the nearest neighbor method relies too much on the currently preset threshold when matching two feature points, the size of the threshold cannot be accurately judged. When the set threshold is large, there will be more incorrect matching point pairs; when the set threshold is small, although the number of incorrect matching point pairs will be reduced, the number of matching point pairs will be significantly reduced, seriously affecting the optimal selection of the transformation matrix H. Therefore, the RANSAC algorithm needs to be introduced to further remove the incorrect matching point pairs and obtain the optimal transformation matrix.

[0118] The idea of the RANSAC algorithm is to randomly select a set of random subsets from the data to fit and estimate the model, and use the estimated model to test other data. If a certain data is applicable to the estimated model, it is classified as an inlier. If there are enough points classified as the assumed inliers, the estimated model is considered reasonable enough. Then, all the assumed inliers are used to re-estimate the model, and the model is evaluated by the error rate between the estimated inliers and the model. Such a process is repeated a fixed number of times, and each generated model is either discarded because there are too few inliers or selected because it is better than the existing model.

[0119] Since only 4 pairs of matching points are required for calculating the transformation matrix H, the present invention proposes an improved RANSAC algorithm, and the steps are as follows:

[0120] Step 1: First, perform initial matching on the effective feature points extracted by the SIFT algorithm using the nearest neighbor method, and the initially selected Euclidean distance threshold is 0.6.

[0121] Step 2: Divide the image into 4 equal regions, and determine whether the number of feature matching point pairs in each of the currently divided 4 regions is greater than 4. If so, proceed to the next step; otherwise, add 0.1 to the Euclidean distance threshold and return to the previous step for re-matching.

[0122] Step 3: Select 4 pairs of matching points with the smallest Euclidean distance from each of the 4 regions, for a total of 16 pairs.

[0123] Step 4: Combine these 16 pairs of matching points in groups of 4 pairs, and sort them from 1, 2, …, N according to the ascending order of the sum of the Euclidean distances of the 4 pairs of matching points after combination, and select the first 50 groups.

[0124] Step 5: First, take 4 pairs of matching points with the serial number 1 in the order of the serial numbers to calculate the transformation matrix H.

[0125] Step 6: Use the matrix H to check all the matching point pairs in the image. When the proportion of the inlier points in the total number of matching point pairs is greater than 50%, it is considered that the currently calculated matrix H is the optimal transformation matrix; otherwise, return to the previous step and select the next group of 4 pairs of matching points in the order of the serial numbers to calculate the transformation matrix H.

[0126] Use the optimal transformation matrix to adjust the auxiliary graph, and mask its corresponding area in the highlight area of the reference graph to obtain the reference graph with highlights removed.

[0127] III. Identification of the withdrawal and insertion states of the pressure plate:

[0128] In order to accurately identify the state of the pressure plate in the reference graph after highlight removal, the present invention adopts a relay protection pressure plate state recognition method based on image processing and morphological feature analysis. First, to improve the accuracy of the overall feature extraction of the pressure plate image, perform color region screening, binarization, and morphological processing on the reference graph after highlight removal, and extract the connected regions in an 8-connected manner; then perform area, size, and shape analysis based on morphological features to accurately extract the effective pressure plate regions from all regions; finally, perform state recognition on the effective regions according to the direction angles during the withdrawal and insertion states of the pressure plate, and at the same time use the centroid coordinates to sort the effective pressure plates to obtain all the effective pressure plate state sequences.

[0129] 1. Extraction of the connected regions of the pressure plate image:

[0130] To better reflect the overall and local feature information of the image and accurately extract the connected regions of the pressing plate image, the following steps are taken:

[0131] Step 1: Since the reference image after highlight removal is a color image, the overall effective pressing plate in the image is red and yellow, the spare pressing plate is camel and red, and the background area is white. Therefore, the red and yellow regions can be screened out by setting certain RGB thresholds. Considering that other components, markings, etc. that may exist on the picture will cause incorrect screening, through a large number of experiments, it is known that the difference between the maximum and minimum values of the R, G, and B channels of the red and yellow pixel points is not less than 40. Therefore, the pixels in this part of the image that are not less than 40 are retained, and the remaining regions are set to equal values of R, G, and B, that is, they are changed to black.

[0132] Step 2: To improve the operation speed, the reference image after screening the red and yellow regions is grayscale processed, the OTSU algorithm is used to obtain the binarization threshold, and the grayscale image is binarized using this threshold.

[0133] Step 3: Since some uneven edges will be generated after the image is binarized, and there will be holes at the connection of the pressing plates, which seriously affect the effect of subsequent feature extraction. Therefore, it is necessary to perform morphological processing on the binary image, use dilation and erosion operations to fill the holes, and extract the connected regions in an 8-connected manner (if a pixel is connected to its neighboring pixels in the up, down, left, right, upper left, lower left, upper right, or lower right, they are considered connected), and number the N connected regions obtained as 1, 2, …, N.

[0134] 2. Screening of the effective pressing plate area:

[0135] To accurately screen out the effective pressing plate area, morphological feature analysis is carried out from three aspects: area, size, and shape. The specific steps are as follows:

[0136] Step 1: Area analysis. Considering that there may be invalid spare pressing plates, markings, etc. in the image after the connected regions are extracted, it can be observed that the area of the connected regions where this part is located is relatively small. Therefore, an area threshold V area-thre is set to remove the interference regions. The formula is as follows:

[0137]

[0138] where the threshold V area-thre is obtained by multiplying the average area of the top 5 pixel regions in terms of pixel area size in the binary image by 0.3, and V area (i) is the area of the i-th region arranged from largest to smallest in terms of region area.

[0139] According to formula (20), it is determined that the pixel area of the connected region is greater than the threshold Varea-thre The area is the alternative effective pressure plate area, otherwise it is an interference area.

[0140] Step 2: Dimension analysis. Considering that the effective pressure plate in the image after connected region extraction has a certain dimension, in the X and Y directions, there is a certain ratio between the boundary length of the effective pressure plate area and the pixel size of the image. Therefore, using the image pixels P X 、P Y in the X and Y directions to set the pixel size thresholds X width-thre 、Y width-thre as follows:

[0141]

[0142] According to formula (21), it is determined that in the X and Y directions, the area where the boundary length of the connected region is greater than the corresponding threshold is the alternative effective pressure plate area, otherwise it is an interference area.

[0143] Step 3: Shape analysis. Considering that the effective pressure plate in the image after connected region extraction has a certain shape, the effective pressure plate area has a certain equivalent length-width ratio in the image. At the same time, to eliminate other interference information with similar shapes in the image, the equivalent length-width ratio threshold S ratio-thre is set to 2 < S ratio-thre < 5. If the equivalent length-width ratio of the connected region is within this threshold, it is determined that this area is the alternative effective pressure plate area, otherwise it is an interference area.

[0144] Step 4: Search all connected regions in the image, and repeat steps 1 - 3 until the search of the Nth region is completed. The regions that are simultaneously determined to be alternative effective pressure plate areas are the final effective pressure plate areas.

[0145] 3. Recognition of the insertion / withdrawal state of the pressure plate:

[0146] To accurately recognize the insertion / withdrawal state of the pressure plate in the image of the protection pressure plate after highlight removal through the effective pressure plate area screening, the present invention uses the direction angle when the pressure plate is inserted / withdrawn, that is, the direction angle is ±90° when the pressure plate is inserted, and the direction angle is ±45° when it is withdrawn, and a margin of ±10° is set. The criterion formula is as follows:

[0147]

[0148] Among them, the insertion state is marked as 1, and the withdrawal state is marked as 0. After recognizing the insertion / withdrawal state of the effective pressure plate, the centroid coordinates are used to sort the effective pressure plates in the order from left to right and from top to bottom, and finally a state sequence containing only 0 and 1 is obtained.

[0149] As can be seen from the above, based on the images of protection pressure plates taken by intelligent robot patrol inspection, the present invention uses a threshold segmentation method to detect the high-light areas in the images, and on this basis, uses an image fusion method to repair the images, eliminating the situation where the light source in the substation site interferes with the identification of the pressure plate state. Finally, based on the repaired images, the inclination angle of the pressure plate edge detection is used to judge its operating state. It better assists the patrol inspection robot in checking the switching state of the pressure plate, improves the accuracy of identifying the switching state of the relay protection pressure plate, reduces the labor intensity of the patrol inspection personnel, reduces misoperations in power grid operations, avoids economic losses, and ensures the safe and stable operation of the power grid.

Claims

1. Highlight region removal method based on image fusion, characterized in that it includes the following steps: Step 1: Use the SIFT algorithm to detect feature points in the reference image and the auxiliary image, generate feature description vectors, and use the nearest neighbor method to perform feature matching on the reference image and the auxiliary image; Step 2: Obtain the perspective transformation matrix through the obtained feature matching points, and use perspective transformation to adjust the perspective and size of the auxiliary image to make it consistent with the reference image; Step 3: According to the position of the highlight region detected in the reference image, use the corresponding region position of the perspective-transformed auxiliary image to perform texture filling and repair on it, thereby removing the highlight in the reference image; The detection of the above highlight region position is realized by the two-dimensional OTSU algorithm, including the following steps: Step 3.1: Given an image I, assuming that the gray level of the image I(x, y) is L levels, then the average gray level of the neighborhood of the image I is also L levels; Step 3.2: Let f(x, y) be the gray value of the pixel point (x, y), and g(x, y) be the average gray value within the K×K neighborhood of the central pixel point (x, y); let f(x, y)=i, g(x, y)=j, and then a binary group (i, j) is formed; Step 3.3: Let the number of occurrences of the binary tuple (i, j) be f ij , and calculate the probability density P corresponding to the binary tuple ij , P ij = f ij / N, where i, j = 1, 2,..., L, and N is the total number of image pixel points; Step 3.4: Arbitrarily select a threshold vector (s, t), divide the two-dimensional histogram of the image into 4 regions, regions B and C represent the foreground and background of the image, and regions A and D represent noise points; Step 3.5: Assume that the probabilities of the background and foreground occurrences are ω 1 , ω 2 , and the corresponding mean vectors are μ 1 , μ 2 ; the mean vector corresponding to the entire image is μ, and the formula is as follows: where: ω 1 is the probability of the background appearance, P ij is the probability density of the appearance of the binary tuple (i, j); Where: ω 2 is the probability of the foreground appearance, P ij is the probability density of the appearance of the binary tuple (i, j); where: μ 1 is the mean vector corresponding to the background; Where: μ 2 is the mean vector corresponding to the foreground; where: μ is the mean vector corresponding to the entire image; Step 3.6: Using the discrete measure matrix S (s,t) to obtain the discrete measure tr(S (s,t) ) of the image. The formula is as follows: S (s,t) = ω 1 (μ 1 - μ)(μ 1 - μ) T + ω 2 (μ 2 - μ)(μ 2 - μ) T (6) where: S (s,t) is the discrete measure matrix of the image; where: tr(S (s,t) ) is the discrete measure of the image; Step 3.7: The greater the discrete measure, the greater the between-class variance, and the optimal threshold (s * , t * ) is the one corresponding to the maximum discrete measure; (s * ,t * ) = argmax{tr(S (s,t) )} (8) Where: (s * , t * ) is the optimal threshold of the image; After obtaining the optimal threshold through the above steps, use this threshold to perform binary processing on the gray image with 0 to 255 brightness levels, separate the foreground region and the background region, and the foreground region at this time is the highlight region.

2. The highlight region removal method based on image fusion according to claim 1, characterized in that: The specific steps of Step 2 include the following steps: Step 2.1: For the effective feature points extracted by the SIFT algorithm, use the nearest neighbor method for initial matching, and the initially selected Euclidean distance threshold is 0.6; divide the image into 4 equal regions, and judge whether the number of pairs of feature matching points in each of the currently divided 4 regions is greater than 4. If so, proceed to the next step, otherwise add 0.1 to the Euclidean distance threshold and re-match; Step 2.2: Select the 4 pairs of matching points with the smallest Euclidean distance from each of the 4 regions, a total of 16 pairs. Combine these 16 pairs of matching points in groups of 4 pairs, and sort them from 1, 2,..., N according to the sum of the Euclidean distances of the 4 pairs of matching points after combination, and select the first 50 groups; Step 2.3: First, take the 4 pairs of matching points with the serial number 1 in order of the serial number to calculate the transformation matrix H, use the matrix H to check all the matching point pairs in the image, and judge whether the proportion of the number of inliers in the total number of matching point pairs is greater than 50%. If it is greater than 50%, the currently calculated matrix H is the optimal transformation matrix, otherwise select the next group of 4 pairs of matching points in order of the serial number to calculate the transformation matrix H.

Images