Vision-based flying electric power inspection robot power line segmentation method and system

Abstract

The invention provides a vision-based flying electric power inspection robot power line segmentation method and system, relates to the technical field of electric power inspection, and aims to perform deep enhancement processing on a collected image through a finely designed multi-scale algorithm, improve the definition and contrast of power line features, improve the image quality, and utilize an advanced local binary pattern (LBP) technology to improve the image quality. Unique texture information of the power line is accurately captured, the power line and the background are effectively distinguished, and redundant data interference is reduced. On the basis of feature extraction, information entropy and edge density analysis of a statistical histogram are further combined, intelligent dimension reduction and key information supplement of features are achieved, key information needed by segmentation is reserved, and calculation complexity is greatly reduced. And finally, a fuzzy C-Means (FCM) clustering algorithm is adopted to efficiently segment the preprocessed features, and the algorithm can flexibly process the fuzziness of the boundary of the power line and ensure that the power line is accurately and rapidly identified and segmented under the finite computing power.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power inspection, and in particular to a method and system for segmenting electric power lines of a flying electric power inspection robot based on vision. Background Art

[0002] With the continuous expansion and complexity of the power network, higher requirements are placed on the inspection of power lines. The traditional manual inspection method is not only inefficient, but also has high safety risks and difficulty in covering all areas. Therefore, the emergence of flying power inspection robots has become an effective way to solve this problem. They can fly autonomously to the vicinity of power lines, and through the high-definition cameras, infrared sensors and other equipment they carry, they can conduct all-round and no-dead-angle inspections of power lines, and promptly discover and report potential safety hazards, such as line aging, damage, foreign objects hanging on the line, etc., to ensure the stable operation of the power network.

[0003] During the flying power inspection process, the robot needs to be able to automatically identify and track the power lines to ensure the continuity and accuracy of the inspection. Power line segmentation technology is the key to achieving this goal. This technology analyzes and processes the images collected by the robot through algorithms such as image processing, feature extraction and pattern recognition, separates the power lines from the complex background, and extracts the outline and location information of the power lines. Based on this information, the robot can adjust its flight posture and speed in real time, maintain a certain distance and angle from the power lines, and realize automatic line inspection.

[0004] Although power line segmentation technology plays an important role in flying power inspection robots, the current technology still has some shortcomings. On the one hand, since power lines are usually located in complex natural environments, such as mountains, rivers, cities, etc., the background is complex and changeable, which brings challenges to the accurate segmentation of power lines. On the other hand, the power lines themselves may also be blocked, reflected, broken, etc., which further increases the difficulty of segmentation. In addition, the existing power line segmentation algorithms still need to be improved in terms of real-time performance, accuracy and robustness to better meet the actual needs of flying power inspection robots. Therefore, researching and developing more efficient and accurate power line segmentation technology is of great significance to promoting the development of flying power inspection robots. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to propose a power line segmentation method of a flying power inspection robot based on vision, comprising:

[0006] Step 1: Collect original images of power lines through a flying power inspection robot;

[0007] Step 2: The original image is processed using the MSR algorithm obtained by weighted fusion of Retinex algorithms at different scales to obtain a reflected image;

[0008] Step 3: setting the size and sliding step of the sliding window, sliding the sliding window on the reflected image, dividing the image in the sliding window into a plurality of sub-blocks according to a preset size, calculating the multi-order entropy and the edge density of the sub-block for each sub-block, and taking the multi-order entropy and the edge density of the sub-block as the local features of the sub-block;

[0009] Step 4: According to the local features and the pre-set K cluster centers, determine the label of each pixel in the sliding window. The number of labels is the same as the number of cluster centers. One label corresponds to one cluster center, and all labels are different. The K labels include one or more power line labels and one or more background labels.

[0010] Step 5: Repeat steps 2 to 4 by sliding the sliding window in the reflected image to obtain the label of each pixel in the reflected image;

[0011] Step 6: During the sliding process of the sliding window, there is a situation where a pixel has different labels in different sliding windows. For the situation where a pixel has multiple different labels, a voting mechanism or the principle of the highest average membership is used to determine the final label of the pixel, and then the final labels of all pixels in the reflected image are determined, that is, the power lines or the background to which each pixel belongs are obtained, and then the power lines in the original image are segmented according to the final label of each pixel in the reflected image to obtain the segmentation result.

[0012] Optionally, step 2 is specifically implemented by the following formula:

[0013]

[0014] Among them, R(x,y) represents the reflected image, I(x,y) represents the original image, K represents the total number of Gaussian kernel functions, G k (x, y) is the kth Gaussian kernel function, G k (x,y)*I(x,y) represents the convolution operation of the kth Gaussian kernel function and the original image.

[0015] Optionally, the multi-order entropy of the sub-block in step 3 is calculated by the following formula:

[0016]

[0017] Where H is the number of sub-blocks in the reflected image, n is the entropy order, h(i) represents the frequency of the i-th sub-block in the LBP histogram, and En (X) is the multi-order entropy.

[0018] Optionally, the edge density of the sub-block in step 3 is calculated by:

[0019] The sub-blocks are calculated by the edge detection method to obtain the target value of each pixel in the sub-block. The target value represents the grayscale change degree of the pixel. The target value of each pixel is compared with the preset grayscale threshold, and the pixel whose target value exceeds the grayscale threshold is obtained and regarded as the edge point. Then the total number of edge points is obtained. According to the total number of edge points and the total number of pixels, the edge density is calculated. Specifically, it is calculated by the following formula:

[0020]

[0021] Among them, N e Represents the total number of edge points, N g Indicates the total number of pixels.

[0022] Optionally, step 4 specifically includes:

[0023] Step 4.1: According to the preset number of cluster centers K, randomly initialize K cluster centers, and according to the number of cluster centers and the number of local features in the sliding window, initialize the membership matrix, the rows of the membership matrix correspond to the cluster centers, the columns correspond to the local features in the sliding window, the parameters in the initialized membership matrix are random, and the parameters in the initialized membership matrix are expressed as u ij ,u ij It represents the membership of the i-th feature in the local features to the j-th cluster center, and the sum of the membership of each local feature to all clusters is 1;

[0024] Step 4.2: Set the initialized membership matrix as the current membership matrix, set the total number of iterations, set the initial number of iterations to 1, and set the initial number of iterations as the current number of iterations;

[0025] Step 4.3: Based on the fuzzy C-means clustering algorithm, update the current membership matrix to obtain the updated current membership matrix, which is specifically implemented by the following formula:

[0026]

[0027] Among them, v′ ij is the membership of the i-th feature to the j-th cluster center in the updated local feature, m is the fuzzy factor used to control the fuzziness of clustering, x i is the i-th feature in the local feature, v j is the jth cluster center, v k is the kth cluster center, ║·║ is the distance metric;

[0028] Step 4.4: Update each cluster center, which is implemented by the following formula:

[0029]

[0030] Among them, v′ j is the updated j-th cluster center, n is the number of local features in the sliding window;

[0031] Step 4.5: The current iteration number is increased by one;

[0032] Step 4.6: Determine whether the current number of iterations is less than the total number of iterations. If the current number of iterations is not less than the total number of iterations, obtain the final membership matrix. If the current number of iterations is less than the total number of iterations, return to step 4.3.

[0033] Step 4.7: For each local feature, in the final membership matrix, obtain all the memberships corresponding to the local feature, obtain the maximum value among all the memberships, and then determine the cluster center corresponding to the maximum value, and use the label of the cluster center as the label of the sub-block corresponding to the local feature, where the feature of each pixel in the sub-block is the label of the cluster center, and then obtain the label of each pixel in all sub-blocks, that is, the label of all pixels in the sliding window.

[0034] Optionally, a voting mechanism is used in step 6 to determine the final label, including:

[0035] Get all the labels corresponding to the pixel, determine the number of each label, get the label with the largest number, and use it as the final label of the pixel.

[0036] Optionally, in step 6, the final label is determined by adopting the principle of the highest average membership, which specifically includes the following steps:

[0037] For different labels, obtain their membership in the final membership matrix corresponding to each sliding window respectively. For the same label, calculate the average of all memberships to obtain the average membership of each label, and obtain the label with the largest average membership as the final label of the pixel.

[0038] A vision-based flying power inspection robot power line segmentation system, which is used to implement the vision-based flying power inspection robot power line segmentation system, including an onboard computer, a flight controller, a visual sensor, a ground station, a six-rotor flight mechanism, a walking mechanism, an efficient data transmission module and a walking controller;

[0039] The visual sensor is used to collect original images of the power line, and is also used to collect images and videos of the surrounding environment of the power line; the data collected by the visual sensor is transmitted to the onboard computer through an efficient data transmission module;

[0040] The onboard computer is used to receive the original image of the power line, and is also used to implement the power line segmentation method of the flying power inspection robot based on vision, and transmit the segmentation result to the flight controller or the ground station through the efficient data transmission module; it is also used to send the instructions for adjusting the flight attitude, speed, and walking mechanism state to the flight controller through the efficient data transmission module; it is also used to send the walking instructions for adjusting the walking speed and direction to the walking controller through the efficient data transmission module;

[0041] The flight controller is used to receive instructions for adjusting the flight attitude, speed, and state of the walking mechanism; it is also used to send instructions for adjusting the flight attitude, speed, and state of the walking mechanism to control the six-rotor flight mechanism; it receives flight status data sent by the six-rotor flight mechanism, and transmits the flight status data to the onboard computer through an efficient data transmission module; it is also used to receive data collected by other sensors;

[0042] The six-rotor flight mechanism is used to receive instructions for adjusting the flight attitude, speed, and state of the walking mechanism, and execute the instructions for adjusting the flight attitude, speed, and state of the walking mechanism; and is also used to collect flight status data and send the flight status data to the flight controller;

[0043] The walking controller is used to receive walking instructions for adjusting walking speed and direction, send the walking instructions for adjusting walking speed and direction to the walking mechanism, receive walking status data sent by the walking mechanism, and transmit the walking status data to the onboard computer or the ground station through an efficient data transmission module;

[0044] The walking mechanism is used to receive walking instructions for adjusting walking speed and direction, and execute the walking instructions for adjusting walking speed and direction, collect walking state data through sensors, and send the walking state data to a walking controller.

[0045] The beneficial effects of adopting the above technical solution are:

[0046] Improve image quality. This invention introduces a multi-scale Retinex (MSR) algorithm to enhance the image, which not only significantly improves the clarity and contrast of the power line features in the image, but also achieves a delicate control of the balance between image details and brightness. Compared with the traditional single-scale Retinex (SSR) algorithm, the MSR algorithm effectively avoids the problems of over-enhancement and detail loss, and provides higher quality image data for subsequent feature extraction and segmentation.

[0047] Reduce the computational complexity of feature extraction. The local binary pattern (LBP) is used in combination with statistical histogram information entropy and edge density analysis to efficiently extract and reduce the dimensionality of image texture features. By measuring the information of the LBP histogram through multi-order entropy, the computational complexity of the feature vector is reduced while retaining the key features required for segmentation. The introduction of edge density further improves the sufficiency and anti-interference ability of texture features, providing more reliable feature input for subsequent clustering segmentation.

[0048] Accurately segment power lines. The fuzzy C-means (FCM) clustering algorithm is used to segment the processed features, achieving accurate identification and segmentation of power lines. The FCM algorithm optimizes the clustering effect by minimizing the objective function, which can better handle fuzzy clustering problems and improve the accuracy and robustness of power line segmentation. In addition, through window processing and clustering result mapping technology, local information and overlapping window problems in the image are effectively handled, further improving the segmentation effect.

[0049] Improve inspection efficiency and safety. The method of the present invention is applied to a highly integrated flying power inspection robot system to achieve real-time segmentation and defect detection of power lines. The system combines the advantages of a six-rotor flying mechanism and a walking mechanism, and can operate flexibly in a complex and changeable power inspection environment, thereby improving inspection efficiency and accuracy. At the same time, it reduces the need for manual inspections, reduces safety risks, and provides a strong guarantee for the stable operation of the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram of the flow of a power line segmentation method of a flying power inspection robot based on vision in an embodiment of the present invention;

[0051] Figure 2 Schematic diagram of the structure of a power line segmentation system of a vision-based flying power inspection robot in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0053] In view of the problems existing in the prior art, the present invention provides a method and system for segmenting power lines of a flying power inspection robot based on vision, which realizes accurate recognition and segmentation of power lines through multi-scale image enhancement, local binary pattern (LBP) texture feature extraction, feature dimension reduction combining statistical histogram information entropy and edge density, and fuzzy C-means (FCM) clustering algorithm. The image is enhanced by the MSR algorithm to improve the clarity and contrast of the power line features; the unique texture of the power line is captured by LBP, and the feature recognition capability is enhanced by combining multi-order entropy and edge density analysis. The FCM clustering algorithm is used to segment the processed features to ensure accurate recognition of the power lines. In addition, the present invention also designs a flying power inspection robot system, which is based on a six-rotor and combines a walking mechanism to realize walking along the power line, and executes the above segmentation method through an onboard computer to realize real-time segmentation and inspection of the power line. The invention improves the automation level of power inspection and provides strong support for the maintenance and management of power lines.

[0054] Specifically, the invention provides a power line segmentation method for a flying power inspection robot based on vision, combined with Figure 1 , which may include the following steps:

[0055] Step 1: Collect original images of power lines through a flying power inspection robot;

[0056] Due to the limitations of the single-scale Retinex algorithm in grayscale preservation, this paper innovatively introduces a multi-scale Retinex (MSR) algorithm. This strategy cleverly combines multiple Retinex algorithms at different scales, and through weighted integration, it not only significantly enhances the clarity and contrast of the power line features in the image, but also achieves a subtle control of the balance between image details and brightness.

[0057] The Single Scale Retinex (SSR) algorithm is an image enhancement technology based on the Retinex theory. Its core idea is to improve the visual effect of the image by adjusting the contrast and brightness of the image while retaining the image detail information. The Retinex theory was proposed by Land and McCann in the 1960s. It believes that the color of an object is determined by the object's ability to reflect long-wave (red), medium-wave (green) and short-wave (blue) light, rather than by the absolute value of the reflected light intensity. This color constancy makes the color of the object consistent without being affected by the non-uniformity of illumination.

[0058] Among them, the basic steps of the single-scale Retinex algorithm include:

[0059] Step A1: Convert the image to the logarithmic domain. Convert the original image from the linear space to the logarithmic space to obtain the image after conversion to the logarithmic domain. This step is because the logarithmic form is closest to the properties of the human process of perceiving brightness, and it also simplifies subsequent calculations. The specific formula is as follows:

[0060] logI(x,y)=logL(x,y)+logR(x,y)

[0061] Among them, I(x,y) represents the original image, that is, the image captured directly by the camera or other image acquisition device, which contains all the information in the scene, including illumination, reflection, color, texture, etc. The original image is an image without any processing or enhancement, which reflects the true situation of the scene (although it may be affected by factors such as the performance of the acquisition device and environmental conditions); L(x,y) represents the illumination image, which is the part of the original image that reflects the illumination distribution. It is not directly observable, but is estimated from the original image by an algorithm (such as the illumination component estimation step in the single-scale Retinex algorithm). In the single-scale Retinex algorithm, the illumination image is usually estimated by Gaussian blurring the logarithmic form of the original image. This process removes high-frequency details in the image and only retains low-frequency illumination information. Therefore, the illumination image can be regarded as a "smoothed" or "blurred" representation of the original image, which mainly reflects the illumination conditions in the scene; R(x,y) represents the reflection image, which is the part of the original image after the illumination effect is removed, and it reflects the inherent properties and color information of the object. In the single-scale Retinex algorithm, the reflection image is calculated by subtracting the estimated illumination component from the logarithmic form of the original image. This process removes the effect of illumination on the image, making the details and color information in the image more prominent. Therefore, the reflection image can be regarded as a "de-illuminated" or "normalized" representation of the original image, which mainly reflects the reflection properties and color information of the object.

[0062] Step A2: Illumination component estimation. Perform Gaussian blur processing on the image after conversion to the logarithmic domain to estimate the illumination component of the image. The purpose of Gaussian blur is to remove high-frequency details in the image and only retain low-frequency illumination information. The specific formula is as follows:

[0063] logR(x,y)=logI(x,y)-log[G k (x,y)*I(x,y)]

[0064] Where G(x,y) is the Gaussian kernel function.

[0065] Step A3: Calculation of reflection component. The reflection component is obtained by subtracting the estimated illumination component from the original logarithmic image. The reflection component represents the details and color information in the image and is the image after the illumination effect is removed.

[0066] Step A4: Convert the reflection component from the logarithmic domain back to the linear space to obtain the final enhanced image.

[0067] Among them, the reflection component image:

[0068] The reflection component image is calculated by subtracting the estimated illumination component from the logarithmic form of the original image. It reflects the inherent properties and color information of the object and is an image after the influence of illumination is removed.

[0069] In the single-scale Retinex algorithm, the calculated reflection component image is usually still in the logarithmic domain and needs to be further converted back to the linear space before it can be used for display or further processing.

[0070] The final enhanced image:

[0071] The final enhanced image is obtained after converting from the logarithmic domain back to the linear space step, which aims to improve the visual effects of the image, such as enhancing contrast and details.

[0072] In the single-scale Retinex algorithm, the final enhanced image is usually obtained by converting the reflectance component image from the logarithmic domain back to the linear space and performing appropriate brightness adjustments (such as image stretching).

[0073] Therefore, although the final enhanced image and the reflected component image are similar in content (both reflect the inherent properties and color information of the object), they are not the same image. The final enhanced image is an image that has been further processed and optimized, and is more suitable for display or further analysis. The reflected component image is a result in the middle of the algorithm, and it itself may need additional processing steps before it can be used in practical applications.

[0074] Step A5: Image stretching: The brightness of the obtained reflection component image is adjusted, usually by linear stretching to normalize the pixel values ​​to the [0, 255] interval to enhance the contrast and details of the image.

[0075] Since the SSR algorithm may be limited in the balance between details and brightness. In some cases, SSR may over-enhance certain parts of the image, resulting in uneven brightness distribution or loss of details. In order to solve this problem, the present invention fuses the SSR algorithms at different scales to obtain the MSR algorithm, and then processes the original image based on the MSR algorithm, specifically referring to step 2.

[0076] Step 2: The original image is processed by the MSR algorithm obtained by weighted fusion of the Retinex algorithm at different scales to obtain the reflected image, which is specifically implemented by the following formula:

[0077]

[0078] Among them, R(x,y) represents the reflected image, I(x,y) represents the original image, K represents the total number of Gaussian kernel functions, G k (x, y) is the kth Gaussian kernel function, G k (x,y)*I(x,y) represents the convolution operation of the kth Gaussian kernel function and the original image.

[0079] In order to overcome the problem of directly using the feature vector of the LBP histogram to calculate too large, a multi-order entropy method is used to measure the information of the LBP histogram and reduce the feature dimension. The LBP algorithm mainly describes the grayscale relationship between the image feature pixels and each pixel. In the basic LBP algorithm, a 3×3 texture unit is usually defined, and the grayscale value of the central pixel of the texture unit is used as the threshold, and the grayscale values ​​of the other 8 adjacent pixels are compared with it. If the grayscale value of the adjacent pixel is greater than or equal to the grayscale value of the central pixel, the pixel is encoded as 1, otherwise it is encoded as 0. In this way, an 8-bit binary number is formed, and then the binary number is converted into a decimal number, which is the LBP value of the central pixel. This value reflects the local texture features around the pixel.

[0080] In the image segmentation task, directly using the feature vector of the LBP histogram will result in excessive calculation and is easily interfered by image noise. Therefore, the concept of high-order entropy in information theory is used to measure the information of the LBP histogram. For details, refer to the calculation formula of multi-order entropy in step 3. In information theory, entropy is usually used to quantify the uncertainty or randomness of information. Therefore, high-order entropy can be understood as the highly repeated or highly correlated part of the information. In order to further improve the adequacy of texture feature extraction, edge density is added to enhance anti-interference ability. For the calculation of edge density, refer to step 3.

[0081] Step 3: Set the size of the sliding window W size ×W size and sliding step length W step, a sliding window is slided on the reflected image to capture local area information, and for the image in the sliding window, the image in the sliding window is divided into multiple sub-blocks according to a preset size, wherein the preset size can be 3x3, 5x5 or other sizes, and the specific size of the sub-block depends on the requirements of the image processing task and the characteristics of the image itself. Therefore, the size of the sub-block may vary from application to application. For each sub-block, the multi-order entropy of the sub-block and the edge density of the sub-block are calculated, and the multi-order entropy of the sub-block and the edge density of the sub-block are used as local features of the sub-block;

[0082] The multi-order entropy of the sub-block is calculated by the following formula:

[0083]

[0084] Where H is the number of sub-blocks in the reflected image, n is the entropy order, h(i) represents the frequency of the i-th sub-block in the LBP histogram, and E n (X) is the multi-order entropy.

[0085] The edge density of the sub-block is calculated in the following way:

[0086] The sub-blocks are calculated by the edge detection method to obtain the target value of each pixel in the sub-block. The target value represents the grayscale change degree of the pixel. The target value of each pixel is compared with the preset grayscale threshold, and the pixel whose target value exceeds the grayscale threshold is obtained and regarded as the edge point. Then the total number of edge points is obtained. According to the total number of edge points and the total number of pixels, the edge density is calculated. Specifically, it is calculated by the following formula:

[0087]

[0088] Among them, N e Represents the total number of edge points, N g Indicates the total number of pixels.

[0089] The edge density can be specifically achieved by applying edge detection algorithms, such as LOG edge detection, Prewitt edge detection, and the like.

[0090] The processed features are segmented using the fuzzy C-means (FCM) clustering algorithm to achieve accurate identification and segmentation of power lines. The FCM algorithm is a clustering algorithm based on an objective function, which achieves clustering by minimizing the objective function. The objective function is usually defined as the weighted sum of the squares of the distances from the sample points to the cluster center, where the weighting coefficient is determined by the membership degree, and the value range of the membership degree is between [0,1]. The fuzzy weighting coefficient m (m>1) is used to control the fuzziness of the clustering, see step 4 for details.

[0091] Step 4: According to the local features and the pre-set K cluster centers, determine the label of each pixel in the sliding window. The number of labels is the same as the number of cluster centers. One label corresponds to one cluster center, and all labels are different. The K labels include one or more power line labels and one or more background labels.

[0092] For example, in the problem of power line segmentation, there are two power lines. The image is segmented into three categories: power line 1 area, power line 2 area, and background area. Then, the labels can be set as:

[0093] Label 1: indicates the power line 1 region. All pixels clustered as the power line 1 region will be assigned this label.

[0094] Label 2: indicates the power line 2 region. All pixels clustered as the power line 2 region will be assigned this label.

[0095] Label 3: indicates the background area. All pixels clustered as background areas will be assigned this label.

[0096] Among them, if the number of power lines is large, the K value should be increased accordingly to ensure that each power line can be accurately divided.

[0097] Image characteristics, such as the thickness of power lines, distribution density, background complexity, etc., will also affect the setting of K value. For example, if the power lines are thin and densely distributed, a larger K value may be needed to capture more detail information. On the contrary, if the power lines are thick and sparsely distributed, the K value can be appropriately reduced.

[0098] When setting the K value, it is necessary to consider the actual situation of the power line and combine it with prior manual experience to determine the optimal K value.

[0099] Step 4.1: According to the preset number of cluster centers K, randomly initialize K cluster centers, and according to the number of cluster centers and the number of local features in the sliding window, initialize the membership matrix, the rows of the membership matrix correspond to the cluster centers, the columns correspond to the local features in the sliding window, the parameters in the initialized membership matrix are random, and the parameters in the initialized membership matrix are expressed as u ij ,u ij It represents the membership of the i-th feature in the local features to the j-th cluster center, and the sum of the membership of each local feature to all clusters is 1;

[0100] Step 4.2: Set the initialized membership matrix as the current membership matrix, set the total number of iterations, set the initial number of iterations to 1, and set the initial number of iterations as the current number of iterations;

[0101] Step 4.3: Based on the fuzzy C-means clustering algorithm, update the current membership matrix to obtain the updated current membership matrix, which is specifically implemented by the following formula:

[0102]

[0103] Among them, u′ ij is the membership of the i-th feature to the j-th cluster center in the updated local feature, m is the fuzzy factor used to control the fuzziness of clustering, x i is the i-th feature in the local feature, v j is the jth cluster center, v k is the kth cluster center, ║·║ is the distance metric;

[0104] Step 4.4: Update each cluster center, which is implemented by the following formula:

[0105]

[0106] Among them, v′ j is the updated j-th cluster center, n is the number of local features in the sliding window;

[0107] Step 4.5: The current iteration number is increased by one;

[0108] Step 4.6: Determine whether the current number of iterations is less than the total number of iterations. If the current number of iterations is not less than the total number of iterations, obtain the final membership matrix. If the current number of iterations is less than the total number of iterations, return to step 4.3.

[0109] Step 4.7: For each local feature, in the final membership matrix, obtain all the memberships corresponding to the local feature, obtain the maximum value among all the memberships, and then determine the cluster center corresponding to the maximum value, and use the label of the cluster center as the label of the sub-block corresponding to the local feature, where the feature of each pixel in the sub-block is the label of the cluster center, and then obtain the label of each pixel in all sub-blocks, that is, the label of all pixels in the sliding window.

[0110] Step 5: Repeat steps 2 to 4 by sliding the sliding window in the reflected image to obtain the label of each pixel in the reflected image;

[0111] Step 6: During the sliding process of the sliding window, there is a situation where a pixel has different labels in different sliding windows. For the situation where a pixel has multiple different labels, a voting mechanism or the principle of the highest average membership is used to determine the final label of the pixel, and then the final labels of all pixels in the reflected image are determined, that is, the power lines or the background to which each pixel belongs are obtained, and then the power lines in the original image are segmented according to the final label of each pixel in the reflected image to obtain the segmentation result.

[0112] Among them, a voting mechanism is used to determine the final label, including:

[0113] Get all the labels corresponding to the pixel, determine the number of each label, get the label with the largest number, and use it as the final label of the pixel.

[0114] Among them, the final label is determined by the principle of the highest average membership, which specifically includes the following steps:

[0115] For different labels, obtain their membership in the final membership matrix corresponding to each sliding window respectively. For the same label, calculate the average of all memberships to obtain the average membership of each label, and obtain the label with the largest average membership as the final label of the pixel.

[0116] The following example illustrates the method of using a voting mechanism or the principle of the highest average membership degree in step 6 to determine the final label of the pixel:

[0117] For example, an image of a power line has two cluster centers, representing the power line area and the background area respectively. Therefore, there will be two cluster labels: label 1 (power line area) and label 2 (background area).

[0118] The image is divided into multiple overlapping sliding windows, and the FCM clustering algorithm is applied independently in each window. Due to the overlap between windows, some pixels in the image may appear in multiple windows and thus obtain multiple cluster labels.

[0119] The final label is determined by a voting mechanism, including:

[0120] If a specific pixel point P in the image appears in three different sliding windows, in the first window, P is clustered as a power line region (label 1); in the second window, P is also clustered as a power line region (label 1); but in the third window, due to the local feature changes or noise in the window, P is incorrectly clustered as a background region (label 2).

[0121] According to the voting mechanism, we count the votes for each cluster label obtained by P:

[0122] Tag 1: 2 votes

[0123] Tag 2: 1 vote

[0124] Since label 1 has the most votes, we use label 1 as the final label for pixel P.

[0125] The final label is determined by the principle of the highest average membership, which includes the following steps:

[0126] For the pixel point P, this time we use the principle of the highest average membership to determine the final label. In each window, we can not only obtain the cluster label of P, but also the membership of P to each cluster center.

[0127] Suppose that in the first window, P's membership to label 1 is 0.9 and its membership to label 2 is 0.1; in the second window, P's membership to label 1 is 0.85 and its membership to label 2 is 0.15; in the third window, P's membership to label 1 is 0.4 and its membership to label 2 is 0.6.

[0128] Then calculate the average membership of P for each label:

[0129] Average membership of label 1: (0.9+0.85+0.4) / 3=0.717

[0130] Average membership of label 2: (0.1+0.15+0.6) / 3=0.283

[0131] Since label 1 has the highest average membership, we take label 1 as the final label of pixel P.

[0132] Combination Figure 2 , a vision-based flying power inspection robot power line segmentation system, used to realize a vision-based flying power inspection robot power line segmentation system, including an onboard computer, a flight controller, a visual sensor, a ground station, a six-rotor flight mechanism, a walking mechanism, an efficient data transmission module and a walking controller;

[0133] The visual sensor is used to collect original images of the power line, and is also used to collect images and videos of the surrounding environment of the power line; the data collected by the visual sensor is transmitted to the onboard computer through an efficient data transmission module, and the visual sensor is configured on the six-rotor flight mechanism;

[0134] The onboard computer is used to receive the original image of the power line, and is also used to implement the power line segmentation method of the flying power inspection robot based on vision, and transmit the segmentation result to the flight controller or the ground station through the efficient data transmission module; it is also used to send the instructions for adjusting the flight attitude, speed, and walking mechanism state to the flight controller through the efficient data transmission module; it is also used to send the walking instructions for adjusting the walking speed and direction to the walking controller through the efficient data transmission module;

[0135] The flight controller is used to receive instructions for adjusting the flight attitude, speed, and state of the walking mechanism; it is also used to send instructions for adjusting the flight attitude, speed, and state of the walking mechanism to control the six-rotor flight mechanism; it receives flight status data sent by the six-rotor flight mechanism, and transmits the flight status data to the onboard computer through an efficient data transmission module; it is also used to receive data collected by other sensors, wherein the flight controller is configured on the six-rotor flight mechanism;

[0136] The six-rotor flight mechanism is used to receive instructions for adjusting the flight attitude, speed, and state of the walking mechanism, and execute the instructions for adjusting the flight attitude, speed, and state of the walking mechanism; and is also used to collect flight status data and send the flight status data to the flight controller;

[0137] The walking controller is used to receive walking instructions for adjusting walking speed and direction, send the walking instructions for adjusting walking speed and direction to the walking mechanism, receive walking status data sent by the walking mechanism, and transmit the walking status data to the onboard computer or the ground station through an efficient data transmission module;

[0138] The walking mechanism is used to receive walking instructions for adjusting walking speed and direction, and execute the walking instructions for adjusting walking speed and direction, collect walking state data through sensors, and send the walking state data to a walking controller.

[0139] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A vision-based flying power inspection robot power line segmentation method, characterized in that: include: Step 1: Collect original images of power lines through a flying power inspection robot; Step 2: The original image is processed using the MSR algorithm obtained by weighted fusion of Retinex algorithms at different scales to obtain a reflected image; Step 3: setting the size and sliding step of the sliding window, sliding the sliding window on the reflected image, dividing the image in the sliding window into a plurality of sub-blocks according to a preset size, calculating the multi-order entropy and the edge density of the sub-block for each sub-block, and taking the multi-order entropy and the edge density of the sub-block as the local features of the sub-block; Step 4: According to the local features and the pre-set K cluster centers, determine the label of each pixel in the sliding window. The number of labels is the same as the number of cluster centers. One label corresponds to one cluster center, and all labels are different. The K labels include one or more power line labels and one or more background labels. Step 5: Repeat steps 2 to 4 by sliding the sliding window in the reflected image to obtain the label of each pixel in the reflected image; Step 6: During the sliding process of the sliding window, there is a situation where a pixel has different labels in different sliding windows. For the situation where a pixel has multiple different labels, a voting mechanism or the principle of the highest average membership is used to determine the final label of the pixel, and then the final labels of all pixels in the reflected image are determined, that is, the power lines or the background to which each pixel belongs are obtained, and then the power lines in the original image are segmented according to the final label of each pixel in the reflected image to obtain the segmentation result.

2. The power line segmentation method of a flying power inspection robot based on vision according to claim 1 is characterized in that: Step 2 is specifically implemented by the following formula: Among them, R(x,y) represents the reflected image, I(x,y) represents the original image, K represents the total number of Gaussian kernel functions, G k (x, y) is the kth Gaussian kernel function, G k (x,y)*I(x,y) represents the convolution operation of the kth Gaussian kernel function and the original image.

3. The power line segmentation method of a flying power inspection robot based on vision according to claim 1 is characterized in that: The multi-order entropy of the sub-block in step 3 is calculated by the following formula: Where H is the number of sub-blocks in the reflected image, n is the entropy order, h(i) represents the frequency of the i-th sub-block in the LBP histogram, and E n (X) is the multi-order entropy.

4. The power line segmentation method of a flying power inspection robot based on vision according to claim 1, characterized in that: The edge density of the sub-block in step 3 is calculated as follows: The sub-blocks are calculated by the edge detection method to obtain the target value of each pixel in the sub-block. The target value represents the grayscale change degree of the pixel. The target value of each pixel is compared with the preset grayscale threshold, and the pixel whose target value exceeds the grayscale threshold is obtained and regarded as the edge point. Then the total number of edge points is obtained. According to the total number of edge points and the total number of pixels, the edge density is calculated. Specifically, it is calculated by the following formula: Among them, N e Represents the total number of edge points, N g Indicates the total number of pixels.

5. The power line segmentation method of a flying power inspection robot based on vision according to claim 1, characterized in that: Step 4 specifically includes: Step 4.1: According to the preset number of cluster centers K, randomly initialize K cluster centers, and according to the number of cluster centers and the number of local features in the sliding window, initialize the membership matrix, the rows of the membership matrix correspond to the cluster centers, the columns correspond to the local features in the sliding window, the parameters in the initialized membership matrix are random, and the parameters in the initialized membership matrix are expressed as u ij ,u ij It represents the membership of the i-th feature in the local features to the j-th cluster center, and the sum of the membership of each local feature to all clusters is 1; Step 4.2: Set the initialized membership matrix as the current membership matrix, set the total number of iterations, set the initial number of iterations to 1, and set the initial number of iterations as the current number of iterations; Step 4.3: Based on the fuzzy C-means clustering algorithm, update the current membership matrix to obtain the updated current membership matrix, which is specifically implemented by the following formula: Among them, u′ ij is the membership of the i-th feature to the j-th cluster center in the updated local feature, m is the fuzzy factor used to control the fuzziness of clustering, x i is the i-th feature in the local feature, v j is the jth cluster center, v k is the kth cluster center, ||·|| is the distance metric; Step 4.4: Update each cluster center, which is implemented by the following formula: Among them, v′ j is the updated j-th cluster center, n is the number of local features in the sliding window; Step 4.5: The current iteration number is increased by one; Step 4.6: Determine whether the current number of iterations is less than the total number of iterations. If the current number of iterations is not less than the total number of iterations, obtain the final membership matrix. If the current number of iterations is less than the total number of iterations, return to step 4.

3. Step 4.7: For each local feature, in the final membership matrix, obtain all the memberships corresponding to the local feature, obtain the maximum value among all the memberships, and then determine the cluster center corresponding to the maximum value, and use the label of the cluster center as the label of the sub-block corresponding to the local feature, where the feature of each pixel in the sub-block is the label of the cluster center, and then obtain the label of each pixel in all sub-blocks, that is, the label of all pixels in the sliding window.

6. The power line segmentation method of a flying power inspection robot based on vision according to claim 1, characterized in that: In step 6, a voting mechanism is used to determine the final label, including: Get all the labels corresponding to the pixel, determine the number of each label, get the label with the largest number, and use it as the final label of the pixel.

7. The power line segmentation method of a flying power inspection robot based on vision according to claim 1, characterized in that: In step 6, the final label is determined by the principle of the highest average membership, which specifically includes the following steps: For different labels, obtain their membership in the final membership matrix corresponding to each sliding window respectively. For the same label, calculate the average of all memberships to obtain the average membership of each label, and obtain the label with the largest average membership as the final label of the pixel.

8. A vision-based flying power inspection robot power line segmentation system, used to implement the vision-based flying power inspection robot power line segmentation system according to claim 1, characterized in that: It includes an onboard computer, a flight controller, a visual sensor, a ground station, a six-rotor flight mechanism, a walking mechanism, an efficient data transmission module and a walking controller; The visual sensor is used to collect original images of the power line, and is also used to collect images and videos of the surrounding environment of the power line; the data collected by the visual sensor is transmitted to the onboard computer through an efficient data transmission module; The onboard computer is used to receive the original image of the power line, and is also used to implement the power line segmentation method of the flying power inspection robot based on vision, and transmit the segmentation result to the flight controller or the ground station through the efficient data transmission module; it is also used to send the instructions for adjusting the flight attitude, speed, and walking mechanism state to the flight controller through the efficient data transmission module; it is also used to send the walking instructions for adjusting the walking speed and direction to the walking controller through the efficient data transmission module; The flight controller is used to receive instructions for adjusting the flight attitude, speed, and state of the walking mechanism; it is also used to send instructions for adjusting the flight attitude, speed, and state of the walking mechanism to control the six-rotor flight mechanism; it receives flight status data sent by the six-rotor flight mechanism, and transmits the flight status data to the onboard computer through an efficient data transmission module; it is also used to receive data collected by other sensors; The six-rotor flight mechanism is used to receive instructions for adjusting the flight attitude, speed, and state of the walking mechanism, and execute the instructions for adjusting the flight attitude, speed, and state of the walking mechanism; and is also used to collect flight status data and send the flight status data to the flight controller; The walking controller is used to receive walking instructions for adjusting walking speed and direction, send the walking instructions for adjusting walking speed and direction to the walking mechanism, receive walking status data sent by the walking mechanism, and transmit the walking status data to the onboard computer or the ground station through an efficient data transmission module; The walking mechanism is used to receive walking instructions for adjusting walking speed and direction, and execute the walking instructions for adjusting walking speed and direction, collect walking state data through sensors, and send the walking state data to a walking controller.

Images