A fan blade automatic identification method and device based on 3D vision

Abstract

The application relates to a fan blade automatic identification method and device based on 3D vision, which comprises the following steps: S1, obtaining a front RGB image of a fan; S2, performing line segment detection on the preprocessed image based on Hough transformation, and obtaining n line segments; S3, taking the mean value of the intersection points of the straight lines where the n line segments are located as an axis point, so that the center of the camera field of view coincides with the axis point; S4, obtaining an observation point, adjusting the flight height and position of the unmanned aerial vehicle until the optical axis of the camera is perpendicular to the plane determined based on the observation point and the axis point; S5, obtaining a target image with a vertical distance from the camera within (Z-d, Z+d); S6, obtaining the contour of the fan blade in the target image, and saving the target image as a front image if the contour is a complete contour; and S7, repeating S1-S5 to obtain a back image. Compared with the prior art, the application has the advantages of small calculation amount, high precision and the like.

Description

Technical Field

[0001] This invention relates to wind turbine blade inspection, and more particularly to an automatic wind turbine blade identification method and apparatus based on 3D vision. Background Technology

[0002] For fault detection of wind turbine blades, it is necessary to obtain clear images of the blades. CN202011000187.7 discloses a wind turbine blade detection method, system, and computer-storable medium thereof. First, a visible light image of the wind turbine blade is acquired using an optical detection module, an infrared detection module, and a fusion detection system. Then, different blurring algorithms are used to process individual blade images to obtain blurred images. Next, the sharpness difference vector between the original wind turbine image and each blurred image is calculated. This sharpness difference vector, along with the corresponding blurring algorithm number, forms a training sample until K training samples are obtained. By observing these different samples, the wind turbine blade to be detected can be performed. CN201810906932.0 discloses a method and system for detecting wind turbine blades using a multi-sensor controlled drone. The drone is placed in front of the wind turbine, placing it within the depth measurement range of a solid-state radar. The drone is then controlled to fly along a preset path. As the drone flies along the path, the solid-state radar collects a point cloud image of the wind turbine. A wind turbine model is built based on this point cloud image. Multiple path points are set on the front and rear sides of each blade in the wind turbine model. A detection flight path is formed based on these path points. The drone is controlled to fly along this detection flight path, and the solid-state radar tracks and detects the blades. The camera captures images of the detected blades. The drone is controlled to fly along the detection flight path from one side of the blade, from the tip to the root region or vice versa. A solid-state radar mounted on the drone detects the target blade. When the solid-state radar detects the blade, a point cloud map corresponding to a specific area of ​​the blade is formed within its field of view. The gimbal on the drone controls the orientation of the solid-state radar's detection port so that the point cloud map is positioned within a preset range within the field of view. This process is repeated to achieve blade tracking and detection using solid-state radar. Based on the point cloud map, data such as the wind turbine rotor plane and tower height are calculated to reconstruct the wind turbine model.

[0003] The first method mentioned above requires the use of deep learning, in which the collection and training of training samples is a very time-consuming process. In addition, the quality of classifier training is also related to the number and types of training samples, resulting in high computational cost and low accuracy. The second method uses a sonar system that introduces a lot of noise into the acquired point cloud data, which affects the accuracy of wind turbine blade detection. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology by providing an automatic wind turbine blade recognition method and device based on 3D vision. It utilizes the Hough transform line segment detection method to detect line segments in the field of view, adaptively adjusts the position information of the UAV so that the center of the field of view is located at the axis point of the line segment, ensures that the optical axis of the camera is perpendicular to the surface determined based on the observation point and the axis point, and obtains a complete and maximized wind turbine blade outline in the field of view, thereby realizing wind turbine blade detection. It requires fewer sample images, has low computational load, and high accuracy.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] An automatic identification method for wind turbine blades based on 3D vision includes the following steps:

[0007] S1. Based on the SLAM algorithm, conduct drone-based wind turbine navigation and acquire the front RGB image of the wind turbine based on the drone's 3D vision.

[0008] S2. Preprocess the RGB image, and then perform line segment detection based on Hough transform on the preprocessed image. Detect the angle between line segments and the lines containing them. If n line segments are detected, and the angle between any two adjacent line segments is... If the degree is reached, then execute S3;

[0009] S3. Using the average value of the intersection points of the straight lines containing the n line segments as the axis point, adjust the flight altitude and position of the UAV so that the center of the camera's field of view coincides with the axis point;

[0010] S4. When the pivot point is located at the center of the camera's field of view of the UAV, draw a circle with the pivot point as the center and half the radius of the wind turbine blade as the radius. Map the circle in the world coordinate system to the pixel coordinate system. Take the intersection of the circle and the straight line containing n line segments as the observation point and map the observation point back to the world coordinate system. Adjust the flight altitude and position of the UAV until the optical axis of the camera is perpendicular to the surface determined based on the observation point and the pivot point.

[0011] S5. Based on the camera's 3D vision, acquire the target image within (Zd, Z+d) of the vertical distance to the camera, where Z is the distance from the camera to the intersection of the lines containing n line segments, and d is the first threshold.

[0012] S6. Obtain the outline of the wind turbine blades in the target image based on the image segmentation algorithm. If the outline is a complete outline, save the target image as a frontal image.

[0013] S7. Obtain the RGB image of the reverse side of the fan. Repeat S2 to S6 to obtain the reverse image.

[0014] Furthermore, the method for determining whether the optical axis of the camera is perpendicular to the plane determined based on the observation point and the axis point is as follows:

[0015] After mapping the viewpoint in the pixel coordinate system to the world coordinate system

[0016] If n is greater than 2, then select 3 observation points to form 2 vectors. Perform a cross product on the 2 vectors to obtain the first normal vector. Based on the normal vector and the axis point, a straight line is determined. If the straight line coincides with the optical axis of the camera, then the optical axis of the camera is perpendicular to the surface determined based on the observation points and the axis point.

[0017] If n equals 2, then there are 2 observation points. The 2 observation points and the axis point form 2 vectors respectively. The cross product of the 2 vectors is used to obtain the second normal vector. If the second normal vector coincides with the optical axis of the camera, then the optical axis of the camera is perpendicular to the surface determined based on the observation points and the axis point.

[0018] Furthermore, the method for determining a complete outline is as follows:

[0019] Obtain the blade tip point, and transform the axis point and blade tip point from the pixel coordinate system to the world coordinate system. In the world coordinate system, if the distance between the axis point and the blade tip point is within the range of (L-th2, L+th2), and the distance between the blade tip point and the field boundary in the pixel coordinate system does not exceed th3 pixels, then the contour is a complete contour. Here, L is the length of a single wind turbine blade, th2 is the second threshold, and th3 is the third threshold.

[0020] Furthermore, obtaining the leaf apex point specifically involves:

[0021] The specific steps to obtain the leaf tip are as follows:

[0022] The target image is preprocessed using grayscale conversion, Canny edge detection, morphological algorithms, hole-filling algorithms, and skeletonization algorithms. Then, a line segment detection algorithm based on Hough transform is used to obtain n line segments, where the lines containing these n line segments satisfy the condition that the angle between any two adjacent lines is... In each of the n line segments, find the n points that are farthest from the axis point, and take the points that are farthest from the axis point as the leaf tip points.

[0023] Furthermore, the angle between the lines containing the detected line segments is specifically defined as follows:

[0024] If we transform the lines containing two line segments in a rectangular coordinate system into two points in a polar coordinate system, then the angle between the two points in the polar coordinate system is the angle between the lines containing the two line segments in the rectangular coordinate system.

[0025] Furthermore, the n line segments are 3, and the angle between any two adjacent lines is 120 degrees.

[0026] Furthermore, the mean value of the intersection points of the straight lines containing the n line segments is specifically used as the axis point:

[0027] Transform the straight lines containing n line segments in the pixel coordinate system into n points in the polar coordinate system, map the n points into n straight lines in the rectangular coordinate system, solve for the intersection points of the n straight lines in the rectangular coordinate system, and use the average value of the intersection points of the n line segments as the axis point. The intersection points of the n line segments are the intersection points of every two adjacent straight lines among the n lines.

[0028] Furthermore, the first threshold is three times the thickness of the wind turbine blades.

[0029] Furthermore, the preprocessing employs grayscale algorithm, Canny edge detection algorithm, morphological algorithm, hole filling algorithm, and skeletonization algorithm.

[0030] In another aspect of the present invention, an automatic wind turbine blade identification device based on 3D vision is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) Using the Hough transform line segment detection method, line segments that meet the conditions in the image are detected. The position information of the UAV is adaptively adjusted so that the center of the field of view is located at the axis point of the line segment. The observation point is found by using the radius size of the wind turbine blade and the correspondence between the world coordinate system and the pixel coordinate system. The optical axis of the camera is perpendicular to the surface determined based on the observation point and the axis point. The UAV is facing the wind turbine blade. Under the condition of facing the blade, the target image with a vertical distance of (Zd, Z+d) from the camera is obtained. Background interference is eliminated. The method of classifying the wind turbine blade is compared with the training of a large number of existing samples. This invention does not require a training step, has a small amount of computation in the calculation process, and has high accuracy.

[0033] (2) When the wind turbine blade outline image is obtained, the distance between the axis point and the blade tip point is within the range of (L-th2, L+th2), and the distance between the blade tip point and the field boundary in the pixel coordinate system does not exceed th3 pixels, so as to ensure that the complete and maximized wind turbine blade outline is obtained, and the image of the complete wind turbine blade can be saved with high accuracy of the blade in the image.

[0034] (3) Based on the 3D vision of the UAV, the front and back images of the wind turbine blades are obtained, taking into account both the front and back images, which is beneficial for subsequent image-based wind turbine blade defect detection. Attached Figure Description

[0035] Figure 1This is a flowchart of the present invention. Detailed Implementation

[0036] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0037] Example 1:

[0038] To enable automated drone patrol and inspection, such as in harsh environments like the sea or the Gobi Desert where there is strong wind, sand, intense sunlight, and rain, as well as in sparsely populated areas, it is necessary to inspect wind turbine blades for damage, including damage to the protective film, paint peeling, icing, cracks, and oil contamination. To determine if the wind turbine blades are damaged, it is necessary to detect the blades from the images taken during the drone patrol and then make an assessment.

[0039] This embodiment provides an automatic wind turbine blade identification method based on 3D vision, realizing wind turbine blade detection. The method flowchart is as follows. Figure 1 As shown, the method includes the following steps:

[0040] S1. The drone is equipped with a 3D camera and uses the SLAM algorithm to cruise the wind turbine and acquire the front RGB image of the wind turbine.

[0041] S2. Preprocess the RGB image. Perform line segment detection based on Hough transform on the preprocessed image, detecting the angle between line segments and the lines containing them. If n line segments are detected, and the angle between any two adjacent line segments is... If the degree is correct, then proceed to step S3.

[0042] Preprocessing includes grayscale conversion of RGB images, Canny edge detection, morphological algorithms, hole filling, and skeletonization algorithms.

[0043] The specific method for detecting the angle between any two adjacent lines is as follows: transform two adjacent line segments of n line segments in the rectangular coordinate system into two points in the polar coordinate system. Then, the angle between the two points in the polar coordinate system is the angle between the two line segments in the rectangular coordinate system.

[0044] S3. Using the average value of the intersection points of the lines containing the n line segments as the pivot point, adjust the flight altitude and position of the drone so that the center of the camera's field of view coincides with the pivot point.

[0045] Using the mean of the intersection points of the lines containing n line segments as the pivot point involves: transforming the lines containing the n line segments in the pixel coordinate system into n points in the polar coordinate system, mapping the n points into n lines in the rectangular coordinate system, solving for the intersection points of the n lines in the rectangular coordinate system, and using the mean of the intersection points of the n lines as the pivot point. The intersection point of the n lines is the intersection point of every two adjacent lines among the n lines.

[0046] S4. When the pivot point is located at the center of the camera's field of view on the drone, draw a circle with the pivot point as the center and half the radius of the wind turbine blade as the radius. Map the circle in the world coordinate system to the pixel coordinate system. Take the intersection of the circle and the straight line containing n line segments as the observation point and map the observation point back to the world coordinate system. Adjust the drone's flight altitude and position until the camera's optical axis is perpendicular to the surface determined based on the observation point and the pivot point.

[0047] After mapping the viewpoint in the pixel coordinate system to the world coordinate system

[0048] If n is greater than 2, then select 3 observation points to form 2 vectors. Perform a cross product on the 2 vectors to obtain the first normal vector. Based on the normal vector and the axis point, a straight line is determined. If the straight line coincides with the optical axis of the camera, then the optical axis of the camera is perpendicular to the surface determined based on the observation points and the axis point.

[0049] If n equals 2, then there are 2 observation points. The 2 observation points and the axis point form 2 vectors respectively. The cross product of the 2 vectors is used to obtain the second normal vector. If the second normal vector coincides with the optical axis of the camera, then the optical axis of the camera is perpendicular to the surface determined based on the observation points and the axis point.

[0050] S5. Based on the camera's 3D vision, acquire the target image within (Zd, Z+d) of the vertical distance from the camera, where Z is the distance from the camera to the intersection of the lines containing n line segments, d is the first threshold, and d can be 3 times the thickness of the wind turbine blade.

[0051] Acquire target images within (Zd, Z+d) of the camera's vertical distance, ensuring that the target images exclude background interference.

[0052] S6. Obtain the outline of the wind turbine blades in the target image based on the image segmentation algorithm. If the outline is a complete outline, save the target image as a frontal image.

[0053] Obtain the blade tip point, and transform the axis point and blade tip point from the pixel coordinate system to the world coordinate system. In the world coordinate system, if the distance between the axis point and the blade tip point is within the range of (L-th2, L+th2), and the distance between the blade tip point and the field of view boundary in the pixel coordinate system does not exceed th3 pixels, then the contour is a complete contour. L is the length of a single wind turbine blade, th2 is the second threshold, and th3 is the third threshold.

[0054] The specific steps for obtaining the leaf tip point are as follows: Preprocessing is performed on the target image using grayscale conversion, Canny edge detection, morphological algorithms, hole-filling algorithms, and skeletonization algorithms. Then, a line segment detection algorithm based on Hough transform is used to obtain n line segments, where the angle between any two adjacent lines is... In each of the n line segments, find the n points that are farthest from the axis point, and take the points that are farthest from the axis point as the leaf tip points.

[0055] If the distance between the axis point and the blade tip point does not meet the range of (L-th2, L+th2), then adjust the flight altitude and position of the drone to move away from the wind turbine blades.

[0056] If the distance between the blade tip and the field of view boundary exceeds th3 pixels in the pixel coordinate system, adjust the drone's flight altitude and position to get closer to the wind turbine blade.

[0057] S7. Obtain the RGB image of the reverse side of the fan. Repeat S2 to S6 to obtain the reverse image.

[0058] In some embodiments, the number of wind turbine blades n is 3, so the angle between the two line segments is 120 degrees. Based on the Hough detection algorithm, 3 line segments are detected, and the 3 line segments satisfy the condition that the angle between any two adjacent line segments is 120 degrees.

[0059] In some embodiments, the number of wind turbine blades n is 4, so the angle between the two line segments is 90 degrees. Based on the Hough detection algorithm, 4 line segments are detected, and the 4 line segments satisfy the condition that the angle between any two adjacent line segments is 90 degrees.

[0060] This invention utilizes the number of wind turbine blades and the angle between them, employing a Hough transform line segment detection method to detect line segments in the field of view. It then calculates line segments that satisfy the condition of the angle between any two adjacent straight lines based on their polar coordinates. The center of the UAV's camera field of view is adjusted to the pivot point, and the UAV's flight altitude and position are adjusted until the camera's optical axis is perpendicular to the plane determined based on the observation point and pivot point. This ensures that the UAV and wind turbine blades are directly facing each other, not viewed from the side, and that the center point is at the intersection, resulting in a clearer image of the blades. While maintaining this direct alignment, a target image unaffected by background interference is acquired. The distance from the center of the target image to the tip of the wind turbine blade is accurately detected, ensuring this distance matches the actual length of the blade, and the distance from the blade tip to the field of view boundary does not exceed a threshold. This maximizes and completes the capture of the wind turbine blades, resulting in a complete and clear image. Capturing both the front and back of the wind turbine blades facilitates subsequent image-based defect detection.

[0061] This embodiment also provides an automatic wind turbine blade identification device based on 3D vision, including a memory and a processor. The memory stores a computer program, and the processor executes the program to implement the above-described method.

[0062] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for automatic identification of wind turbine blades based on 3D vision, characterized in that, Includes the following steps: S1. Based on the SLAM algorithm, conduct drone-based wind turbine navigation and acquire the front RGB image of the wind turbine based on the drone's 3D vision. S2. Preprocess the RGB image, and then perform line segment detection based on Hough transform on the preprocessed image. Detect the angle between line segments and the lines containing them. If n line segments are detected, and the angle between any two adjacent line segments is... If the degree is reached, then execute S3; S3. Using the average value of the intersection points of the straight lines containing the n line segments as the axis point, adjust the flight altitude and position of the UAV so that the center of the camera's field of view coincides with the axis point; S4. When the pivot point is located at the center of the camera's field of view of the UAV, draw a circle with the pivot point as the center and half the radius of the wind turbine blade as the radius. Map the circle in the world coordinate system to the pixel coordinate system. Take the intersection of the circle and the straight line containing n line segments as the observation point and map the observation point back to the world coordinate system. Adjust the flight altitude and position of the UAV until the optical axis of the camera is perpendicular to the surface determined based on the observation point and the pivot point. S5. Based on the camera's 3D vision, acquire the target image with a vertical distance to the camera within (Zd, Z+d), where Z is the distance from the camera to the intersection of the lines containing n line segments, and d is the first threshold. S6. Obtain the outline of the wind turbine blades in the target image based on the image segmentation algorithm. If the outline is a complete outline, save the target image as a frontal image. S7. Obtain the RGB image of the reverse side of the fan. Repeat S2~S6 to obtain the reverse image. The method for determining a complete outline is as follows: Obtain the blade tip point, transform the axis point and the blade tip point from the pixel coordinate system to the world coordinate system. In the world coordinate system, if the distance between the axis point and the blade tip point is within the range of (L-th2, L+th2), and the distance between the blade tip point and the field boundary in the pixel coordinate system does not exceed th3 pixels, then the contour is a complete contour. Here, L is the length of a single wind turbine blade, th2 is the second threshold, and th3 is the third threshold. The specific steps to obtain the leaf tip are as follows: The target image is preprocessed using grayscale conversion, Canny edge detection, morphological algorithms, hole-filling algorithms, and skeletonization algorithms. Then, a line segment detection algorithm based on Hough transform is used to obtain n line segments, where the lines containing these n line segments satisfy the condition that the angle between any two adjacent lines is... In each of the n line segments, find the n points that are farthest from the axis point, and take the points that are farthest from the axis point as the leaf tip points.

2. The method for automatic identification of wind turbine blades based on 3D vision according to claim 1, characterized in that, The method for determining whether the optical axis of the camera is perpendicular to the plane determined based on the observation point and the axis point is as follows: After mapping the viewpoint in the pixel coordinate system to the world coordinate system If n is greater than 2, then select 3 observation points to form 2 vectors. Perform a cross product on the 2 vectors to obtain the first normal vector. Based on the normal vector and the axis point, a straight line is determined. If the straight line coincides with the optical axis of the camera, then the optical axis of the camera is perpendicular to the surface determined based on the observation points and the axis point. If n equals 2, then there are 2 observation points. The 2 observation points and the axis point form 2 vectors respectively. The cross product of the 2 vectors is used to obtain the second normal vector. If the second normal vector coincides with the optical axis of the camera, then the optical axis of the camera is perpendicular to the surface determined based on the observation points and the axis point.

3. The method for automatic identification of wind turbine blades based on 3D vision according to claim 1, characterized in that, The angle between the lines containing the detected line segments is specifically: If we transform the lines containing two line segments in a rectangular coordinate system into two points in a polar coordinate system, then the angle between the two points in the polar coordinate system is the angle between the lines containing the two line segments in the rectangular coordinate system.

4. The method for automatic identification of wind turbine blades based on 3D vision according to claim 1, characterized in that, The n line segments are 3 in total, and the angle between any two adjacent lines is 120 degrees.

5. The method for automatic identification of wind turbine blades based on 3D vision according to claim 1, characterized in that, Specifically, the mean value of the intersection points of the straight lines containing the n line segments is used as the pivot point: Transform the straight lines containing n line segments in the pixel coordinate system into n points in the polar coordinate system, map the n points into n straight lines in the rectangular coordinate system, solve for the intersection points of the n straight lines in the rectangular coordinate system, and use the average value of the intersection points of the n line segments as the axis point. The intersection points of the n line segments are the intersection points of every two adjacent straight lines among the n lines.

6. The method for automatic identification of wind turbine blades based on 3D vision according to claim 1, characterized in that, The first threshold is 3 times the thickness of the wind turbine blade.

7. The method for automatic identification of wind turbine blades based on 3D vision according to claim 1, characterized in that, The preprocessing involves using grayscale algorithms, Canny edge detection algorithms, morphological algorithms, hole-filling algorithms, and skeletonization algorithms.

8. A 3D vision-based automatic wind turbine blade identification device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

Images