Method for correcting gimbal attitude of fan automatic inspection unmanned aerial vehicle based on laser point cloud

Abstract

The present application relates to the technical field of unmanned aerial vehicle application, especially a method for correcting the posture of a fan automatic inspection unmanned aerial vehicle holder based on laser point cloud, comprising the following steps: calibrating the laser radar load and the visible light camera, mapping the point cloud data to a binary image using the internal parameters, external parameters and distortion parameters of the laser radar load, performing connected domain analysis on the binary image, eliminating holes, extracting the outline of the fan blade, calculating the center of the largest inscribed circle of the largest outline and the largest inscribed rectangle, and determining the direction of the blade through the main direction of the largest inscribed rectangle, the present application converts three-dimensional distance information data into two-dimensional image data by image mapping of the laser point cloud data, and calculates the distance of the binary image through the camera internal parameters, which greatly reduces the calculation amount of the algorithm in point cloud processing and greatly shortens the calculation time, providing a feasible technical solution for real-time holder correction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unmanned aerial vehicle application, and particularly relates to a method for correcting a gimbal attitude of an unmanned aerial vehicle for automatic inspection of a fan based on laser point clouds. BACKGROUND

[0002] With the development of unmanned aerial vehicle technology and the application of the technology in different industries, a new type of inspection operation mode based on unmanned aerial vehicles is gradually emerging in the wind power industry. Relying on the high-altitude flight, long-distance and fast operation capabilities of unmanned aerial vehicles, wind power practitioners can greatly improve the inspection efficiency.

[0003] A multi-rotor unmanned aerial vehicle is equipped with a high-resolution visible light camera and a laser radar load, and an automatic route planning algorithm can ensure that the unmanned aerial vehicle automatically collects data when inspecting the fan blades. The rear-end processing software detects defects in the picture data collected by the unmanned aerial vehicle, so that common abnormalities such as sand eyes, gel coat damage, lightning strikes and rust on the fan blades can be effectively identified.

[0004] The automatic route planning in the fan inspection process is mostly calculated and designed based on the existing data parameters of the fan, mainly including the height of the fan tower, the length of the blade and the latitude and longitude information of the fan. Since the blade models produced by different fan manufacturers are different and the degree of blade bending is different, it is difficult to ensure that the fan blades photographed at each waypoint are in the central position of the image only by calculating the route based on the blade length. In addition, since high-definition images of the blade need to be obtained during fan inspection, a long-focus lens of more than 50mm is mostly used for the camera, and the field of view of the long-focus camera is relatively narrow. When the attitude angle of the load has a very small error, the blade will deviate from the center position of the image on the image, and even the blade may not be collected. Therefore, it is particularly necessary to correct the gimbal attitude in real time to make the blade in the center of the image during the fan blade inspection process. The present application provides a feasible technical solution for this problem. SUMMARY

[0005] The purpose of the present application is to solve the problems existing in the prior art, and to provide a method for correcting the gimbal attitude of an unmanned aerial vehicle for automatic inspection of a fan based on laser point clouds.

[0006] In order to achieve the above-mentioned purpose, the present application adopts the following technical scheme:

[0007] The method for correcting the gimbal attitude of an unmanned aerial vehicle for automatic inspection of a fan based on laser point clouds comprises the following steps:

[0008] S1, calibrate the laser radar load and the visible light camera;

[0009] S2. Using the intrinsic, extrinsic, and distortion parameters of the lidar payload, map the point cloud data to a binary image;

[0010] S3. Perform connected component analysis on the binary image to eliminate holes and extract the outline of the wind turbine blades;

[0011] S4. Calculate the center of the largest inscribed circle of the largest profile and the largest inscribed rectangle, and determine the direction of the blade by the main direction of the largest inscribed rectangle.

[0012] S5. Calculate the perpendicular point of the perpendicular line from the image center point to the contour line using the image center point, the center of the inscribed circle of the connected region contour, and the direction of the connected region.

[0013] S6. Calculate the actual offset distance of the gimbal load using the image center point and vertical point;

[0014] The difference between the coordinates of the perpendicular point P′ and the coordinates of the image center point O is the image pixel difference in the image coordinate system. The differences in the horizontal and vertical directions are denoted as D, respectively. x =k*P′ x D y =k*P′ y .

[0015] If we disregard camera distortion, rotation, and translation (i.e., the world coordinate system and camera coordinate system coincide), the transformation relationship from the image coordinate system to the world coordinate system is as follows:

[0016] Among them, D wx and D wy Z represents the distance the load needs to move in the horizontal and vertical directions, respectively; c The distance from the target point cloud to the camera is calculated as follows:

[0017] Where N is the number of target point clouds, X i The distance of the point cloud to the i-th point in the X direction that satisfies the constraints;

[0018] S7. Based on the gimbal load offset distance and point cloud data, calculate the yaw angle and pitch angle that the gimbal needs to be corrected.

[0019] The calculation formula is:

[0020]

[0021] Preferably, the lidar payload mainly includes a line lidar and a visible light camera. First, the lidar and the visible light camera need to be calibrated. The calibration content mainly includes the intrinsic parameter K, distortion parameters (k1, k2, k3, p1, p2) of the visible light camera, and the extrinsic parameter R,t that can be transferred from the lidar coordinate system to the visible light camera coordinate system.

[0022] Preferably, the initial constraints for the laser point cloud data captured by the payload at the waypoint location are as follows:

[0023] Where i is the i-th point cloud, X i Y i Z i d represents the point cloud coordinates, and d represents the distance.

[0024] The beneficial effects of this invention are as follows: This invention converts three-dimensional distance information data into two-dimensional image data by performing image mapping on laser point cloud data, and calculates the distance on the binary image using camera intrinsic parameters. This greatly reduces the computational load of the algorithm in point cloud processing and significantly shortens the computation time, providing a feasible technical solution for real-time gimbal correction. Attached Figure Description

[0025] Figure 1 This is a flowchart of the wind turbine automatic inspection UAV gimbal attitude correction method based on laser point cloud according to the present invention;

[0026] Figure 2 This is a point cloud data map collected by the lidar payload of the wind turbine automatic inspection UAV gimbal attitude correction method based on laser point cloud in this invention;

[0027] Figure 3 This invention relates to a binary image of a point cloud mapped onto a two-dimensional plane, representing the gimbal attitude correction method for an automatic wind turbine inspection drone based on laser point clouds.

[0028] Figure 4 This invention relates to the maximum inscribed circle center and maximum inscribed rectangle of a binary image for the attitude correction method of a wind turbine automatic inspection UAV gimbal based on laser point clouds.

[0029] Figure 5 The gimbal attitude of the wind turbine automatic inspection UAV gimbal attitude correction method based on laser point cloud in this invention is the image taken before modification;

[0030] Figure 6 The image is taken after gimbal attitude correction according to the gimbal attitude correction method of the wind turbine automatic inspection UAV based on laser point cloud of this invention. Detailed Implementation

[0031] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0032] Reference Figures 1-6A method for attitude correction of a gimbal for an automatic wind turbine inspection drone based on laser point clouds includes the following steps:

[0033] S1. Calibrate the lidar payload and the visible light camera;

[0034] The calibration content mainly includes the intrinsic parameters K and distortion parameters (k1, k2, k3, p1, p2) of the visible light camera, and the extrinsic parameters R and t that can be transferred from the lidar coordinate system to the visible light camera coordinate system;

[0035] S2. Using the intrinsic, extrinsic, and distortion parameters of the lidar payload, map the point cloud data to a binary image;

[0036] Set initial constraints on the laser point cloud data captured by the payload at the waypoint:

[0037]

[0038] Where i is the i-th point cloud, X i Y i Z i Let d be the point cloud coordinates, d be the distance (30), and r be the radius (12). The constrained point cloud is mapped to the camera's image coordinate system using intrinsic and extrinsic parameters between the camera and radar. The binary image after laser point cloud mapping is then cropped using the visible light camera's image size and scaled to a fixed size, as shown in the attached figure. Figure 3 As shown, the scaling factor is set to k;

[0039] S3. Perform connected component analysis on the binary image to eliminate holes and extract the outline of the wind turbine blades;

[0040] The mapped binary image is dilated and eroded to remove small connected components and eliminate holes. Then, the largest connected component is selected as the outline of the wind turbine blade.

[0041] S4. Calculate the center of the largest inscribed circle of the largest profile and the largest inscribed rectangle, and determine the direction of the blade by the main direction of the largest inscribed rectangle.

[0042] For the maximum contour, calculate the center of the maximum inscribed circle, denoted as P. Then, use the maximum inscribed rectangle algorithm to calculate the maximum inscribed rectangle. The four vertices of the inscribed rectangle are denoted as A, B, C, and D. Determine the orientation θ of the rectangle, as shown in the attached diagram. Figure 4 As shown, the method for calculating the orientation angle θ of the rectangle is as follows:

[0043] if:

[0044] Then we have:

[0045] if:

[0046] Then we have:

[0047] Where α is a coefficient, usually taking α∈[1,2], and the magnitude of θ is converted to the range of [-90,90];

[0048] S5. Calculate the perpendicular point of the perpendicular line from the image center point to the contour line using the image center point, the center of the inscribed circle of the connected region contour, and the direction of the connected region.

[0049] Given the center point P and direction θ of the connected region contour, let the image center point be O. Calculate the perpendicular point P′ from O (origin) to the line. The calculation method is as follows:

[0050] The rotation matrix of point P and its direction θ is: Let point O be the origin, and the coordinates of point P be P(P...). x P y Point P after being rotated by the rotation matrix Then we have:

[0051] That is, we can obtain the coordinates of the perpendicular point P′ after rotation. for Will After rotating using the inverse matrix, the coordinates of the perpendicular point P′ before rotation can be obtained:

[0052] S6. Calculate the actual offset distance of the gimbal load using the image center point and vertical point;

[0053] The difference between the coordinates of the perpendicular point P′ and the coordinates of the image center point O is the image pixel difference in the image coordinate system. The differences in the horizontal and vertical directions are denoted as D, respectively. x =k*P′ x D y =k*P′ y ;

[0054] If we disregard camera distortion, rotation, and translation (i.e., the world coordinate system and camera coordinate system coincide), the transformation relationship from the image coordinate system to the world coordinate system is as follows:

[0055] Among them, D wx and D wy Z represents the distance the load needs to move in the horizontal and vertical directions, respectively; c The distance from the target point cloud to the camera is calculated as follows:

[0056] Where N is the number of target point clouds, X iThe distance of the point cloud to the i-th point in the X direction that satisfies the constraints;

[0057] S7. Based on the gimbal load offset distance and point cloud data, calculate the yaw angle and pitch angle that the gimbal needs to be corrected.

[0058] In obtaining D wx and D wy Afterwards, through Calculate the angle the gimbal needs to rotate;

[0059] Compared to current drone-based wind turbine inspection methods, the advantages of this solution are specifically manifested in the following ways:

[0060] The method for gimbal attitude correction of an automatic wind turbine inspection drone based on laser point clouds can calculate the required attitude angle of the gimbal based on the current attitude of the gimbal load and the point cloud data captured by the laser radar, enabling real-time gimbal correction. It remains applicable even in scenarios with slight load drift, greatly improving the applicability of the entire inspection solution. Compared to directly processing point cloud data, this method converts the point cloud data into a binary image. Using the binary image and the intrinsic parameters of the visible light camera, the required rotation angle of the gimbal camera is calculated. The calculation is simple and the processing is efficient. It has already been applied to the inspection operations of the Qipanshan and Ganqika wind farms. Statistics show that the gimbal correction efficiency exceeds 90% during wind turbine blade inspection, demonstrating strong practical application value. Since the laser radar primarily collects wind turbine data and is independent of the external environment, its applicability is broad. Compared to existing wind turbine inspection solutions, this invention has a significant advantage in the accuracy of captured images. Furthermore, due to the presence of the laser radar, the current wind turbine data can be reconstructed during flight, making inspections safer.

[0061] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for correcting the gimbal attitude of an unmanned aerial vehicle for automatic inspection of a fan based on a laser point cloud, characterized in that, The method comprises the following steps: S1, calibrating the laser radar load and the visible light camera; S2, mapping the point cloud data into a binary image using the internal parameter, external parameter and distortion parameter of the laser radar load; S3, performing connected domain analysis on the binary image, eliminating holes and extracting the outline of the fan blade; S4, calculating the center of the largest inscribed circle of the largest contour and the largest inscribed rectangle, and determining the direction of the blade by the main direction of the largest inscribed rectangle; S5. The center of the inscribed circle of the connected component contour is located at the center point of the image. And the direction of the connected components, calculate the perpendicular point of the perpendicular line from the image center point to the contour line. ; S6, calculating the distance of the actual offset of the gimbal load through the image center point and the foot point; vertical dot The coordinate difference between the coordinate of the vertical dot and the coordinate of the image center point is the image pixel difference in the image coordinate system, and the difference in the horizontal and vertical directions is denoted as respectively; If the distortion, rotation and translation of the camera are not considered, that is, the world coordinate system and the camera coordinate system coincide, the conversion relationship from the image coordinate system to the world coordinate system is: wherein , and respectively are the distances the load needs to move in horizontal and vertical directions; is the distance of the target point cloud from the camera, which is calculated as: wherein is the number of target point clouds, is the distance of the th point in the direction of the point cloud satisfying the constraint condition; S7, calculating the yaw angle and the pitch angle that need to be corrected by the gimbal according to the offset distance of the gimbal and the point cloud data; The calculation formula is: ， wherein, and are the distances the load needs to move in horizontal and vertical directions, respectively; is the distance of the target point cloud from the camera.

2. The method for laser point cloud based fan automatic inspection unmanned aerial vehicle gimbal attitude correction according to claim 1, characterized in that: The laser radar load mainly includes a line laser radar and a visible light camera. First, the laser radar and the visible light camera need to be calibrated. The calibration contents mainly include internal parameters of the visible light camera , distortion parameters and external parameters of a laser radar coordinate system to a visible light camera coordinate system .

3. The method for laser point cloud based fan automatic inspection unmanned aerial vehicle gimbal attitude correction according to claim 1, characterized in that: The laser point cloud data captured by the load at the waypoint position sets the initial constraint condition as: ， wherein, is the point cloud, is the point cloud coordinate position, is the distance between multiple point clouds, is the three-dimensional radius.

Images