A fan inspection method based on efficient matrix shooting of a unmanned aerial vehicle

Abstract

The application discloses a fan inspection method based on high-efficiency matrix shooting of an unmanned aerial vehicle, and the fan inspection method comprises the following steps: collecting fan data, and calculating fan pose parameters based on the fan data; calculating fan blade regions in single shooting at different positions in combination with the fan pose parameters, and generating an inspection route of current fan blades according to the fan blade regions; calculating required fan blade endpoint ENU coordinates at shooting points of the inspection route, and converting the fan blade endpoint ENU coordinates into fan blade endpoint image coordinate system coordinates; generating final fan blade endpoint image coordinates in combination with fan blade region image coordinate system coordinates and the fan blade endpoint image coordinate system coordinates; and calculating a matrix shooting array corresponding to a required shooting area based on the final fan blade endpoint image coordinates. The application effectively solves the problem that fan blade photos are not clear enough in the inspection process, and realizes high-definition and high-zoom shooting of the fan.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of camera imaging, in particular to a fan inspection method based on high-efficiency matrix shooting of a UAV. BACKGROUND

[0002] In the field of wind power generation, the state monitoring of fan blades and nacelles is crucial for improving the operating efficiency of fans and prolonging the service life of equipment. In recent years, fan detection technology based on UAVs has gradually attracted attention. UAVs can quickly cover the entire wind farm, and the detection time of fans is usually only a small part of traditional manual inspection. For large wind farms, UAVs can complete the detection of multiple fans in a short time, significantly improving the inspection efficiency.

[0003] In existing UAV inspection shooting methods, there are mainly two types: static shooting and dynamic shooting. Static shooting limits the shooting accuracy because the shooting range of a single shot is large due to the inability of the UAV inspection route to be too close to the flight point. This limits the performance of the camera. Dynamic shooting methods attempt to solve the above problems by flying and shooting simultaneously, but the shooting process is prone to dynamic blur and other problems due to the constant change of the camera position during flight, which affects the clarity of the photos.

[0004] Currently, there is no effective solution to the problems in the related art. SUMMARY

[0005] (I) Technical problems solved

[0006] To address the shortcomings of the prior art, the present application provides a fan inspection method based on high-efficiency matrix shooting of a UAV, which has the advantage of high shooting accuracy, thereby solving the problem of unclear fan blade photos during the inspection process.

[0007] (II) Technical solutions

[0008] To achieve the above-mentioned advantage of high shooting accuracy, the present application employs the following specific technical solutions:

[0009] A fan inspection method based on high-efficiency matrix shooting of a UAV, the fan inspection method comprising:

[0010] Collecting fan data and calculating fan pose parameters based on the fan data, wherein the fan pose parameters include fan hub center positioning coordinates, fan blade length, fan yaw angle, fan blade angle, fan pitch angle, and fan cone angle;

[0011] Establishing an ENU (East-North-Up) geocentric coordinate system with the fan hub center as the origin, calculating the fan blade area of a single shot at different positions in combination with the fan pose parameters, and generating an inspection route for the current fan blade based on the fan blade area.

[0012] At the shooting points on the inspection route, the ENU coordinates of the fan blade end points required for shooting are calculated, and the ENU coordinates of the fan blade end points are converted into fan blade end point image coordinate system coordinates;

[0013] Combined with the fan blade region image coordinate system coordinates and the fan blade end point image coordinate system coordinates, the final fan blade end point image coordinates are generated;

[0014] Based on the final fan blade end point image coordinates, the matrix shooting array corresponding to the required shooting area is calculated, and the gimbal is mobilized to sequentially perform zoom shooting on the matrix shooting array.

[0015] Preferably, the fan data is collected, and the fan pose parameters are calculated based on the fan data, including:

[0016] The camera imaging picture center of the camera device carried by the unmanned aerial vehicle is aligned with the fan hub center point, and the fan data is collected by the laser radar carried by the unmanned aerial vehicle;

[0017] The fan data includes the distance between the unmanned aerial vehicle and the fan hub center point, the current yaw angle of the camera gimbal, the current pitch angle of the camera gimbal, and the current positioning coordinates of the unmanned aerial vehicle.

[0018] The fan hub center positioning coordinates are calculated based on the fan data, and the current yaw angle of the camera gimbal is reversed as the fan yaw angle.

[0019] The camera imaging picture is input into the deep learning segmentation network model, the mask picture and the fan hub pixel coordinates are generated by using the deep learning segmentation network model, and the fan blade angle is calculated combined with the mask picture and the fan hub pixel coordinates.

[0020] Preferably, the camera imaging picture is input into the deep learning segmentation network model, the mask picture and the fan hub pixel coordinates are generated by using the deep learning segmentation network model, and the fan blade angle is calculated combined with the mask picture and the fan hub pixel coordinates.

[0021] The camera imaging picture is input into the deep learning segmentation network model, and the mask picture and the fan hub pixel coordinates are obtained by using the deep learning segmentation network model.

[0022] The fan hub region center point coordinates are calculated based on the fan hub pixel coordinates, and the calculation formula of the fan hub region center point coordinates is:

[0023] Centre X =(X1+X2) / 2;

[0024] Centre Y =(Y1+Y2) / 2;

[0025] In the formula, Centre(Centre X , Centre Y ) represents the coordinates of the center point of the fan hub area; (X1, Y1) represents the coordinates of the upper left pixel of the fan hub area; (X2, Y2) represents the coordinates of the lower right pixel of the fan hub area;

[0026] A Hough line algorithm is used to extract straight lines represented by two end point coordinates from the mask image, and the angle of each straight line is calculated;

[0027] Straight lines meeting preset conditions are filtered, and the remaining straight lines are clustered based on the angle of each straight line. After clustering is completed, the average angle in each category is taken as the blade angle;

[0028] The angle closest to the tower straight line is selected from the blade angles as the first fan blade angle closest to the tower, and the fan blade angle is calculated based on the first fan blade angle closest to the tower.

[0029] Preferably, the fan blade area of a single shot at different positions is calculated in combination with the fan pose parameters, and the inspection route of the current fan blade is generated according to the fan blade area, which comprises:

[0030] The fan blade direction vector is calculated based on the fan blade angle, the fan elevation angle, and the fan cone angle;

[0031] The pan tilt pitch angle, the pan tilt yaw angle, and the shooting distance required for the camera device to aim at the fan blade on the inspection route shooting point of the fan blade are calculated;

[0032] The straight line equations of two field of view lines are defined according to the predefined camera device parameters, and the straight line equations are solved to obtain the fan blade area of a single shot;

[0033] The number of inspection route points of a single fan blade is determined according to the fan blade area of a single shot, and the coordinates of each shooting point in the inspection route are determined based on the user-defined route starting point and the fan blade direction vector;

[0034] The number of inspection route points of a single fan blade, the coordinates of each shooting point in the inspection route, and the fan yaw angle are combined to generate the inspection route of the current fan blade.

[0035] Preferably, the fan blade direction vector is calculated based on the fan blade angle, the fan elevation angle, and the fan cone angle, which comprises:

[0036] The fan elevation angle matrix is calculated, and the expression of the fan elevation angle matrix is:

[0037]

[0038] The fan blade direction vector is calculated based on the fan blade angle, the fan elevation angle, the fan cone angle and the fan elevation angle matrix, and the expression of the fan blade direction vector is:

[0039] element1 = COS(BladeAngle BladeId ) * COS(WindPowerConeAngle);

[0040] element2 = SIN(WindPowerConeAngle);

[0041] element3 = SIN(BladeAngle BladeId ) * COS(WindPowerConeAngle);

[0042] BladeUnitVector BladeId = WindPowerPitchMatrix * [element1 element2 element3];

[0043] In the formula, WindPowerPitchMatrix represents the fan elevation angle matrix; WindPowerPitchAngle represents the fan elevation angle; BladeUnitVector BladeId represents the direction vector of different fan blades; WindPowerConeAngle represents the fan cone angle; BladeAngle BladeId represents the fan blade angle of different fan blades.

[0044] Preferably, on the photographing points of the inspection route, the ENU coordinates of the fan blade end points required to be photographed are calculated, and the ENU coordinates of the fan blade end points are converted into the image coordinates of the fan blade end point image coordinate system, which comprises:

[0045] On the photographing points of the inspection route, the ENU coordinates of the fan blade region end points required to be currently photographed are calculated;

[0046] According to the camera device parameters and the photographing target points, a field of view imaging body is established, and the ENU coordinates of the fan blade region end points are converted into the camera imaging body coordinates based on the field of view imaging body;

[0047] The camera imaging body coordinates are converted into the pixel coordinates in the camera image in combination with the cosine theorem and the sine theorem, to obtain the pixel coordinates of the fan blade region end points;

[0048] The camera imaging picture is segmented by using a deep learning segmentation network model, to obtain a segmentation mask image, and the image coordinates of the fan blade end points in the image coordinate system are extracted in the segmentation mask image.

[0049] Preferably, a deep learning segmentation network model is used to segment the camera image to obtain a segmentation mask image. The coordinates of the wind turbine blade endpoint image coordinate system are extracted from the segmentation mask image, including:

[0050] The camera image is input into a deep learning segmentation network model to obtain a segmentation mask map. Connected regions in the segmentation mask map are calculated, connected regions exceeding a preset threshold are retained, and gap pixels in the connected regions are filled.

[0051] The slope of the wind turbine blade trajectory is calculated based on the pixel coordinates of the endpoints of the wind turbine blade region, and the angle of the wind turbine blade trajectory is calculated based on the slope of the wind turbine blade trajectory.

[0052] The current wind turbine blade trajectory direction is determined based on the slope of the wind turbine blade trajectory, and the coordinates of the wind turbine blade endpoint image coordinate system are searched in the segmented mask image based on the wind turbine blade trajectory direction.

[0053] Preferably, the current wind turbine blade trajectory direction is determined based on the slope of the wind turbine blade trajectory, and the coordinates of the wind turbine blade endpoint image coordinate system are searched in the segmented mask image according to the wind turbine blade trajectory direction, including:

[0054] The current wind turbine blade trajectory direction is determined based on the slope of the wind turbine blade trajectory. If the wind turbine blade is vertical, a fixed sliding window is used to slide in the opposite direction in the up and down direction of the Y-axis of the segmented mask image until the sliding window contains the pixels of the wind turbine blade area.

[0055] If the fan blades are horizontal, then in the segmented shielding Figure X The left and right directions of the axis are slid in opposite directions in a fixed sliding window until the sliding window contains the pixels of the fan blade region; the mass coordinates of the fan blade region are extracted as the coordinates of the fan blade endpoint image coordinate system.

[0056] Preferably, calculating the matrix imaging array corresponding to the required imaging area based on the final wind turbine blade tip image coordinates includes:

[0057] Based on the final image coordinates of the wind turbine blade endpoints, a linear equation for the wind turbine blade trajectory is established.

[0058] Calculate the focal length corresponding to the current zoom level of the camera device, calculate the field of view of each matrix shooting array grid corresponding to the current zoom level based on the focal length, and calculate the pixel range of the matrix shooting array grid on the camera image based on the field of view.

[0059] The wind turbine blade orientation is determined based on the final image coordinates of the wind turbine blade endpoints, and a matrix imaging array grid is inserted into the wind turbine blade trajectory based on the wind turbine blade orientation.

[0060] Preferably, the calculation formula of the focal length corresponding to the current shooting zoom ratio of the camera device is:

[0061] FLenght=MINFLENGHT*ZOOMRATIO

[0062] The calculation formula of the field of view range of each matrix shooting array grid corresponding to the current shooting zoom ratio is:

[0063]

[0064] The calculation formula of the pixel range of the matrix shooting array grid on the camera imaging picture is:

[0065]

[0066] In the formula, FLenght represents the focal length corresponding to the current shooting zoom ratio of the camera device; MINFLENGHT represents the minimum focal length of the camera device; ZOOMRATIO represents the current shooting zoom ratio of the camera device; GFov represents the field of view range of each matrix shooting array grid corresponding to the current shooting zoom ratio; CMOS represents the CMOS size of the camera device; ARCTAN represents the inverse tangent function; H / V represents the horizontal direction; GPixer represents the pixel range of the matrix shooting array grid on the camera imaging picture; FOV represents the maximum field of view range of the camera device; and DPI represents the image resolution of the camera device.

[0067] (III) Advantages

[0068] Compared with the prior art, the fan inspection method based on high-efficiency matrix shooting of a UAV provided by the application has the following advantages:

[0069] The fan inspection method based on high-efficiency matrix shooting of a UAV provided by the application effectively solves the problem that the fan blade photos are not clear enough in the inspection process, realizes high-definition and high-zoom shooting of the fan, and simultaneously determines the complete inspection flight path and the accurate shooting action in the flight path generation stage, avoids the real-time decision-making demand in the inspection process, and thus improves the shooting accuracy and significantly improves the inspection efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the application, and for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0071] Figure 1is a flow chart of a wind turbine inspection method based on high-efficiency matrix shooting of a UAV according to an embodiment of the present application;

[0072] Figure 2 is a Mask chart containing hub pixel coordinates in a wind turbine inspection method based on high-efficiency matrix shooting of a UAV according to an embodiment of the present application;

[0073] Figure 3 is a schematic diagram of a wind turbine blade angle in a wind turbine inspection method based on high-efficiency matrix shooting of a UAV according to an embodiment of the present application;

[0074] Figure 4 is a schematic diagram of a gimbal pitch angle, yaw angle and shooting distance in a wind turbine inspection method based on high-efficiency matrix shooting of a UAV according to an embodiment of the present application;

[0075] Figure 5 is a schematic diagram of a single shooting blade range of a current waypoint in a wind turbine inspection method based on high-efficiency matrix shooting of a UAV according to an embodiment of the present application;

[0076] Figure 6 is a schematic diagram of an inspection route in a wind turbine inspection method based on high-efficiency matrix shooting of a UAV according to an embodiment of the present application;

[0077] Figure 7 is a schematic diagram of image coordinates of a current shot wind turbine blade area endpoint in a wind turbine inspection method based on high-efficiency matrix shooting of a UAV according to an embodiment of the present application;

[0078] Figure 8 is a schematic diagram of a field of view imaging body in a wind turbine inspection method based on high-efficiency matrix shooting of a UAV according to an embodiment of the present application;

[0079] Figure 9 is one of schematic diagrams of field of view imaging body parameters in a wind turbine inspection method based on high-efficiency matrix shooting of a UAV according to an embodiment of the present application;

[0080] Figure 10 is one of schematic diagrams of conversion of camera field of view body coordinates of a target point to pixel coordinates in a camera image in a wind turbine inspection method based on high-efficiency matrix shooting of a UAV according to an embodiment of the present application;

[0081] Figure 11 is another one of schematic diagrams of conversion of camera field of view body coordinates of a target point to pixel coordinates in a camera image in a wind turbine inspection method based on high-efficiency matrix shooting of a UAV according to an embodiment of the present application;

[0082] Figure 12 is a segmentation schematic diagram in a wind turbine inspection method based on high-efficiency matrix shooting of a UAV according to an embodiment of the present application;

[0083] Figure 13 is one of the schematic diagrams of searching the blade region endpoint in the Mask graph in the wind turbine inspection method based on the high-efficiency matrix shooting of the unmanned aerial vehicle according to the embodiment of the present application;

[0084] Figure 14 is one of the schematic diagrams of searching the blade region endpoint in the Mask graph in the wind turbine inspection method based on the high-efficiency matrix shooting of the unmanned aerial vehicle according to the embodiment of the present application;

[0085] Figure 15 is the schematic diagram of the final blade region pixel coordinate in the wind turbine inspection method based on the high-efficiency matrix shooting of the unmanned aerial vehicle according to the embodiment of the present application;

[0086] Figure 16 is the schematic diagram of the high-efficiency matrix array in the horizontal blade in the wind turbine inspection method based on the high-efficiency matrix shooting of the unmanned aerial vehicle according to the embodiment of the present application;

[0087] Figure 17 is the schematic diagram of the high-efficiency matrix array in the vertical blade in the wind turbine inspection method based on the high-efficiency matrix shooting of the unmanned aerial vehicle according to the embodiment of the present application;

[0088] Figure 18 is the second schematic diagram of the field of view imaging body parameter in the wind turbine inspection method based on the high-efficiency matrix shooting of the unmanned aerial vehicle according to the embodiment of the present application;

[0089] Figure 19 is the third schematic diagram of the field of view imaging body parameter in the wind turbine inspection method based on the high-efficiency matrix shooting of the unmanned aerial vehicle according to the embodiment of the present application;

[0090] Figure 20 is the fourth schematic diagram of the field of view imaging body parameter in the wind turbine inspection method based on the high-efficiency matrix shooting of the unmanned aerial vehicle according to the embodiment of the present application. DETAILED DESCRIPTION

[0091] For further illustrating the embodiments, the present application provides drawings which are part of the disclosure, mainly used to explain the embodiments, and can explain the operation principle of the embodiments in cooperation with the related description of the specification. Those skilled in the art should understand other possible implementation manners and advantages of the present application by referring to these contents.

[0092] According to the embodiment of the present application, a wind turbine inspection method based on the high-efficiency matrix shooting of the unmanned aerial vehicle is provided.

[0093] The present application will be further described in conjunction with the drawings and specific embodiments. As shown in the drawings, the wind turbine inspection method based on the high-efficiency matrix shooting of the unmanned aerial vehicle according to the embodiment of the present application comprises: Figure 1

[0094] ​S1, collect fan data, and calculate fan pose parameters based on the fan data, and the fan pose parameters include fan hub center positioning coordinates, fan blade length, fan yaw angle, fan blade angle, fan elevation angle and fan cone angle.

[0095] The fan data includes the distance between the unmanned aerial vehicle and the fan hub center point, the current yaw angle of the camera holder, the current pitch angle of the camera holder and the current positioning coordinates of the unmanned aerial vehicle.

[0096] The camera imaging picture center of the camera equipment carried by the unmanned aerial vehicle is aligned with the fan hub center point, and the fan data is collected by using the laser radar carried by the unmanned aerial vehicle.

[0097] The fan data includes the distance between the unmanned aerial vehicle and the fan hub center point, the current yaw angle of the camera holder, the current pitch angle of the camera holder and the current positioning coordinates of the unmanned aerial vehicle.

[0098] The fan hub center positioning coordinates are calculated based on the fan data, and the current yaw angle of the camera holder is reversed as the fan yaw angle.

[0099] The camera imaging picture is input into a deep learning segmentation network model, and the deep learning segmentation network model is used to generate a mask picture and fan hub pixel coordinates, and the fan blade angle is calculated in combination with the mask picture and the fan hub pixel coordinates.

[0100] The camera imaging picture is input into a deep learning segmentation network model, and the deep learning segmentation network model is used to generate a mask picture and fan hub pixel coordinates, and the fan blade angle is calculated in combination with the mask picture and the fan hub pixel coordinates.

[0101] The camera imaging picture is input into a deep learning segmentation network model, and the deep learning segmentation network model is used to generate a mask picture and fan hub pixel coordinates, and the fan blade angle is calculated in combination with the mask picture and the fan hub pixel coordinates.

[0102] The Hough straight line algorithm is used to extract straight lines represented by two end point coordinates from the mask picture, and the angle of each straight line is calculated.

[0103] The straight lines meeting the preset conditions are filtered, and the remaining straight lines are clustered based on the angle of each straight line, and the average angle in each class is taken as the blade angle after clustering is completed.

[0104] The angle closest to the tower tube straight line is selected from the blade angle as the first fan blade angle closest to the tower tube, and the fan blade angle is calculated based on the first fan blade angle.

[0105] In order to facilitate the understanding of the above technical solutions of the present application, the following will be described in detail in the actual process of collecting fan data and calculating fan pose parameters based on fan data.

[0106] Step one, when the UAV takes off, the camera picture (camera imaging picture) center is aligned with the wind turbine hub center point, the current UAV and the wind turbine hub center distance is obtained by using the laser radar carried by the UAV, the current yaw angle and pitch angle of the camera gimbal and the current GPS coordinates of the UAV.

[0107] Step two, calculate the wind turbine hub center GPS (positioning) coordinates HubGpsPosition:

[0108]

[0109] HubGpsPositionAlt = DroneGpsPositionAlt + (DroneToHubDistance x SIN(GimbalPitchAngle));

[0110] In the formula, HubGpsPosition represents the wind turbine hub center GPS coordinates, wherein HubGpsPositionLat, HubGpsPositionLon and HubGpsPositionAlt represent the latitude, longitude and height respectively; EARTH_RADIUS represents the earth radius constant; PI represents the circular constant; DroneToHubDistance represents the current UAV and the wind turbine hub center distance; GimbalYawAngle represents the current yaw angle of the camera gimbal; CimbalPitchAngle represents the current pitch angle of the camera gimbal; DroneGpsPosition represents the current GPS coordinates of the UAV.

[0111] Step three, calculate the wind turbine yaw angle WindPowerYawAngle, the camera is directly opposite to the wind turbine in step one, so the wind turbine yaw angle is the reverse of the gimbal yaw angle:

[0112] WindPowerYawAngle = GimbalYawAngle + 180;

[0113] In the formula, GimbalYawAngle represents the current yaw angle of the camera gimbal; WindPowerYawAngle represents the wind turbine yaw angle.

[0114] Step four, input the camera imaging picture in step one into the detection model (the detection model is an open source YOLO model, which can be realized by using YOLO, SSD and other detection models in deep learning), and UNet-CS segmentation network model (UNet-CS segmentation network model is a deep learning image segmentation network model based on the improvement of classic UNet architecture) to infer Mask graph (Mask graph is used for target area identification or operation control, that is, mask graph) and hub pixel coordinates (X1, Y1), (X2, Y2).

[0115] As shown in Figure 2 , first, according to the left upper (X1, Y1) and right lower (X2, Y2) pixel coordinates of the hub area obtained by the detection model, the hub area center point coordinate Centre is calculated, and the calculation formula includes:

[0116] Centre X =(X1+X2) / 2;

[0117] Centre Y =(Y1+Y2) / 2;

[0118] In the formula, Centre(Centre X , Centre Y ) represents the center point coordinate of the fan hub area; (X1, Y1) represents the left upper pixel coordinate of the fan hub area; (X2, Y2) represents the right lower pixel coordinate of the fan hub area.

[0119] In the Mask graph, the straight lines in the whole graph are extracted by the Hough straight line algorithm, and each straight line is represented by two end point coordinates(Line StartX , Line StartY )(Line EndX , Line EndY ). First, the angle of each straight line is calculated, and the calculation method includes:

[0120]

[0121] LineAngle=ARCTAN(Line K );

[0122] In the formula, (Line StartX , Line StartY )(Line EndX , Line EndY ) represents the two end point coordinates of each straight line; Line K represents the angle of each straight line; LineAngle represents the blade angle; ARCTAN(Line K ) represents the inverse tangent function.

[0123] As Figure 3 shown, Figure 3 FirstLineAngle represents the first blade angle; BladeAngle represents the fan blade angle; VerticalLine represents the tower straight line; FirstBladeLine represents the first blade straight line; Centre Y the Y-axis coordinate of the fan hub area center point; (IMAGEWIDTH, IMAGEHEIGHT) represents the camera image resolution; the filtered endpoint Y coordinate is less than or equal to the Y-axis coordinate of the fan hub area center point Centre Y and LineAngle ∈ {-100, -80}, because these straight lines are approximately vertical and all below the hub center, which are basically the edge straight lines of the tower; for the remaining straight lines, clustering is performed according to LineAngle, and the clustering result is 2 or 3 categories, because there may be a blade coinciding with the tower, and then the average angle in the two categories is calculated as the angle of the two blades, and then one of the two blade angles is selected as the first blade angle FirstLineAngle closest to -90 (tower straight line VerticalLine), but this angle is the angle between the first blade straight line and the X-axis, and the angle with the tower (fan blade angle) BladeAngle = FirstLineAngle + 90.

[0124] S2, an ENU north-east earth center coordinate system with the fan hub center as the origin is established, the fan blade area of single shooting at different positions is calculated in combination with the fan pose parameters, and a patrol route of the current fan blade is generated according to the fan blade area.

[0125] The combination of the fan pose parameters to calculate the fan blade area of single shooting at different positions and the generation of the patrol route of the current fan blade according to the fan blade area include:

[0126] The fan blade direction vector is calculated based on the fan blade angle, the fan elevation angle and the fan cone angle.

[0127] The combination of the fan pose parameters to calculate the fan blade area of single shooting at different positions and the generation of the patrol route of the current fan blade according to the fan blade area include:

[0128] The fan blade direction vector is calculated based on the fan blade angle, the fan elevation angle and the fan cone angle.

[0129] The gimbal pitch angle, the gimbal yaw angle and the shooting distance required by the camera device to aim at the fan blade on the patrol route shooting point of the fan blade are calculated.

[0130] A straight line equation of two field of view lines is defined according to the predefined camera device parameters, and the straight line equation is solved to obtain a single shot fan blade region;

[0131] According to the single shot fan blade region, the number of inspection flight path points of a single fan blade is determined, and the coordinates of each shooting point in the inspection flight direction are determined based on the self-defined flight path starting point and the fan blade direction vector;

[0132] The number of inspection flight path points of a single fan blade, the coordinates of each shooting point in the inspection flight direction, and the fan yaw angle are combined to generate the inspection flight path of the current fan blade.

[0133] In order to facilitate the understanding of the above technical solutions of the present application, the following will be described in detail in the actual process of the present application: establishing an ENU north-eastern geocentric coordinate system with the fan hub center as the origin, calculating the single shot fan blade region at different positions combined with the fan pose parameter, and generating the inspection flight path of the current fan blade according to the fan blade region:

[0134] Step 1: Establish a CSYS coordinate system (ENU north-eastern geocentric coordinate system) with the fan hub center as the origin.

[0135] Step 2: According to the fan blade angle BladeAngle, and based on the fan pose parameter (fan elevation angle WindPowerPitchAngle, fan cone angle WindPowerConeAngle), calculate the blade vector BladeUnitVector, wherein BladeId is different fan blades, and the conventional fan has three blades [1, 2, 3]. WindPowerPitchMatrix is the fan elevation angle matrix:

[0136]

[0137] element1=COS(BladeAngle BladeId )×COS(WindPowerConeAngle);

[0138] element2=SIN(WindPowerConeAngle);

[0139] element3=SIN(BladeAngle BladeId )×COS(WindPowerConeAngle);

[0140] BladeUnitVector BladeId =WindPowerPitchMatrix*[element1 element2element3];

[0141] wherein WindPowerPitcMatrix represents the wind turbine pitch matrix; WindPowerPitchAngle represents the wind turbine pitch angle; BladeUnitVector represents the direction vector of different wind turbine blades; WindPowerConeAngle represents the wind turbine cone angle; BladeAngle represents the wind turbine blade angle of different wind turbine blades; and element has no actual meaning, and is only used for individually naming three matrix elements. BladeId BladeId wherein element1, element2 and element3 have no actual meaning, and are only used for individually naming three matrix elements.

[0142] Step three, calculating the pan-tilt angle and shooting distance of the camera when it is perpendicular to the wind turbine blade on the inspection route shooting point. It is worth noting that when calculating the pan-tilt angle, only the tilt angle needs to be considered, because the direction of the inspection route is parallel to the direction of the blade, and the yaw angle is always constant, as shown in FIG. 4. Figure 4

[0143] Defining the straight line equation of BladeLine and solving BladeLine k , BladeLine b :

[0144] BladeLine k =TAN(BladeAngle Bladeld );

[0145] HubPosition z =BladeLine k ×HubPosition x +BladeLine b ;

[0146] Defining the straight line equation of BladeVerticalLine and solving BladeVerticalLine k , BladeVerticalLine b :

[0147] BladeVerticalLine k =TAN(BladeAngle BladeId +90);

[0148] CapturePosition z =BladeVerticalLine k ×CapturePosition​​x + BladeVerticalLine b

[0149] Define CameraHorizontalLine line equation and solve CameraHorizontalLine b :

[0150] CapturePosition z = CameraHorizontalLine b ;

[0151] Solve the intersection Node1 of BladeLine, BladeVerticallLine x , and bring Node1 x into BladeLine equation to get Node1 z , and solve the intersection Node2 in the same way:

[0152]

[0153] Calculate CameraHorizontalLine, PitchRange distance:

[0154]

[0155] According to the theorem of triangle, calculate NeedPitchAngle:

[0156] NeedPitchAngle = ARCSIN (PitchRange / (CameraHorizontalLine / SIN (90)));

[0157] Calculate NeedYawAngle:

[0158]

[0159] The shooting distance CaptureDistance is the distance between CapturePosition and Node1:

[0160]

[0161] In the formula, HubPosition (0, 0, 0) represents the hub center point, which is also the origin of the CSYS coordinate system; CapturePosition (x, y, z) represents the shooting waypoint coordinates; BladeLine represents the blade straight line; BladeLinek represents the slope of the blade line equation; BladeLine brepresents the intercept of the straight line equation of the blade; BladeVerticalLine represents the straight line perpendicular to the blade; CameraHorizontalLine represents the capture straight line when the camera is horizontal; NeedPitchAngle represents the gimbal required pitch angle; NeedYawAngle represents the gimbal required yaw angle; CaptureDistance represents the shooting distance; PitchRange represents the actual shooting range corresponding to the NeedPitchAngle adjustment of the pitch angle; WindPowerYawAngle represents the fan yaw angle.

[0162] Step four, calculate the captured blade range CaptureRange:

[0163] As shown in Figure 5 , according to the camera parameter FOV, define the straight line equations of two FovEdge field of view lines (FovEdge k , FovEdge b , respectively), and solve FovEdge k , FovEdge b :

[0164] FovEdge1 k = TAN(2 / FOV+NeedPitchAngle+90);

[0165] CapturePosition z =FovEdge1 k ×CapturePosition x +FovEdge1 b ;

[0166] FovEdge2 k =TAN(-2 / FOV+NeedPitchAngle+90);

[0167] CapturePosition z =FovEdge2 k ×CapturePosition x +FovEdge2 b ;

[0168] Calculate the intersection Node of the two FovEdge field of view lines and BladeLine:

[0169]

[0170] Calculate the distance of Node1, Node2, which is the single-shot blade range of the current waypoint of CaptureRange:

[0171]

[0172] In the formula, FovEdge k , FovEdge b are two FovEdge field of view lines respectively; Node1, Node2 are the intersection points of two FovEdge field of view lines FovEdge k , FovEdge b and the blade straight line BladeLine; CaptureRange represents the single-shot blade range of the current waypoint; CapturePosition(x, y, z) represents the shooting waypoint coordinates.

[0173] Step five, generate the inspection route:

[0174] As shown in Figure 6 , according to the fan blade length BladeLength, the single-shot blade area CaptureRange determines the number of inspection route waypoints of a single blade WayPointNum; based on the user-defined route starting point RouteStartPosition and blade direction vector BladeUnitVector, the coordinates of each shooting waypoint in the heading WayPointPosition WayPointId are determined, and the fan yaw angle WindPowerYawAngle is added, thereby generating the inspection route Route, wherein CEIL is the ceiling function:

[0175]

[0176] WayPointNum=CEIL(CaptureRange / BladeLength);

[0177] WayPointId={0, 1, 2...WayPointNum-1};

[0178] WayPointPosition WayPointId =WayPowerYawMatix×(RouteStartPosition+WayPointId×CaptureRange×BladeUnitVector);

[0179] Route={WayPointPosition0, WayPointPosition1,...WayPointPositionWayPointNum}WayPointNum-1};

[0180] wherein, WayPowerYawMatrix represents the current yaw angle rotation matrix of the wind turbine; BladeLength represents the length of the wind turbine blade; CaptureRange represents the single shot blade area; WayPointNum represents the number of inspection route waypoints of a single blade; RouteStartPosition represents the custom route starting point; BladeUnitVector represents the blade direction vector; WayPointPosition WayPointId represents the coordinates of each shot waypoint in the route; WindPowerYawAngle represents the yaw angle of the wind turbine; Route represents the inspection route; WayPointId represents the waypoint subscript.

[0181] S3, at the shot waypoint of the inspection route, calculate the required shot wind turbine blade endpoint ENU coordinates, and convert the wind turbine blade endpoint ENU coordinates into wind turbine blade endpoint image coordinate system coordinates.

[0182] wherein, at the shot waypoint of the inspection route, calculating the required shot wind turbine blade endpoint ENU coordinates, and converting the wind turbine blade endpoint ENU coordinates into wind turbine blade endpoint image coordinate system coordinates comprises:

[0183] at the shot waypoint of the inspection route, calculate the current required shot wind turbine blade area endpoint ENU coordinates;

[0184] establish a field of view imaging body according to the camera device parameters and the shot target point, and convert the wind turbine blade area endpoint ENU coordinates to camera imaging body coordinates based on the field of view imaging body;

[0185] convert the camera imaging body coordinates to pixel coordinates in the camera image by combining the cosine theorem and the sine theorem, to obtain the wind turbine blade area endpoint pixel coordinates;

[0186] segment the camera imaging picture by using a deep learning segmentation network model to obtain a segmentation mask image, and extract the wind turbine blade endpoint image coordinate system coordinates in the segmentation mask image.

[0187] wherein, segmenting the camera imaging picture by using a deep learning segmentation network model to obtain a segmentation mask image, and extracting the wind turbine blade endpoint image coordinate system coordinates in the segmentation mask image comprises:

[0188] input the camera imaging picture into the deep learning segmentation network model to infer the segmentation mask image, calculate the connected regions in the segmentation mask image, retain the connected regions exceeding a preset threshold, and fill the gap pixels in the connected regions;

[0189] The fan blade trajectory slope is calculated based on the fan blade region endpoint pixel coordinates, and the fan blade trajectory angle is calculated according to the fan blade trajectory slope;

[0190] The current fan blade trajectory direction is determined based on the fan blade trajectory slope, and the fan blade endpoint image coordinate system coordinates are searched in the segmentation mask image according to the fan blade trajectory direction.

[0191] The current fan blade trajectory direction is determined based on the fan blade trajectory slope, and the fan blade endpoint image coordinate system coordinates are searched in the segmentation mask image according to the fan blade trajectory direction.

[0192] The current fan blade trajectory direction is determined based on the fan blade trajectory slope, and the fan blade endpoint image coordinate system coordinates are searched in the segmentation mask image according to the fan blade trajectory direction.

[0193] If the fan blade is horizontal, a fixed sliding window is sequentially slid in the up and down direction of the Y axis of the segmentation mask image in the opposite direction until the fan blade region pixel is contained in the sliding window. Figure X If the fan blade is horizontal, a fixed sliding window is sequentially slid in the up and down direction of the Y axis of the segmentation mask image in the opposite direction until the fan blade region pixel is contained in the sliding window.

[0194] In order to facilitate the understanding of the above technical solutions of the present application, the following will be described in detail in the actual process of the present application on the inspection route shooting waypoint, calculating the required shooting fan blade endpoint ENU coordinates, and converting the fan blade endpoint ENU coordinates into fan blade endpoint image coordinate system coordinates:

[0195] Step one, as shown in the formula, calculate the shooting blade region endpoint TargetStartPosition, TargerEndPosition: Figure 7

[0196] TargetStartPosition WayPointId = HubPosition + WayPointId x BladeUnitVector;

[0197] TargetEndPosition WayPointId = HubPosition + (WayPointId + 1) x BladeUnitVector;

[0198] In the formula, BladeUnitVector represents the blade direction vector; HubPosition represents the hub center point.

[0199] Step two, as shown in the formula, calculate the fan blade endpoint image coordinate system coordinates: Figure 8 ​As shown, according to the camera parameters and the target point to be photographed, the field of view imaging body FovSpace is established:

[0200] According to the yaw angle NeedYawAngle and the pitch angle NeedPitchAngle of the holder, the direction vector FovLineUnitVector of the camera sight line (which is actually the normal vector of the FovBottomPlane) is calculated, and the plane equation FovBottomPlane of the bottom surface of the field of view imaging body is solved based on the coordinate origin, that is, the fan hub center point HubPosition(x, y, z) and the sight line direction vector:

[0201]

[0202] FovBottomPlane d =FovLineUnitVector x ×HubPosition x +FovLineUnitVector y ×HubPosition y +FovLineUnitVector z ×HubPosition z ;

[0203] Secondly, the vertical distance TargetDistance between the photographed blade area end point (TargetStartPosition, TargetEndPosition) and the bottom surface FovBottomPlane of the field of view imaging body is calculated, and the longest distance is MaxTargetDistance as the height FovSpace of the field of view imaging body z , and based on the longitudinal field of view CameraFov vertical and the horizontal field of view CameraFov horizontal of the camera parameters, the length FovSpace y and the width FovSpace x of the field of view imaging body are calculated:

[0204] molecule=ABS(FovLineUnitVector x ×TargetStartPosition x +FovLineUnitVector y ×TargetStartPosition y +FovLineUnitVectorz×TargetStartPosition z+FovBottomPlane d );

[0205]

[0206] TargetDistance TargetEndPosition , and calculate MaxTargetDistance as the height FovSpace z of the camera field of view body, and the width FovSpace x of the camera field of view body, and the length FovSpace y of the camera field of view body:

[0207] FovSpace z = MAX(TargetDistance TargetStartPosition , TargetDistance TargetEndPosition );

[0208] FovSpace x = 2 x FovSpace z x TAN(CameraFov horizontal );

[0209] FovSpace y = 2 x FovSpace z x TAN(CameraFov vertical );

[0210] As shown in Figure 9 and Figures 18-20 , finally determine the upper left FovTopPlaneLT and the upper right FovTopPlaneRT coordinates of the top surface FovTopPlane of the camera field of view body of TargetStartPosition and TargetEndPosition according to the current waypoint position WayPointPosition and the pitch angle NeedPitchAngle and the yaw angle NeedYawAngle, and set FovSpace z = TargetDistance TargetStartPosition , that is, TargetDistance TargetStartPosition > TargetDistance TargetEndPosition :

[0211]

[0212] Step three, as shown in Figures 10-11 , convert the camera field of view body coordinates of the target point to pixel coordinates in the camera image.

[0213] The relative horizontal and vertical coordinates of TargetStartPosition, TargetEndPosition in FovTopPlane are calculated by the cosine theorem and the sine theorem respectively, and are normalized and mapped into the camera image coordinates, so that the pixel coordinates of the target point in the camera image PixerPosition are obtained (here, the imaging distortion of the camera itself is not considered), and the following is listed for TargetStartPosition, and TargetEndPosition can be obtained in the same way, where IMAGEWIDTH and IMAGEHEIGHT are the camera image resolution.

[0214]

[0215] TargetStartPosition FovX = ToLTDistance TargetStartPositon / SIN(90) x SIN(90-A);

[0216] TargetStartPosition FovY = ToLTDistanceTargetStartPositon / SIN(90) x SIN(180-90-(90-A));

[0217] SMPixerPosition x = TargetStartPosition FovX / FovSpace x x IMAGEWIDTH;

[0218] SMPixerPosition y = TargetStartPosition Fovy / FovSpace y x IMAGEHEIGHT;

[0219] Step four, after obtaining the preliminary blade area coordinates and blade trajectory through spatial mapping, the UNet-CS network model is used to segment the camera captured picture, and the fan blade endpoint image coordinates are extracted in the segmentation result Mask graph.

[0220] The camera picture image is input into the UNet-CS model (the UNet-CS network here is based on the deep learning open source model UNet and is optimized and improved, mainly in that a spatial and channel attention mechanism is added in the down-sampling process of the model, so that the segmentation effect of the model is further improved, and here the network structure, training process and inference process are not introduced in detail, and the leaf extraction process of the model segmentation result Mask graph is mainly explained), and the Mask graph of the current picture is inferred, as shown in Figure 12

[0221] After obtaining the leaf Mask graph, firstly, the connected regions in the Mask graph are calculated, the connected regions exceeding the threshold are retained, and the gap pixels in the connected regions are filled; secondly, according to the calculated leaf region endpoint pixel coordinates, the leaf trajectory slope is calculated, and the leaf trajectory angle BladeLineAngle SM is determined, wherein SM represents the coordinate obtained by spatial mapping:

[0222]

[0223] BladeLineAngle SM =ARCTAN(BadeLineK SM );

[0224] As shown in Figures 13-14 , according to the slope, the direction of the current leaf trajectory is judged, and then based on the direction, the leaf region endpoint is searched in the Mask graph, wherein the searching method includes:

[0225] If the leaf is longitudinal, the fixed sliding window WINDOWSIZE is sequentially slid in the upward and downward directions of the Y axis of the image in the reverse direction until the leaf region pixels are contained in the sliding window; if it is transverse, the fixed sliding window WINDOWSIZE is sequentially slid in the left and right directions of the X axis of the image in the reverse direction until the leaf region pixels are contained in the sliding window; and the mass point coordinates of the leaf region are extracted as the pixel coordinates CVPixerPosition of the leaf endpoint, so that multiple groups of pixel coordinates of the leaf endpoint (there may be multiple connected regions) are obtained, wherein CV represents the coordinate obtained by visual processing:

[0226]

[0227] S4, combining the fan blade region image coordinate system coordinate and the fan blade endpoint image coordinate system coordinate, the final fan blade endpoint image coordinate is generated.

[0228] ​For the convenience of understanding the above technical solutions of the present application, the following will be described in detail in combination with the image coordinates of the fan blade region and the image coordinates of the fan blade end point in the actual process to generate the final image coordinates of the fan blade end point:

[0229] As shown in Figure 15 After the spatial mapping calculation and visual positioning, SMPixerPosition and multiple sets of CVPixerPosition coordinates are obtained, and now SMPixerPosition coordinates are needed to verify CVPixerPosition coordinates (the actual meaning here is that in the same picture, a set of SMPixerPosition and multiple sets of CVPixerPosition pixel coordinates of the blade region are obtained, and need to be selected; in the actual scene, CVPixerPosition is used first, because in all calculations of SMPixerPosition, the trajectory of the blade is always a straight line, but the real blade is not a pure straight line, but will be affected by pre-bending, gravity and wind speed, etc. to become a curve, and this blade curve cannot be fitted by a function, so the trajectory of the SMPixerPosition coordinate is more accurate, and the specific coordinate value is closer to the blade tip, the deviation is larger, so the visual coordinate is used first, but because multiple fan regions may appear in the same picture, and the model cannot distinguish the current photographed blade region in detail, SMPixerPosition is needed to filter multiple CVPixerPosition to retain a set of CVPixerPosition coordinates closest to SMPixerPosition), the method is as follows:

[0230] Loop to calculate the rectangular union UnionArea and intersection IntersectionArea of each set of CVPixerPosition and SMPixerPosition, and calculate the proportion of the intersection in the union, retain the set of CVPixerPosition with the largest proportion, if the proportion of the CVPixerPosition is greater than or equal to the set threshold, then the set of CVPixerPosition is used as the final blade region pixel coordinate PixerPosition, otherwise, SMPixerPosition is used as the final blade region pixel coordinate PixerPosition.

[0231] S5, based on the final fan blade end point image coordinates, calculate the matrix shooting array corresponding to the required shooting region, and mobilize the holder to sequentially zoom in shooting on the matrix shooting array.

[0232] Among them, based on the final fan blade end point image coordinates, calculating the matrix shooting array corresponding to the required shooting region includes:

[0233] Based on the final fan blade endpoint image coordinates, a fan blade trajectory straight line equation is established;

[0234] The focal length corresponding to the current shooting zoom ratio of the camera device is calculated, the field of view range of each matrix shooting array grid corresponding to the current shooting zoom ratio is calculated based on the focal length, and the pixel range of the matrix shooting array grid on the camera imaging picture is calculated according to the field of view range.

[0235] Based on the final fan blade endpoint image coordinates, the fan blade direction is determined, and the matrix shooting array grid is inserted on the fan blade trajectory based on the fan blade direction.

[0236] In order to facilitate the understanding of the above technical solutions of the present application, the following will be based on the final fan blade endpoint image coordinates of the present application in the actual process, the corresponding matrix shooting array of the required shooting area is calculated, and the pan-tilt is mobilized to perform zoom shooting on the matrix shooting array in sequence. Detailed description:

[0237] Step one, according to the blade endpoint image coordinates StartPixerPosition, EndPixerPosition, the blade trajectory straight line equation BladeAreaLine is established:

[0238]

[0239] BladeAreaLine b =StartPixerPosition y -(StartPixerPosition x ×BladeAreaLine k );

[0240] Step two, calculate the field of view range ZoomFov captured by the current shooting zoom ratio ZOOMRATIO and the corresponding picture pixel range, first calculate the focal length FLenght corresponding to ZOOMRATIO, then calculate GFov (that is, the field of view range of each high-efficiency array grid) corresponding to the current zoom ratio through the focal length, and finally calculate the pixel range GPixer of each high-efficiency matrix grid on the image; wherein MINFLENGHT is the minimum focal length of the camera, CMOS is the size of the camera cmos, FOV is the maximum field of view range of the camera, DPI is the image resolution of the camera, H / V is the horizontal and vertical direction:

[0241] The calculation formula of the focal length corresponding to the current shooting zoom ratio of the camera device is:

[0242] FLenght=MINFLENGHT×ZOOMRATIO;

[0243] The calculation formula of the field of view range of each matrix shooting array grid corresponding to the current shooting zoom ratio is:

[0244]

[0245] The calculation formula of the pixel range of the matrix shooting array grid on the camera imaging picture is:

[0246]

[0247] In the formula, FLenght represents the focal length corresponding to the current shooting zoom ratio of the camera device; MINFLENGHT represents the minimum focal length of the camera device; ZOOMRATIO represents the current shooting zoom ratio of the camera device; GFov represents the field of view range of each matrix shooting array grid corresponding to the current shooting zoom ratio; CMOS represents the CMOS size of the camera device; ARCTAN represents the inverse tangent function; H / V represents the horizontal direction; Gpixer represents the pixel range of the matrix shooting array grid on the camera imaging picture; FOV represents the maximum field of view range of the camera device; and DPI represents the image resolution of the camera device.

[0248] Step three, determining the blade direction according to StartPixerPosition and EndPixerPosition, the principle is the same as the above-mentioned search method, when the blade direction is the horizontal direction, GPixer is used H As an interval, the high-efficiency matrix grid is inserted on the blade track (GPixer H As the x value is substituted into the BladeAreaLine equation, the y value is obtained, so the pixel coordinates of each high-efficiency matrix grid are determined); when the blade direction is the vertical direction, the same principle is used for solving; and the schematic diagram of the high-efficiency matrix array in the horizontal and vertical blades is as shown in Figures 16-17 .

[0249] In summary, with the above technical solutions of the present application, the fan inspection method based on the high-efficiency matrix shooting of the unmanned aerial vehicle provided by the present application effectively solves the problem that the fan blade photos are not clear enough in the inspection process, realizes high-definition and high-zoom shooting of the fan, and at the same time, the method determines the complete inspection flight track and accurate shooting action in the flight path generation stage, and avoids the real-time decision-making requirement in the inspection process, thereby improving the shooting accuracy and significantly improving the inspection efficiency.

[0250] The above only describes the preferred embodiments of the present application and should not be used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A wind turbine inspection method based on efficient matrix imaging by unmanned aerial vehicles (UAVs), characterized in that, The wind turbine inspection method includes: Collect wind turbine data and calculate wind turbine attitude parameters based on the wind turbine data. The wind turbine attitude parameters include the wind turbine hub center positioning coordinates, wind turbine blade length, wind turbine yaw angle, wind turbine blade angle, wind turbine elevation angle, and wind turbine cone angle. Establish the ENU Northeast Geocentric Coordinate System with the center of the wind turbine hub as the origin, calculate the wind turbine blade area of ​​a single shot at different positions in combination with the wind turbine pose parameters, and generate the current inspection route of the wind turbine blade based on the wind turbine blade area. At the shooting waypoints of the inspection route, calculate the ENU coordinates of the wind turbine blade endpoints to be photographed, and convert the ENU coordinates of the wind turbine blade endpoints into the wind turbine blade endpoint image coordinate system coordinates; By combining the coordinates of the wind turbine blade region image coordinate system and the coordinates of the wind turbine blade tip image coordinate system, the final wind turbine blade tip image coordinates are generated. Based on the final image coordinates of the wind turbine blade endpoints, the matrix shooting array corresponding to the required shooting area is calculated, and the gimbal is adjusted to perform zoom shooting on the matrix shooting array in sequence. The process of collecting wind turbine data and calculating wind turbine pose parameters based on the wind turbine data includes: Align the center of the camera image on the drone with the center of the wind turbine hub, and use the lidar on the drone to collect wind turbine data. The wind turbine data includes the distance between the UAV and the center point of the wind turbine hub, the current yaw angle and pitch angle of the camera gimbal, and the current positioning coordinates of the UAV. The wind turbine hub center positioning coordinates are calculated based on wind turbine data, and the current yaw angle of the camera gimbal is reversed as the wind turbine yaw angle. The camera image is input into a deep learning segmentation network model, which generates a mask image and the pixel coordinates of the wind turbine hub. The wind turbine blade angle is then calculated by combining the mask image and the pixel coordinates of the wind turbine hub.

2. The wind turbine inspection method based on efficient matrix imaging by unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The step of inputting the camera image into a deep learning segmentation network model, using the deep learning segmentation network model to generate a mask image and the pixel coordinates of the wind turbine hub, and combining the mask image and the pixel coordinates of the wind turbine hub to calculate the wind turbine blade angle includes: The camera image is input into a deep learning segmentation network model, and the mask image and the pixel coordinates of the wind turbine hub are obtained by reasoning using the deep learning segmentation network model. The coordinates of the center point of the wind turbine hub region are calculated based on the pixel coordinates of the wind turbine hub. The formula for calculating the coordinates of the center point of the hub region is as follows: Center X <(X1+X2) / 2; Centre Y =(Y1+Y2) / 2; In the formula, Centre(Centre) X Centre Y (X1, Y1) represents the coordinates of the center point of the wind turbine hub area; (X2, Y2) represents the coordinates of the top left pixel of the wind turbine hub area; (X2, Y2) represents the coordinates of the bottom right pixel of the wind turbine hub area. The Hough line algorithm is used to extract lines represented by the coordinates of two endpoints from the mask image, and the angle of each line is calculated. Filter out straight lines that meet the preset conditions, and cluster the remaining straight lines based on the angle of each line. After the clustering is completed, take the average angle in each category as the blade angle. Select the angle closest to the straight line of the tower from the blade angle as the first wind turbine blade angle closest to the tower, and calculate the wind turbine blade angle based on the first wind turbine blade angle.

3. The wind turbine inspection method based on efficient matrix imaging by unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The process of calculating the wind turbine blade area at different locations using wind turbine pose parameters and generating the current inspection route for the wind turbine blades based on the wind turbine blade area includes: Calculate the direction vector of the fan blades based on the fan blade angle, fan elevation angle and fan cone angle; Calculate the required gimbal pitch angle, gimbal yaw angle, and shooting distance for the camera equipment to be aimed at the wind turbine blades along the inspection route and shooting waypoints. Define the linear equations of two field lines based on the predefined camera equipment parameters, and solve the linear equations to obtain the wind turbine blade area in a single shot; Based on the wind turbine blade area captured in a single shot, the number of waypoints for the inspection route of a single wind turbine blade is determined, and the coordinates of each shooting waypoint in the inspection route are determined based on the custom route start point and the wind turbine blade direction vector. The inspection route for the current wind turbine blade is generated by combining the number of waypoints on the inspection route of a single wind turbine blade, the coordinates of each shooting waypoint in the inspection route, and the wind turbine yaw angle.

4. The wind turbine inspection method based on efficient matrix imaging by unmanned aerial vehicles according to claim 3, characterized in that, The calculation of the wind turbine blade direction vector based on the wind turbine blade angle, wind turbine elevation angle, and wind turbine cone angle includes: Calculate the wind turbine elevation angle matrix. The expression for the wind turbine elevation angle matrix is: The direction vector of the wind turbine blades is calculated based on the blade angle, elevation angle, cone angle, and elevation angle matrix. The expression for the direction vector of the wind turbine blades is as follows: element1=COS(BladeAngle BladeId )×COS(WindPowerConeAngle); element2=SIN(WindPowerConeAngle); element3<SIN(BladeAngle BladeId )×COS(WindPowerConeAngle); BladeUnitVector BladeId =WindPowerPitchMatrix* [element1, element2, element3]; In the formula, WindPowerPitchMatrix represents the wind turbine pitch angle matrix; WibdPowerPitchAngle represents the wind turbine pitch angle; BladeUnitVector BladeId Represents the direction vector of different wind turbine blades; WindPowerConeAngle represents the wind turbine cone angle; BladeAngle BladeId This indicates the blade angle of different fan blades.

5. The wind turbine inspection method based on efficient matrix imaging by unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The process of calculating the ENU coordinates of the wind turbine blade tip at the shooting waypoints along the inspection route and converting these ENU coordinates into wind turbine blade tip image coordinates includes: Calculate the ENU coordinates of the endpoint of the wind turbine blade area to be photographed at the shooting point on the inspection route. Based on the camera equipment parameters and the target shooting point, a field of view imaging volume is established, and the ENU coordinates of the end point of the wind turbine blade area are transformed to the coordinates of the camera imaging volume based on the field of view imaging volume. By combining the cosine and sine laws, the camera imaging volume coordinates are transformed into pixel coordinates in the camera image, thus obtaining the pixel coordinates of the end point of the wind turbine blade region. A deep learning segmentation network model is used to segment the camera image to obtain a segmentation mask image. The coordinates of the wind turbine blade endpoint image coordinate system are then extracted from the segmentation mask image.

6. The wind turbine inspection method based on efficient matrix imaging by unmanned aerial vehicles according to claim 5, characterized in that, The process of segmenting the camera image using a deep learning segmentation network model to obtain a segmentation mask image, and extracting the coordinates of the wind turbine blade endpoint image from the segmentation mask image, includes: The camera image is input into a deep learning segmentation network model to obtain a segmentation mask map. Connected regions in the segmentation mask map are calculated, connected regions exceeding a preset threshold are retained, and gap pixels in the connected regions are filled. The slope of the wind turbine blade trajectory is calculated based on the pixel coordinates of the endpoints of the wind turbine blade region, and the angle of the wind turbine blade trajectory is calculated based on the slope of the wind turbine blade trajectory. The current wind turbine blade trajectory direction is determined based on the slope of the wind turbine blade trajectory, and the coordinates of the wind turbine blade endpoint image coordinate system are searched in the segmented mask image based on the wind turbine blade trajectory direction.

7. A wind turbine inspection method based on efficient matrix imaging by unmanned aerial vehicles (UAVs) according to claim 6, characterized in that, The process of determining the current wind turbine blade trajectory direction based on the slope of the wind turbine blade trajectory, and searching for the coordinates of the wind turbine blade endpoint image coordinate system in the segmented mask image based on the wind turbine blade trajectory direction, includes: The current wind turbine blade trajectory direction is determined based on the slope of the wind turbine blade trajectory. If the wind turbine blade is vertical, a fixed sliding window is used to slide in the opposite direction in the up and down direction of the Y-axis of the segmented mask image until the sliding window contains the pixels of the wind turbine blade area. If the wind turbine blade is horizontal, then slide a fixed sliding window in the left and right directions of the segmentation mask image along the X-axis in opposite directions until the sliding window contains the pixels of the wind turbine blade region; extract the mass point coordinates of the wind turbine blade region as the coordinates of the wind turbine blade endpoint image coordinate system.

8. The wind turbine inspection method based on efficient matrix imaging by unmanned aerial vehicles according to claim 1, characterized in that, The calculation of the matrix imaging array corresponding to the required imaging area based on the final wind turbine blade tip image coordinates includes: Based on the final image coordinates of the wind turbine blade endpoints, a linear equation for the wind turbine blade trajectory is established. Calculate the focal length corresponding to the current zoom level of the camera device, calculate the field of view of each matrix shooting array grid corresponding to the current zoom level based on the focal length, and calculate the pixel range of the matrix shooting array grid on the camera image based on the field of view. The wind turbine blade orientation is determined based on the final image coordinates of the wind turbine blade endpoints, and a matrix imaging array grid is inserted into the wind turbine blade trajectory based on the wind turbine blade orientation.

9. A wind turbine inspection method based on efficient matrix imaging by unmanned aerial vehicles (UAVs) according to claim 8, characterized in that, The formula for calculating the focal length corresponding to the current zoom level of the camera device is as follows: FLenght=MINFLENGHT×ZOOMRATIO; The formula for calculating the field of view of each matrix shooting array grid corresponding to the current shooting zoom level is as follows: The formula for calculating the pixel range of the matrix imaging array grid on the camera image is as follows: In the formula, FLENGHT represents the focal length corresponding to the current zoom level of the camera device; MINFLENGHT represents the minimum focal length of the camera device; ZOOMRATIO represents the current zoom level of the camera device; GFov represents the field of view of each matrix shooting array grid corresponding to the current zoom level; CMOS represents the CMOS size of the camera device; ARCTAN represents the arctangent function; H / V represents the horizontal direction; GPixer represents the pixel range of the matrix shooting array grid on the camera's image screen; FOV represents the maximum field of view of the camera device; and DPI represents the image resolution of the camera device.

Images