Fan inspection method based on efficient matrix shooting of unmanned aerial vehicle

Abstract

The invention discloses a fan inspection method based on efficient matrix shooting of an unmanned aerial vehicle, and the method comprises the steps: collecting fan data, and calculating a fan pose parameter based on the fan data; fan blade areas shot at different positions at a single time are calculated in combination with the fan pose parameters, and an inspection route of the current fan blade is generated according to the fan blade areas; on a shooting waypoint of the inspection route, fan blade endpoint ENU coordinates needing to be shot are calculated, and the fan blade endpoint ENU coordinates are converted into fan blade endpoint image coordinate system coordinates; combining the coordinates of the fan blade area image coordinate system and the coordinates of the fan blade endpoint image coordinate system to generate final fan blade endpoint image coordinates; and based on the final fan blade end point image coordinates, calculating a matrix shooting array corresponding to the area needing to be shot. According to the invention, the problem that the picture of the fan blade is not clear enough in the inspection process is effectively solved, and high-definition and high-zoom shooting of the fan is realized.

Description

Technical Field

[0001] The present invention relates to the field of camera imaging technology, and in particular to a wind turbine inspection method based on efficient matrix photography by unmanned aerial vehicles. Background Art

[0002] In the field of wind power generation, the condition monitoring of wind turbine blades and nacelles is crucial to improving the operating efficiency of wind turbines and extending the life of equipment. In recent years, wind turbine inspection technology based on drones has gradually attracted attention. Drones can quickly cover the entire wind farm, and the time to inspect wind turbines is usually only a small fraction of the time of traditional manual inspections. For large wind farms, drones can complete the inspection of multiple wind turbines in a short period of time, significantly improving inspection efficiency.

[0003] Among the existing drone inspection photography methods, there are two main types: static photography and dynamic photography. Static photography is due to the fact that the waypoints in the drone inspection route cannot be too close, resulting in a large coverage range for a single shot, which limits the accuracy of the photos taken and fails to give full play to the performance advantages of the camera. The dynamic photography method attempts to solve the above problems by performing inspections while the drone is flying. However, due to the constant changes in the camera position during flight, problems such as dynamic blur are prone to occur during the shooting process, which in turn affects the clarity of the photos.

[0004] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a wind turbine inspection method based on high-efficiency matrix photography by unmanned aerial vehicles, which has the advantage of high shooting accuracy, thereby solving the problem that the photos of wind turbine blades are not clear enough during the inspection process.

[0007] (II) Technical solution

[0008] In order to achieve the above-mentioned advantage of high shooting accuracy, the specific technical solution adopted by the present invention is as follows:

[0009] A wind turbine inspection method based on efficient matrix photography of a drone, the wind turbine inspection method comprising:

[0010] Collect wind turbine data and calculate wind turbine posture parameters based on the wind turbine data, and the wind turbine posture parameters include wind turbine hub center positioning coordinates, wind turbine blade length, wind turbine yaw angle, wind turbine blade angle, wind turbine pitch angle and wind turbine cone angle;

[0011] Establish an ENU northeast geocentric coordinate system with the center of the wind turbine hub as the origin, calculate the wind turbine blade area of ​​a single shot at different positions based on the wind turbine posture parameters, and generate the inspection route of the current wind turbine blade based on the wind turbine blade area;

[0012] At the shooting waypoint of the inspection route, the ENU coordinates of the wind turbine blade endpoint to be photographed are calculated, and the ENU coordinates of the wind turbine blade endpoint are converted into the coordinates of the wind turbine blade endpoint image coordinate system;

[0013] The coordinates of the fan blade region image coordinate system and the coordinates of the fan blade endpoint image coordinate system are combined to generate the final fan blade endpoint image coordinates;

[0014] Based on the final image coordinates of the wind turbine blade endpoints, the matrix shooting array corresponding to the required shooting area is calculated, and the pan / tilt is mobilized to zoom and shoot the matrix shooting array in sequence.

[0015] Preferably, collecting the fan data and calculating the fan posture parameters based on the fan data includes:

[0016] Aim the center of the camera image of the camera device carried by the drone at the center point of the wind turbine hub, and use the laser radar carried by the drone to collect wind turbine data;

[0017] The wind turbine data includes the distance between the drone and the center point of the wind turbine hub, the current yaw angle and pitch angle of the camera gimbal, and the current positioning coordinates of the drone;

[0018] Calculate the center coordinates of the wind turbine hub based on the wind turbine data, and use the current yaw angle of the camera gimbal as the wind turbine yaw angle;

[0019] The camera imaging image is input into the deep learning segmentation network model, and the deep learning segmentation network model is used to generate a mask image and fan hub pixel coordinates, and the fan blade angle is calculated in combination with the mask image and fan hub pixel coordinates.

[0020] Preferably, the camera imaging picture is input into a deep learning segmentation network model, a mask image and fan hub pixel coordinates are generated by using the deep learning segmentation network model, and the fan blade angle is calculated in combination with the mask image and the fan hub pixel coordinates, including:

[0021] The camera image is input into the deep learning segmentation network model, and the deep learning segmentation network model is used to infer the mask image and the pixel coordinates of the wind turbine hub;

[0022] The coordinates of the center point of the fan hub area are calculated based on the pixel coordinates of the fan hub. The calculation formula for the coordinates of the center point of the hub area is:

[0023] Centre X =(X1+X2) / 2;

[0024] Centre Y =(Y1+Y2) / 2;

[0025] In the formula, Centre(Centre X , Centre Y ) represents the coordinates of the center point of the fan hub area; (X1, Y1) represents the coordinates of the upper left pixel of the fan hub area; (X2, Y2) represents the coordinates of the lower right pixel of the fan hub area;

[0026] The Hough line algorithm is used to extract the lines represented by the coordinates of the two endpoints from the mask image, and the angle of each line is calculated;

[0027] Filter the straight lines that meet the preset conditions, and cluster the remaining straight lines based on the angle of each straight line. After clustering, the average angle in each category is used as the blade angle;

[0028] An angle closest to the tower straight line is selected from the blade angles as a first wind turbine blade angle closest to the tower, and the wind turbine blade angle is calculated based on the first wind turbine blade angle.

[0029] Preferably, calculating the wind turbine blade area captured in a single shot at different positions in combination with the wind turbine posture parameters, and generating the inspection route of the current wind turbine blade according to the wind turbine blade area includes:

[0030] Calculate the fan blade direction vector based on the fan blade angle, fan elevation angle and fan cone angle;

[0031] Calculate the pan / tilt angle, pan / tilt yaw angle and shooting distance required for the camera to aim at the wind blades at the inspection route shooting point of the wind blades;

[0032] Define the straight line equations of the two field of view lines according to the predefined camera device parameters, and solve the straight line equations to obtain the wind turbine blade area captured in a single shot;

[0033] According to the wind turbine blade area captured in a single shot, the number of inspection route waypoints for a single wind turbine blade is determined, and based on the custom route starting point and wind turbine blade direction vector, the coordinates of each shooting waypoint in the inspection heading are determined;

[0034] The number of inspection route waypoints of a single wind turbine blade, the coordinates of each shooting waypoint in the inspection heading and the yaw angle of the wind turbine are combined to generate the inspection route of the current wind turbine blade.

[0035] Preferably, calculating the fan blade direction vector based on the fan blade angle, the fan pitch angle and the fan cone angle comprises:

[0036] Calculate the wind turbine elevation matrix, and the expression of the wind turbine elevation matrix is:

[0037]

[0038]

[0039] The fan blade direction vector is calculated based on the fan blade angle, fan elevation angle, fan cone angle and fan elevation angle matrix, and the expression of the fan blade direction vector is:

[0040] element1=COS(BladeAngle BladeId )×COS(WindPowerConeAngle);

[0041] element2=SIN(WindPowerConeAngle);

[0042] element3 = SIN(BladeAngle BladeId )×COS(WindPowerConeAngle);

[0043] BladeUnitVector BladeId =WindPowerPitchMatrix*[element1 element2element3];

[0044] Where WindPowerPitchMatrix represents the wind turbine pitch matrix; WindPowerPitchAngle represents the wind turbine pitch angle; BladeUnitVector BladeId Indicates the direction vector of different fan blades; WindPowerConeAngle indicates the fan cone angle; BladeAngle BladeId Represents the fan blade angle of different fan blades.

[0045] Preferably, at the shooting waypoint of the inspection route, calculating the ENU coordinates of the wind blade endpoint to be photographed, and converting the ENU coordinates of the wind blade endpoint to the coordinates of the wind blade endpoint image coordinate system includes:

[0046] At the shooting waypoint of the inspection route, calculate the ENU coordinates of the endpoint of the wind turbine blade area that needs to be photographed;

[0047] Establish a field of view imaging volume according to the camera equipment parameters and the shooting target point, and convert the ENU coordinates of the end points of the wind turbine blade area into the camera imaging volume coordinates based on the field of view imaging volume;

[0048] Combining the cosine theorem and the sine theorem, the camera imaging volume coordinates are converted to the pixel coordinates in the camera image to obtain the pixel coordinates of the end points of the wind turbine blade area;

[0049] The camera imaging picture is segmented using a deep learning segmentation network model to obtain a segmentation mask map, and the image coordinate system coordinates of the wind turbine blade endpoints are extracted from the segmentation mask map.

[0050] Preferably, the camera imaging picture is segmented using a deep learning segmentation network model to obtain a segmentation mask map, and the coordinates of the wind turbine blade endpoint image coordinate system are extracted from the segmentation mask map, including:

[0051] The camera image is input into the deep learning segmentation network model to infer the segmentation mask map, the connected areas in the segmentation mask map are calculated, the connected areas exceeding the preset threshold are retained, and the empty pixels in the connected areas are filled;

[0052] The fan blade trajectory slope is calculated based on the pixel coordinates of the end points of the fan blade region, and the fan blade trajectory angle is calculated based on the fan blade trajectory slope;

[0053] The current direction of the fan blade trajectory is determined based on the fan blade trajectory slope, and the image coordinate system coordinates of the fan blade endpoints are searched in the segmentation mask image according to the fan blade trajectory direction.

[0054] Preferably, judging the current direction of the fan blade trajectory based on the fan blade trajectory slope, and searching for the fan blade endpoint image coordinates in the segmentation mask image according to the fan blade trajectory direction comprises:

[0055] The current direction of the fan blade trajectory is determined based on the slope of the fan blade trajectory. If the fan blade is longitudinal, a fixed sliding window is sequentially slid in the opposite direction in the up and down directions of the Y axis of the segmentation mask until the sliding window contains the fan blade area pixels.

[0056] If the fan blades are horizontal, then the split mask Figure X The left and right directions of the axis are slid in the opposite direction in sequence with a fixed sliding window until the sliding window contains the pixels of the fan blade area; the coordinates of the particles in the fan blade area are extracted as the coordinates of the fan blade endpoint image coordinate system.

[0057] Preferably, based on the final wind turbine blade endpoint image coordinates, calculating the matrix shooting array corresponding to the required shooting area includes:

[0058] Based on the final image coordinates of the fan blade endpoints, the fan blade trajectory straight line equation is established;

[0059] Calculate the focal length corresponding to the current shooting zoom ratio of the camera device, calculate the field of view of each matrix shooting array grid corresponding to the current shooting zoom ratio based on the focal length, and calculate the pixel range of the matrix shooting array grid on the camera imaging screen according to the field of view;

[0060] The direction of the fan blade is determined based on the final image coordinates of the fan blade endpoints, and a matrix shooting array grid is inserted on the fan blade trajectory based on the direction of the fan blade.

[0061] Preferably, the calculation formula for the focal length corresponding to the shooting zoom ratio of the current camera device is:

[0062] FLenght=MINFLENGHT×ZOOMRATIO;

[0063] The calculation formula for the field of view of each matrix shooting array grid corresponding to the current shooting zoom ratio is:

[0064]

[0065] The calculation formula for the pixel range of the matrix shooting array grid on the camera imaging screen is:

[0066]

[0067] Wherein, FLencht represents the focal length corresponding to the current camera device's shooting zoom ratio; MINFLENGHT represents the minimum focal length of the camera device; ZOOMRATIO represents the current camera device's shooting zoom ratio; GFov represents the field of view of each matrix shooting array grid corresponding to the current shooting zoom ratio; CMOS represents the camera device CMOS size; ARCTAN represents the inverse tangent function; H / V represents the horizontal direction; GPixer represents the pixel range of the matrix shooting array grid on the camera imaging screen; FOV represents the maximum field of view of the camera device; and DPI represents the image resolution of the camera device.

[0068] (III) Beneficial effects

[0069] Compared with the prior art, the present invention provides a wind turbine inspection method based on efficient matrix photography by drones, which has the following features:

[0070] Beneficial effects:

[0071] The wind turbine inspection method based on high-efficiency matrix shooting of unmanned aerial vehicles provided by the present invention effectively solves the problem of unclear photos of wind turbine blades during the inspection process, and realizes high-definition and high-zoom shooting of wind turbines. At the same time, the method determines the complete inspection route flight trajectory and precise shooting actions in the route generation stage, and avoids the need for real-time decision-making during the inspection process, thereby significantly improving the inspection efficiency while improving the shooting accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 is a flow chart of a wind turbine inspection method based on efficient matrix photography by a drone according to an embodiment of the present invention;

[0074] Figure 2 is a Mask image containing hub pixel coordinates in a wind turbine inspection method based on efficient matrix photography by a drone according to an embodiment of the present invention;

[0075] Figure 3 is a schematic diagram of a wind turbine blade angle in a wind turbine inspection method based on efficient matrix photography by a drone according to an embodiment of the present invention;

[0076] Figure 4 It is a schematic diagram of the pan / tilt angle, yaw angle and shooting distance in the wind turbine inspection method based on efficient matrix shooting of a drone according to an embodiment of the present invention;

[0077] Figure 5 is a schematic diagram of a single-shot blade range of a current waypoint in a wind turbine inspection method based on efficient matrix photography by a drone according to an embodiment of the present invention;

[0078] Figure 6 is a schematic diagram of an inspection route in a wind turbine inspection method based on efficient matrix photography by a drone according to an embodiment of the present invention;

[0079] Figure 7 is a schematic diagram of the image coordinates of the endpoints of the wind turbine blade area currently photographed in the wind turbine inspection method based on the efficient matrix photography of the drone according to an embodiment of the present invention;

[0080] Figure 8 is a schematic diagram of a field of view imaging body in a wind turbine inspection method based on efficient matrix photography by a drone according to an embodiment of the present invention;

[0081] Fig. 9 It is one of the schematic diagrams of the field of view imaging volume parameters in the wind turbine inspection method based on efficient matrix photography of a drone according to an embodiment of the present invention;

[0082] Fig.10 It is one of the schematic diagrams of converting the camera field of view volume coordinates of a target point into pixel coordinates in a camera image in a wind turbine inspection method based on efficient matrix shooting of a drone according to an embodiment of the present invention;

[0083] Fig.11 This is a second schematic diagram of converting the camera field of view volume coordinates of a target point into pixel coordinates in a camera image in a wind turbine inspection method based on efficient matrix photography of a drone according to an embodiment of the present invention;

[0084] Fig.12 is a schematic diagram of segmentation in a wind turbine inspection method based on efficient matrix photography by a drone according to an embodiment of the present invention;

[0085] Fig.13 It is one of the schematic diagrams of searching for blade area endpoints in a Mask image in a wind turbine inspection method based on efficient matrix photography of a drone according to an embodiment of the present invention;

[0086] Fig.14 This is a second schematic diagram of searching for blade area endpoints in a Mask image in a wind turbine inspection method based on efficient matrix photography by a drone according to an embodiment of the present invention;

[0087] Fig.15 is a schematic diagram of the final pixel coordinates of the blade area in the wind turbine inspection method based on efficient matrix photography by a drone according to an embodiment of the present invention;

[0088] Fig.16 is a schematic diagram of a high-efficiency matrix array in a transverse blade in a wind turbine inspection method based on high-efficiency matrix photography by a drone according to an embodiment of the present invention;

[0089] Fig.17 is a schematic diagram of a high-efficiency matrix array in a longitudinal blade in a wind turbine inspection method based on high-efficiency matrix photography by a drone according to an embodiment of the present invention;

[0090] Fig.18 This is a second schematic diagram of field of view imaging volume parameters in a wind turbine inspection method based on efficient matrix photography by a drone according to an embodiment of the present invention;

[0091] Fig.19 This is a third schematic diagram of field of view imaging volume parameters in a wind turbine inspection method based on efficient matrix photography by a drone according to an embodiment of the present invention;

[0092] Fig. 20 This is the fourth schematic diagram of the field of view imaging volume parameters in the wind turbine inspection method based on high-efficiency matrix photography by a drone according to an embodiment of the present invention. DETAILED DESCRIPTION

[0093] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments. They can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementation methods and advantages of the present invention.

[0094] According to an embodiment of the present invention, a wind turbine inspection method based on efficient matrix photography by a drone is provided.

[0095] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to the wind turbine inspection method based on efficient matrix photography of a drone according to an embodiment of the present invention, the wind turbine inspection method includes:

[0096] S1. Collect wind turbine data and calculate wind turbine posture parameters based on the wind turbine data, and the wind turbine posture parameters include wind turbine hub center positioning coordinates, wind turbine blade length, wind turbine yaw angle, wind turbine blade angle, wind turbine pitch angle and wind turbine cone angle.

[0097] Among them, collecting fan data and calculating fan posture parameters based on the fan data include:

[0098] Aim the center of the camera image of the camera device carried by the drone at the center point of the wind turbine hub, and use the laser radar carried by the drone to collect wind turbine data;

[0099] The wind turbine data includes the distance between the drone and the center point of the wind turbine hub, the current yaw angle and pitch angle of the camera gimbal, and the current positioning coordinates of the drone;

[0100] Calculate the center coordinates of the wind turbine hub based on the wind turbine data, and use the current yaw angle of the camera gimbal as the wind turbine yaw angle;

[0101] The camera imaging image is input into the deep learning segmentation network model, and the deep learning segmentation network model is used to generate a mask image and fan hub pixel coordinates, and the fan blade angle is calculated in combination with the mask image and fan hub pixel coordinates.

[0102] The camera imaging picture is input into the deep learning segmentation network model, the deep learning segmentation network model is used to generate the mask image and the fan hub pixel coordinates, and the fan blade angle is calculated by combining the mask image and the fan hub pixel coordinates, including:

[0103] The camera image is input into the deep learning segmentation network model, and the deep learning segmentation network model is used to infer the mask image and the pixel coordinates of the wind turbine hub;

[0104] The Hough line algorithm is used to extract the lines represented by the coordinates of the two endpoints from the mask image, and the angle of each line is calculated;

[0105] Filter the straight lines that meet the preset conditions, and cluster the remaining straight lines based on the angle of each straight line. After clustering, the average angle in each category is used as the blade angle;

[0106] An angle closest to the tower straight line is selected from the blade angles as a first wind turbine blade angle closest to the tower, and the wind turbine blade angle is calculated based on the first wind turbine blade angle.

[0107] In order to facilitate understanding of the above technical solution of the present invention, the following is a detailed description of the collection of fan data in the actual process of the present invention and the calculation of fan posture parameters based on the fan data:

[0108] Step 1: When the drone takes off, align the center of the camera screen (camera imaging screen) with the center point of the wind turbine hub, and use the laser radar carried by the drone to obtain the current distance between the drone and the center of the wind turbine hub, the current yaw angle and pitch angle of the camera gimbal, and the current GPS coordinates of the drone.

[0109] Step 2: Calculate the wind turbine hub center GPS (positioning) coordinates HubGpsPosition:

[0110]

[0111]

[0112] HubGpsPositionAlt=DroneGpsPositionAlt+(DroneToHubDistance×SIN(GimbalPitchAngle));

[0113] Wherein, HubGpsPosition represents the GPS coordinate of the center of the wind turbine hub, HubGpsPositionLat, HubGpsPositionLon and HubGpsPositionAlt represent the latitude, longitude and altitude respectively; EARTH_RADIUS represents the earth radius constant; PI represents the pi; DroneToHubDistance represents the distance between the current drone and the center of the wind turbine hub; GimbalYawAngle represents the current yaw angle of the camera gimbal; GimbalPitchAngle represents the current pitch angle of the camera gimbal; DroneGpsPosition represents the current GPS coordinate of the drone.

[0114] Step 3: Calculate the wind turbine yaw angle WindPowerYawAngle. In step 1, the camera is facing the wind turbine, so the wind turbine yaw angle is the opposite of the gimbal yaw angle:

[0115] WindPowerYawAngle=GimbalYawAngle+180;

[0116] Where GimbalYawAngle represents the current yaw angle of the camera gimbal; WindPowerYawAndle represents the yaw angle of the wind turbine.

[0117] Step 4. Input the camera imaging picture in step 1 into the detection model (the detection model is the open source YOLO model, which can be implemented using YOLO, SSD and other detection models in deep learning), UNet-CS segmentation network model (UNet-CS segmentation network model is a deep learning image segmentation network model improved based on the classic UNet architecture) to infer the Mask map (the Mask map is used to mark or operate the target area, that is, the mask map) and the hub pixel coordinates (X1, Y1), (X2, Y2).

[0118] like Figure 2 As shown, firstly, according to the pixel coordinates of the upper left (X1, Y1) and lower right (X2, Y2) of the hub area obtained by the detection model, the coordinates of the center point of the hub area Centre are calculated. The calculation formula includes:

[0119] Centre X =(X1+X2) / 2;

[0120] Centre Y =(Y1+Y2) / 2;

[0121] In the formula, Centre(Centre X , Centre Y ) represents the coordinates of the center point of the fan hub area; (X1, Y1) represents the coordinates of the upper left pixel of the fan hub area; (X2, Y2) represents the coordinates of the lower right pixel of the fan hub area.

[0122] In the Mask image, the Hough line algorithm is used to extract the straight lines in the whole image. Each straight line consists of two endpoint coordinates (Line StartX , Line StartY )(Line EndX , Line EndY ) indicates that the angle of each straight line is first calculated using the following calculation methods:

[0123]

[0124] LineAngle=ARCTAN(Line K );

[0125] In the formula, (Line StartX , Line StartY )(Line EndX , Line EndY ) represents the coordinates of the two endpoints of each straight line; Line K Indicates the angle of each straight line; LineAngle indicates the blade angle; ARCTAN (Line K ) represents the inverse tangent function.

[0126] like Figure 3 As shown, Figure 3 Where FirstLineAngle indicates the first blade angle; BladeAngle indicates the fan blade angle; VerticalLine indicates the tower straight line; FirstBladeLine indicates the first blade straight line; Centre Y Y-axis coordinate of the center point of the fan hub area; (IMAGEWIDTH, IMAGEHEIGHT) represents the camera image resolution; the Y-axis coordinate of the filter endpoint is less than or equal to the Y-axis coordinate of the center point of the fan hub area Centre Y and LineAngle∈{-100,-80}, because these lines are approximately vertical and are all below the center of the hub, which are basically the edge lines of the tower; the remaining lines are clustered according to LineAngle, and the clustering result is 2 or 3 categories, because there may be a blade that overlaps with the tower, and then the average angles in the two categories are calculated as the angles of the two blades, and then the angle closest to -90 (tower straight line VerticalLine) is selected from these two blade angles as the first blade angle FirstLineAngle closest to the tower, but this angle is the angle between the first blade straight line and the X-axis, and the angle with the tower (wind turbine blade angle) BladeAngle=FirstLineAngle+90.

[0127] S2. Establish an ENU northeast geocentric coordinate system with the center of the wind turbine hub as the origin, calculate the wind turbine blade area of ​​a single shot at different positions based on the wind turbine posture parameters, and generate the inspection route of the current wind turbine blade based on the wind turbine blade area.

[0128] Among them, the wind turbine blade area of ​​a single shot at different positions is calculated in combination with the wind turbine posture parameters, and the inspection route of the current wind turbine blade is generated according to the wind turbine blade area, including:

[0129] The fan blade direction vector is calculated based on the fan blade angle, fan pitch angle and fan cone angle.

[0130] The calculation of the fan blade direction vector based on the fan blade angle, fan elevation angle and fan cone angle includes:

[0131] Calculate the fan elevation matrix; calculate the fan blade direction vector based on the fan blade angle, fan elevation angle, fan cone angle and fan elevation matrix;

[0132] Calculate the pan / tilt angle, pan / tilt yaw angle and shooting distance required for the camera to aim at the wind blades at the inspection route shooting point of the wind blades;

[0133] Define the straight line equations of the two field of view lines according to the predefined camera device parameters, and solve the straight line equations to obtain the wind turbine blade area captured in a single shot;

[0134] According to the wind turbine blade area captured in a single shot, the number of inspection route waypoints for a single wind turbine blade is determined, and based on the custom route starting point and wind turbine blade direction vector, the coordinates of each shooting waypoint in the inspection heading are determined;

[0135] The number of inspection route waypoints of a single wind turbine blade, the coordinates of each shooting waypoint in the inspection heading and the yaw angle of the wind turbine are combined to generate the inspection route of the current wind turbine blade.

[0136] In order to facilitate understanding of the above technical solution of the present invention, the following is a detailed description of the establishment of the ENU northeast geocentric coordinate system with the center of the wind turbine hub as the origin in the actual process of the present invention, the calculation of the wind turbine blade area shot at different positions in a single shot in combination with the wind turbine posture parameters, and the generation of the inspection route of the current wind turbine blade according to the wind turbine blade area:

[0137] Step 1: Establish the CSYS coordinate system (ENU northeast geocentric coordinate system) with the center of the wind turbine hub as the origin.

[0138] Step 2: According to the fan blade angle BladeAngle and based on the fan posture parameters (fan pitch angle WindPowerPitchAngle, fan cone angle WindPowerConeAngle), calculate the blade vector BladeUnitVector, where BladeId is different fan blades, a conventional fan has three blades [1, 2, 3], and WindPowerPitchMatrix is ​​the fan pitch angle matrix:

[0139]

[0140] element1=COS(BladeAngleBladeId)×COS(WindPowerConeAngle);

[0141] element2=SIN(WindPowerConeAngle);

[0142] element3=SIN(BladeAngleBladeId)×COS(WindPowerConeAngle);

[0143] BladeUnitVector BladeId =WindPowerPitchMatrix*[element1element2element3];

[0144] Where WindPowerPitchMatrix represents the wind turbine pitch matrix; WindPowerPitchAngle represents the wind turbine pitch angle; BladeUnitVector BladeId Indicates the direction vector of different fan blades; WindPowerConeAngle indicates the fan cone angle; BladeAngle BladeId It represents the fan blade angle of different fan blades. In addition, it should be noted that element has no actual meaning, and only three matrix elements are named separately through element1, element2 and element3.

[0145] Step 3: Calculate the pan / tilt angle, yaw angle, and shooting distance required for the camera to face the wind turbine blades at the inspection route shooting point. It should be noted that when solving the pan / tilt angle required to face the blades, only the pitch angle needs to be considered, because the direction of the inspection route is parallel to the direction of the blades, and the yaw angle is always constant. Figure 4 shown.

[0146] Define the BladeLine equation and solve for BladeLine k , BladeLine b :

[0147] BladeLine k =TAN(BladeAngle BladeId );

[0148] HubPosition z =BladeLine k ×HubPosition x +BladeLine b ;

[0149] Define the equation of the BladeVerticalLine line and solve it k , BladeVerticalLine b :

[0150] BladeVerticalLine k =TAN(BladeAngle BladeId +90);

[0151] CapturePosition z =BladeVerticalLine k ×CapturePosition x+BladeVerticalLine b

[0152] Define the CameraHorizontalLine line equation and solve it b :

[0153] CapturePosition z =CameraHorizontalLine b ;

[0154] Find the intersection point Node1 of BladeLine and BladeVerticalLine x Node1 x Substitute into the BladeLine equation to get Node1 z , and solve the intersection point Node2 in the same way:

[0155]

[0156] Calculate CameraHorizontalLine, PitchRange distance:

[0157]

[0158]

[0159] According to the triangle theorem, calculate NeedPitchAngle:

[0160] NeedPitchAngle=ARCSIN(PitchRange / (CameraHorizontalLine / SIN(90)));

[0161] Calculate NeedYawAngle:

[0162]

[0163] The shooting distance CaptureDistance is the distance between CapturePosition and Node1:

[0164]

[0165] Where, HubPosition(0,0,0) represents the hub center point, which is also the origin of the CSYS coordinate system; CapturePosition(x,y,z) represents the coordinates of the captured waypoint; BladeLine represents the blade straight line; BladeLine kIndicates the slope of the blade straight line equation; BladeLine b Represents the intercept of the blade straight line equation; BladeVerticalLine represents the straight line perpendicular to the blade; CameraHorizontalLine represents the capture straight line when the camera is horizontal; NeedPitchAngle represents the pitch angle required by the gimbal; NeedYawAngle represents the yaw angle required by the gimbal; CaptureDistance represents the shooting distance; PitchRange represents the actual shooting range corresponding to the pitch angle adjusted by NeedPitchAngle; WindPowerYawAngle represents the yaw angle of the wind turbine.

[0166] Step 4: Calculate the captured leaf range CaptureRange:

[0167] like Figure 5 As shown, according to the camera parameter FOV, two FovEdge field of view lines are defined (FovEdge k ,FovEdge b ) and solve for FovEdge k ,FovEdge b :

[0168] FovEdge1 k =TAN(2 / FOV+NeedPitchAngle+90);

[0169] CapturePosition z =FovEdge1 k ×CapturePosition x +FovEdge1 b ;

[0170] FovEdge2 k =TAN(-2 / FOV+NeedPitchAngle+90);

[0171] CapturePosition z =FovEdge2 k ×CapturePosition x +FovEdge2 b ;

[0172] Calculate the intersection point Node of the two FovEdge field lines and BladeLine:

[0173]

[0174] Calculate the distance between Node1 and Node2, that is, the single-shot leaf range of the current waypoint in CaptureRange:

[0175]

[0176] Where, FovEdge k ,FovEdge b They are two FovEdge field lines respectively; Node1 and Node2 are two FovEdge field lines respectively k 、FovEdge b The intersection with the blade straight line BladeLine; CaptureRange represents the single shooting blade range of the current waypoint; CapturePosition(x, y, z) represents the shooting waypoint coordinates.

[0177] Step 5: Generate inspection routes:

[0178] like Figure 6 As shown, according to the length of the wind turbine blade BladeLength, the number of inspection route waypoints WayPointNum of a single blade is determined by a single shooting of the blade area CaptureRange; based on the custom route starting point RouteStartPosition and the blade direction vector BladeUnitVector, the coordinates WayPointPosition of each shooting waypoint in the heading are determined WayPointId , and add the wind turbine yaw angle WindPowerYawAngle to generate the inspection route Route, where CEIL is rounded up:

[0179]

[0180] WayPointNum=CEIL(CaptureRange / BladeLength);

[0181] WayPointId={0, 1, 2...WayPointNum-1};

[0182] WayPointPosition WayPointId =WayPowerYawMatrix×(RouteStartPosition+WayPointId×CaptureRange×BladeUnitVector);

[0183] Route = {WayPointPosition 0 , WayPointPosition 1, …WayPointPosition WayPointNum-1};

[0184] Wherein, WayPowerYawMatrix represents the current yaw angle rotation matrix of the wind turbine; BladeLength represents the length of the wind turbine blade; CaptureRange represents the blade area captured in a single shot; WayPointNum represents the number of inspection route waypoints for a single blade; RouteStartPosition represents the starting point of a custom route; BladeUnitVector represents the blade direction vector; WayPointPositionWayPointId represents the coordinates of each captured waypoint in the heading; WindPowerYawAngle represents the yaw angle of the wind turbine; Route represents the inspection route; WayPointId represents the waypoint subscript.

[0185] S3. At the shooting waypoint of the inspection route, the ENU coordinates of the wind blade endpoint to be shot are calculated, and the ENU coordinates of the wind blade endpoint are converted into coordinates of the wind blade endpoint image coordinate system.

[0186] Among them, at the shooting waypoint of the inspection route, calculating the ENU coordinates of the wind turbine blade endpoint to be photographed, and converting the ENU coordinates of the wind turbine blade endpoint into the coordinates of the wind turbine blade endpoint image coordinate system includes:

[0187] At the shooting waypoint of the inspection route, calculate the ENU coordinates of the endpoint of the wind turbine blade area that needs to be photographed;

[0188] Establish a field of view imaging volume according to the camera equipment parameters and the shooting target point, and convert the ENU coordinates of the end points of the wind turbine blade area into the camera imaging volume coordinates based on the field of view imaging volume;

[0189] Combining the cosine theorem and the sine theorem, the camera imaging volume coordinates are converted to the pixel coordinates in the camera image to obtain the pixel coordinates of the end points of the wind turbine blade area;

[0190] The camera imaging picture is segmented using a deep learning segmentation network model to obtain a segmentation mask map, and the image coordinate system coordinates of the wind turbine blade endpoints are extracted from the segmentation mask map.

[0191] Among them, the camera imaging picture is segmented by using the deep learning segmentation network model to obtain the segmentation mask map. The coordinates of the wind turbine blade endpoint image coordinate system are extracted from the segmentation mask map, including:

[0192] The camera image is input into the deep learning segmentation network model to infer the segmentation mask map, the connected areas in the segmentation mask map are calculated, the connected areas exceeding the preset threshold are retained, and the empty pixels in the connected areas are filled;

[0193] The fan blade trajectory slope is calculated based on the pixel coordinates of the end points of the fan blade region, and the fan blade trajectory angle is calculated based on the fan blade trajectory slope;

[0194] The current direction of the fan blade trajectory is determined based on the fan blade trajectory slope, and the image coordinate system coordinates of the fan blade endpoints are searched in the segmentation mask image according to the fan blade trajectory direction.

[0195] Wherein, judging the current direction of the fan blade trajectory based on the fan blade trajectory slope, and searching the image coordinate system coordinates of the fan blade endpoint in the segmentation mask image according to the fan blade trajectory direction include:

[0196] The current direction of the fan blade trajectory is determined based on the slope of the fan blade trajectory. If the fan blade is longitudinal, a fixed sliding window is sequentially slid in the opposite direction in the up and down directions of the Y axis of the segmentation mask until the sliding window contains the fan blade area pixels.

[0197] If the fan blades are horizontal, then the split mask Figure X The left and right directions of the axis are slid in the opposite direction in sequence with a fixed sliding window until the sliding window contains the pixels of the fan blade area; the coordinates of the particles in the fan blade area are extracted as the coordinates of the fan blade endpoint image coordinate system.

[0198] In order to facilitate understanding of the above technical solution of the present invention, the following is a detailed description of the present invention in the actual process of calculating the ENU coordinates of the wind blade endpoints to be photographed at the shooting waypoints of the inspection route, and converting the ENU coordinates of the wind blade endpoints into the coordinates of the wind blade endpoint image coordinate system:

[0199] Step 1: Figure 7 As shown, the endpoints TargetStartPosition and TargetEndPosition of the captured leaf area are calculated:

[0200] TargetStartPosition WayPointId =HubPosition+WayPointId×BladeUnitVector;

[0201] TargetEndPosition WayPointId =HubPosition+(WayPointId+1)×BladeUnitVector;

[0202] Wherein, BladeUnitVector represents the blade direction vector; HubPosition represents the hub center point.

[0203] Step 2: Figure 8As shown in the figure, the field of view imaging volume FovSpace is established according to the camera parameters and the target point being photographed:

[0204] According to the gimbal yaw angle NeedYawAngle and pitch angle NeedPitchAngle, the camera line of sight direction vector FovLineUnitVector (actually the normal vector of FovBottomPlane) is calculated, and the plane equation FovBottomPlane of the bottom surface of the field of view imaging volume is solved based on the coordinate origin, which is the center point of the fan hub HubPosition (x, y, z) and the line of sight direction vector:

[0205]

[0206] -FovBottomPlane d =FovLineUnitVector x ×HubPosition x +FovLineUnitVector y ×HubPosition y +FovLineUnitVector z ×HubPosition z ;

[0207] Secondly, calculate the vertical distance TargetDistance between the endpoints of the photographed leaf area (TargetStartPosition, TargetEndPosition) and the bottom plane FovBottomPlane of the field of view imaging body, and take the longest distance as MaxTargetDistance as the high FovSpace of the field of view imaging body. z , and based on the vertical field of view CameraFov of the camera parameters vertical and horizontal field of view CameraFov horizontal , calculate the length of the field of view imaging volume FovSpace y With Wide FovSpace x :

[0208] molecule=ABS(FovLineUnitVector x ×TargetStartPosition x +FovLineUnitVector y ×TargetStartPosition y +FovLineUnitVector z ×TargetStartPositionz +FovBottomPlane d );

[0209]

[0210] Similarly, solve TargetDistance TargetEndPosition , and calculate MaxTargetDistance as the high FovSpace of the field of view imaging volume z and the width of the field of view imaging volume FovSpace x and the length of the field of view imaging volume FovSpace y :

[0211] FovSpace z =MAX(TargetDistance TargetStartPosition ,

[0212] TargetDistance TargetEndPosition );

[0213] FovSpace x =2×FovSpace z ×TAN(CameraFov horizontal );

[0214] FovSpace y =2×FovSpace z ×TAN(CameraFov vertical );

[0215] like Fig. 9 and Figure 18-Figure 20 As shown, finally, according to the current waypoint position WayPointPosition and the pitch angle NeedPitchAngle and yaw angle NeedYawAngle, the upper left FovTopPlaneLT and upper right coordinates FovTopPlaneRT of the top surface of the field of view imaging volume FovTopPlane of TargetStartPosition and TargetEndPosition are determined. Here, FovSpace is set z TargetDistance TargetStartPosition , that is, TargetDistance TargetStartPosition >TargetDistance TargetEndPosition :

[0216]

[0217]

[0218]

[0219]

[0220]

[0221] Step 3: Figure 10-11 As shown, the camera field of view coordinates of the target point are converted into pixel coordinates in the camera image.

[0222] The relative horizontal and vertical coordinates of TargetStartPosition and TargetEndPosition in FovTopPlane are calculated by the cosine theorem and the sine theorem respectively, and normalized and mapped to the camera image coordinates, so that the pixel coordinates PixerPosition of the target point in the camera image are obtained (the imaging distortion of the camera itself is not considered here). TargetStartPosition is listed below, and TargetEndPosition can be obtained similarly, where IMAGEWIDTH and IMAGEWIDTH are the camera image resolutions respectively.

[0223]

[0224]

[0225]

[0226]

[0227] TargetStartPosition FovX =ToLTDistance TargetStartPosition / SIN(90)×SIN(90-A);

[0228] TargetStartPosition FovY =ToLTDistance TargetStartPositon / SIN(90)×SIN(180-90-(90-A));

[0229] SMPixerPosition x =TargetStartPosition FovX / FovSpace x ×IMAGEWIDTH;

[0230] SMPixerPosition y =TargetStartPosition FovY / FovSpace y ×IMAGEHEIGHT;

[0231] Step 4: After obtaining the preliminary blade area coordinates and blade trajectory through spatial mapping, use the UNet-CS network model to segment the camera captured image, and extract the image coordinates of the wind turbine blade endpoints in the segmentation result Mask image.

[0232] Input the camera screen image into the UNet-CS model (the UNet-CS network here is optimized and improved based on the deep learning open source model UNet. The main purpose is to add spatial and channel attention mechanisms in the downsampling process of the model to further improve the segmentation effect of the model. The network structure, training process and reasoning process are not introduced too much here. The leaf extraction process of the mask image of the segmentation result of the model is mainly explained in detail). The mask image of the current picture is inferred, such as Fig.12 As shown:

[0233] After obtaining the leaf mask map, first calculate the connected area in the mask map, retain the connected domain that exceeds the threshold, and fill the gap pixels in the connected domain; secondly, calculate the slope of the leaf trajectory according to the calculated pixel coordinates of the leaf area endpoints, and determine the blade trajectory angle BladeLineAngle SM , where SM represents the coordinates obtained by spatial mapping:

[0234]

[0235] BladeLineAngle SM =ARCTAN(BladeLineK SM );

[0236] like Figure 13-14 As shown, the direction of the current blade trajectory is determined according to the slope, and then based on the direction, the blade area endpoint is searched in the Mask map, wherein the search method includes:

[0237] If the leaf is vertical, the fixed sliding window WINDOWSIZE is slid in the opposite direction in the up and down directions of the image Y axis at the same time until the sliding window contains the leaf area pixels; if it is horizontal, the fixed sliding window WINDOWSIZE is slid in the opposite direction in the left and right directions of the image X axis at the same time until the sliding window contains the leaf area pixels; and the particle coordinates of the leaf area are extracted as the pixel coordinates CVPixerPosition of the leaf endpoint, so that multiple sets of pixel coordinates of the leaf endpoints are obtained (there may be multiple connected domains), where CV represents the coordinates obtained by visual processing:

[0238]

[0239] S4. Combining the coordinates of the fan blade region image coordinate system and the coordinates of the fan blade endpoint image coordinate system, the final fan blade endpoint image coordinates are generated.

[0240] In order to facilitate understanding of the above technical solution of the present invention, the following is a detailed description of the present invention in the actual process of combining the coordinates of the wind blade area image coordinate system and the coordinates of the wind blade end point image coordinate system to generate the final wind blade end point image coordinates:

[0241] like Fig.15 As shown in the figure, after the spatial mapping calculation and visual positioning, SMPixerPosition and multiple sets of CVPixerPosition coordinates are obtained. Now we need to use the SMPixerPosition coordinates to verify the CVPixerPosition coordinates (the actual meaning here is that in the same picture, we now have a set of SMPixerPosition and multiple sets of CVPixerPosition leaf area pixel coordinates, which need to be traded off; in the actual scene, CVPixerPosition is used first, because in all calculations of SMPixerPosition, the trajectory of the leaf is always straight Line, but the real blade is not a pure straight line, but will be affected by pre-bending, gravity and wind speed and other factors to become a curve. This blade curve cannot be fitted by a function, so the trajectory of the SMPixerPosition coordinate is more accurate. The closer the specific coordinate value is to the blade tip, the greater the deviation, so the visual coordinate is used first. However, because multiple wind turbine areas may appear in the same picture, and the model cannot distinguish the currently photographed blade area in detail, it is necessary to filter multiple groups of CVPixerPosition through SMPixerPosition and retain a group of CVPixerPosition coordinates closest to SMPixerPosition), the method is as follows:

[0242] Loop through the calculations of the rectangular union UnionArea and intersection IntersectionArea of ​​each set of CVPixerPosition and SMPixerPosition, and calculate the proportion of the intersection in the union. Keep the set of CVPixerPosition with the largest proportion. If the proportion of the CVPixerPosition is greater than or equal to the set threshold, use it as the final leaf area pixel coordinate PixerPosition. Otherwise, use SMPixerPosition as the final leaf area pixel coordinate PixerPosition.

[0243] S5. Based on the final image coordinates of the wind turbine blade endpoints, a matrix shooting array corresponding to the required shooting area is calculated, and the pan / tilt head is mobilized to perform zoom shooting on the matrix shooting array in sequence.

[0244] Wherein, based on the final wind turbine blade endpoint image coordinates, calculating the matrix shooting array corresponding to the required shooting area includes:

[0245] Based on the final image coordinates of the fan blade endpoints, the fan blade trajectory straight line equation is established;

[0246] Calculate the focal length corresponding to the current shooting zoom ratio of the camera device, calculate the field of view of each matrix shooting array grid corresponding to the current shooting zoom ratio based on the focal length, and calculate the pixel range of the matrix shooting array grid on the camera imaging screen according to the field of view.

[0247] The direction of the fan blade is determined based on the final image coordinates of the fan blade endpoints, and a matrix shooting array grid is inserted on the fan blade trajectory based on the direction of the fan blade.

[0248] In order to facilitate understanding of the above technical solution of the present invention, the following is a detailed description of the present invention in the actual process, based on the final wind turbine blade endpoint image coordinates, calculating the matrix shooting array corresponding to the required shooting area, and mobilizing the pan / tilt to sequentially zoom the matrix shooting array:

[0249] Step 1: According to the blade endpoint image coordinates StarPixerPosition and EndPixerPosition, establish the blade trajectory straight line equation BladeAreaLine:

[0250]

[0251] BladeAreaLine b =StartPixerPosition y -(StartPixerPosition x ×BladeAreaLine k );

[0252] Step 2: Calculate the field of view ZoomFov captured by the current shooting zoom ratio ZOOMRATIO and the corresponding image pixel range. First, calculate the focal length FLencht corresponding to ZOOMRATIO, then calculate the GFov corresponding to the current zoom ratio (that is, the field of view of each efficient array grid) through the focal length, and finally calculate the pixel range GPixer of each efficient matrix grid on the image; where MINFLENGHT is the minimum focal length of the camera, CMOS is the camera cmos size, FOV is the maximum field of view of the camera, DPI is the image resolution of the camera, H / V is horizontal, vertical:

[0253] The calculation formula for the focal length corresponding to the current camera device's shooting zoom ratio is:

[0254] FLenght=MINFLENGHT×ZOOMRATIO;

[0255] The calculation formula for the field of view of each matrix shooting array grid corresponding to the current shooting zoom ratio is:

[0256]

[0257] The calculation formula for the pixel range of the matrix shooting array grid on the camera imaging screen is:

[0258]

[0259] Wherein, FLencht represents the focal length corresponding to the current camera device's shooting zoom ratio; MINFLENGHT represents the minimum focal length of the camera device; ZOOMRATIO represents the current camera device's shooting zoom ratio; GFov represents the field of view of each matrix shooting array grid corresponding to the current shooting zoom ratio; CMOS represents the camera device CMOS size; ARCTAN represents the inverse tangent function; H / V represents the horizontal direction; GPixer represents the pixel range of the matrix shooting array grid on the camera imaging screen; FOV represents the maximum field of view of the camera device; and DPI represents the image resolution of the camera device.

[0260] Step 3: Determine the direction of the blade according to StartPixerPosition and EndPixerPosition. The principle is the same as the above search method. When the blade direction is horizontal, use GPixer H Insert efficient matrix grids as intervals on the blade trajectory (GPixer HSubstitute the x value into the BladeAreaLine equation to get the y value, thus determining the pixel coordinates of each efficient matrix grid); when the blade direction is longitudinal, the solution is similar; the schematic diagram of the efficient matrix array in the transverse and longitudinal blades is as follows Figure 16-Figure 17 shown.

[0261] In summary, with the aid of the above-mentioned technical scheme of the present invention, the wind turbine inspection method based on high-efficiency matrix shooting of drones provided by the present invention effectively solves the problem of unclear photos of wind turbine blades during the inspection process, and realizes high-definition and high-zoom shooting of wind turbines. At the same time, the method determines the complete inspection route flight trajectory and precise shooting actions in the route generation stage, and avoids the need for real-time decision-making during the inspection process, thereby significantly improving the inspection efficiency while improving the shooting accuracy.

[0262] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A wind turbine inspection method based on efficient matrix photography by unmanned aerial vehicles, characterized in that: The fan inspection method includes: Collect wind turbine data and calculate wind turbine posture parameters based on the wind turbine data, and the wind turbine posture parameters include wind turbine hub center positioning coordinates, wind turbine blade length, wind turbine yaw angle, wind turbine blade angle, wind turbine pitch angle and wind turbine cone angle; Establish an ENU northeast geocentric coordinate system with the center of the wind turbine hub as the origin, calculate the wind turbine blade area of ​​a single shot at different positions based on the wind turbine posture parameters, and generate the inspection route of the current wind turbine blade based on the wind turbine blade area; At the shooting waypoint of the inspection route, the ENU coordinates of the wind turbine blade endpoint to be photographed are calculated, and the ENU coordinates of the wind turbine blade endpoint are converted into the coordinates of the wind turbine blade endpoint image coordinate system; The coordinates of the fan blade region image coordinate system and the coordinates of the fan blade endpoint image coordinate system are combined to generate the final fan blade endpoint image coordinates; Based on the final image coordinates of the wind turbine blade endpoints, the matrix shooting array corresponding to the required shooting area is calculated, and the pan / tilt is mobilized to zoom and shoot the matrix shooting array in sequence.

2. The wind turbine inspection method based on efficient matrix photography of unmanned aerial vehicles according to claim 1 is characterized in that: The collecting of fan data and calculating the fan posture parameters based on the fan data includes: Aim the center of the camera image of the camera device carried by the drone at the center point of the wind turbine hub, and use the laser radar carried by the drone to collect wind turbine data; The wind turbine data includes the distance between the UAV and the center point of the wind turbine hub, the current yaw angle and pitch angle of the camera gimbal, and the current positioning coordinates of the UAV; Calculate the center coordinates of the wind turbine hub based on the wind turbine data, and use the current yaw angle of the camera gimbal as the wind turbine yaw angle; The camera imaging image is input into the deep learning segmentation network model, and the deep learning segmentation network model is used to generate a mask image and fan hub pixel coordinates, and the fan blade angle is calculated in combination with the mask image and fan hub pixel coordinates.

3. The wind turbine inspection method based on efficient matrix photography of unmanned aerial vehicles according to claim 2 is characterized in that: The step of inputting the camera imaging image into the deep learning segmentation network model, generating a mask image and fan hub pixel coordinates using the deep learning segmentation network model, and calculating the fan blade angle in combination with the mask image and the fan hub pixel coordinates includes: The camera image is input into the deep learning segmentation network model, and the deep learning segmentation network model is used to infer the mask image and the pixel coordinates of the wind turbine hub; The coordinates of the center point of the fan hub area are calculated based on the pixel coordinates of the fan hub. The calculation formula of the coordinates of the center point of the hub area is: Center X =(X1+X2) / 2; Centre Y =(Y1+Y2) / 2; In the formula, Centre(Centre X , Centre Y ) represents the coordinates of the center point of the fan hub area; (X1, Y1) represents the coordinates of the upper left pixel of the fan hub area; (X2, Y2) represents the coordinates of the lower right pixel of the fan hub area; The Hough line algorithm is used to extract the lines represented by the coordinates of the two endpoints from the mask image, and the angle of each line is calculated; Filter the straight lines that meet the preset conditions, and cluster the remaining straight lines based on the angle of each straight line. After clustering, the average angle in each category is used as the blade angle; An angle closest to the tower straight line is selected from the blade angles as a first wind turbine blade angle closest to the tower, and the wind turbine blade angle is calculated based on the first wind turbine blade angle.

4. The wind turbine inspection method based on efficient matrix photography of unmanned aerial vehicles according to claim 3 is characterized in that: The method of calculating the wind turbine blade area captured in a single shot at different positions in combination with the wind turbine posture parameters, and generating the inspection route of the current wind turbine blade according to the wind turbine blade area includes: Calculate the fan blade direction vector based on the fan blade angle, fan elevation angle and fan cone angle; Calculate the pan / tilt angle, pan / tilt yaw angle and shooting distance required for the camera to aim at the wind blades at the inspection route shooting point of the wind blades; Define the straight line equations of the two field of view lines according to the predefined camera device parameters, and solve the straight line equations to obtain the wind turbine blade area captured in a single shot; According to the wind turbine blade area captured in a single shot, the number of inspection route waypoints for a single wind turbine blade is determined, and based on the custom route starting point and wind turbine blade direction vector, the coordinates of each shooting waypoint in the inspection heading are determined; The number of inspection route waypoints of a single wind turbine blade, the coordinates of each shooting waypoint in the inspection heading and the yaw angle of the wind turbine are combined to generate the inspection route of the current wind turbine blade.

5. The wind turbine inspection method based on efficient matrix photography of unmanned aerial vehicles according to claim 4 is characterized in that: The calculation of the fan blade direction vector based on the fan blade angle, the fan elevation angle and the fan cone angle comprises: Calculate the wind turbine elevation matrix. The expression of the wind turbine elevation matrix is: The fan blade direction vector is calculated based on the fan blade angle, fan elevation angle, fan cone angle and fan elevation angle matrix. The expression of the fan blade direction vector is: element1=COS(BladeAngle BladeId )×COS(WindPowerConeAngle); element2=SIN(WindPowerConeAngle); element3=SIN(BladeAngle BladeId )×COS(WindPowerConeAngle); BladeUnitVector BladeId =WindPowerPitchMatrix* [element1 element2 element3]; Where WindPowerPitchMatrix represents the wind turbine pitch matrix; WindPowerPitchAngle represents the wind turbine pitch angle; BladeUnitVector BladeId Indicates the direction vector of different fan blades; WindPowerConeAngle indicates the fan cone angle; BladeAngle BladeId Represents the fan blade angle of different fan blades.

6. The wind turbine inspection method based on efficient matrix photography of unmanned aerial vehicles according to claim 5 is characterized in that: The step of calculating the ENU coordinates of the wind turbine blade endpoints to be photographed at the shooting waypoints of the inspection route and converting the ENU coordinates of the wind turbine blade endpoints into the coordinates of the wind turbine blade endpoint image coordinate system includes: At the shooting waypoint of the inspection route, calculate the ENU coordinates of the endpoint of the wind turbine blade area that needs to be photographed; Establish a field of view imaging volume according to the camera equipment parameters and the shooting target point, and convert the ENU coordinates of the end points of the wind turbine blade area into the camera imaging volume coordinates based on the field of view imaging volume; Combining the cosine theorem and the sine theorem, the camera imaging volume coordinates are converted to the pixel coordinates in the camera image to obtain the pixel coordinates of the end points of the wind turbine blade area; The camera imaging picture is segmented using a deep learning segmentation network model to obtain a segmentation mask map, and the image coordinate system coordinates of the wind turbine blade endpoints are extracted from the segmentation mask map.

7. The wind turbine inspection method based on efficient matrix photography of unmanned aerial vehicles according to claim 6 is characterized in that: The camera imaging picture is segmented by using the deep learning segmentation network model to obtain a segmentation mask map, and the coordinates of the wind turbine blade endpoint image coordinate system are extracted from the segmentation mask map, including: The camera image is input into the deep learning segmentation network model to infer the segmentation mask map, the connected areas in the segmentation mask map are calculated, the connected areas exceeding the preset threshold are retained, and the empty pixels in the connected areas are filled; The fan blade trajectory slope is calculated based on the pixel coordinates of the end points of the fan blade region, and the fan blade trajectory angle is calculated based on the fan blade trajectory slope; The current direction of the fan blade trajectory is determined based on the fan blade trajectory slope, and the image coordinate system coordinates of the fan blade endpoints are searched in the segmentation mask image according to the fan blade trajectory direction.

8. The wind turbine inspection method based on efficient matrix photography of unmanned aerial vehicles according to claim 7 is characterized in that: The method of determining the current direction of the fan blade trajectory based on the fan blade trajectory slope, and searching for the fan blade endpoint image coordinates in the segmentation mask image according to the fan blade trajectory direction includes: The current direction of the fan blade trajectory is determined based on the slope of the fan blade trajectory. If the fan blade is longitudinal, a fixed sliding window is sequentially slid in the opposite direction in the up and down directions of the Y axis of the segmentation mask until the sliding window contains the fan blade area pixels. If the fan blade is horizontal, a fixed sliding window is slid in the opposite direction in the left and right directions of the segmentation mask image X-axis until the sliding window contains the fan blade area pixels; the particle coordinates of the fan blade area are extracted as the coordinates of the fan blade endpoint image coordinate system.

9. The wind turbine inspection method based on efficient matrix photography of unmanned aerial vehicles according to claim 8 is characterized in that: The step of calculating the matrix shooting array corresponding to the required shooting area based on the final wind turbine blade endpoint image coordinates includes: Based on the final image coordinates of the fan blade endpoints, the fan blade trajectory straight line equation is established; Calculate the focal length corresponding to the current shooting zoom ratio of the camera device, calculate the field of view of each matrix shooting array grid corresponding to the current shooting zoom ratio based on the focal length, and calculate the pixel range of the matrix shooting array grid on the camera imaging screen according to the field of view; The direction of the fan blade is determined based on the final image coordinates of the fan blade endpoints, and a matrix shooting array grid is inserted on the fan blade trajectory based on the direction of the fan blade.

10. The wind turbine inspection method based on efficient matrix photography of unmanned aerial vehicles according to claim 9 is characterized in that: The calculation formula of the focal length corresponding to the shooting zoom ratio of the current camera device is: FLenght=MINFLENGHT×ZOOMRATIO; The calculation formula for the field of view of each matrix shooting array grid corresponding to the current shooting zoom ratio is: The calculation formula for the pixel range of the matrix shooting array grid on the camera imaging screen is: Wherein, FLencht represents the focal length corresponding to the current camera device's shooting zoom ratio; MINFLENGHT represents the minimum focal length of the camera device; ZOOMRATIO represents the current camera device's shooting zoom ratio; GFov represents the field of view of each matrix shooting array grid corresponding to the current shooting zoom ratio; CMOS represents the camera device CMOS size; ARCTAN represents the inverse tangent function; H / V represents the horizontal direction; GPixer represents the pixel range of the matrix shooting array grid on the camera imaging screen; FOV represents the maximum field of view of the camera device; and DPI represents the image resolution of the camera device.

Images