A method and system for correcting the attitude of a UAV based on visual feedback

Abstract

The application provides a kind of unmanned aerial vehicle attitude correction method and system based on visual feedback, it is related to unmanned aerial vehicle visual servo control technical field.The application obtains the original color image of fan blade after, based on HSV grass green threshold generation grass binary mask and carry out grass gray scale suppression, obtain grass suppression gray scale chart;Edge map is generated and is subjected to probability hough transform line segment detection, according to the angle distribution of line segment determines horizontal dominant mode or vertical dominant mode, respectively determines slope, intercept or inverse slope, inverse intercept and filters line segment, obtains blade center line equation by K-means clustering;Horizontal movement component and vertical movement component are calculated again, and are converted into unmanned aerial vehicle drift angle adjustment amount and gimbal pitch angle adjustment amount.The application can reduce the interference of complex background line segment, reduce the parameter instability caused by fixed slope expression under vertical imaging attitude.

Description

Technical Field

[0001] This invention relates to the field of visual servo control technology for unmanned aerial vehicles (UAVs), and in particular to a method and system for UAV attitude correction based on visual feedback. Background Technology

[0002] Wind turbine blades are critical components in wind turbine generators, responsible for capturing wind energy and transferring loads. Operating in the open environment for extended periods, their surface structure is susceptible to alternating loads, wind and sand erosion, rain and snow impacts, lightning strikes, and ultraviolet aging, leading to defects such as leading-edge wear, surface cracks, coating peeling, and skin damage. These blade defects affect the aerodynamic performance and operational safety of the wind turbine generator; therefore, inspecting the surface condition of wind turbine blades is a crucial task in wind farm operation and maintenance. With the development of unmanned aerial vehicle (UAV) platforms, airborne visible light imaging equipment, and image processing technology, using UAVs for close-range image acquisition of wind turbine blades has become a common method in blade inspection. The UAV approaches the blade surface according to preset waypoints, and the onboard camera captures images of the blade from different shooting positions. Subsequent image processing or defect identification algorithms are then used to analyze the surface condition of the blade. This method reduces the risks of manual high-altitude operations and improves the efficiency of blade image acquisition.

[0003] During UAV inspection of wind turbine blades, the image position of the blades in the image is affected by various factors. Positioning obstructions near the wind turbine tower, airflow disturbances near the blades, changes in UAV attitude, low-frequency blade oscillations, and limitations in the onboard camera's field of view can all cause the blades to deviate from the image center. When the blade image position deviates from the image center, the acquired image is prone to problems such as incomplete target areas, insufficient local clarity, or shift in the effective detection area, thus affecting subsequent defect identification results. To ensure the quality of the inspection images, it is necessary to correct the UAV's yaw angle or gimbal pitch angle based on the images acquired by the onboard camera, so that the blade image position is closer to the image center. Existing visual feedback attitude correction methods typically extract edge features from the blade image, then determine the blade centerline based on the blade edge line, and use the positional deviation of the blade centerline relative to the image center as the basis for attitude correction.

[0004] Existing methods for extracting blade centerlines typically begin by converting, filtering, and performing edge detection on blade images acquired by an airborne camera. Then, Hough transform or probabilistic Hough transform is used to detect straight line features, and lines closely aligned with the blade's direction are selected from the detection results. Since wind turbine blades are usually elongated in images, their edges are relatively stable, making the method of determining the blade centerline based on edge lines feasible. Some methods also utilize color priors to suppress background areas such as grass, reducing interference from background texture edges on the line detection results. After obtaining the blade centerline, existing methods typically generate adjustments to the UAV's yaw angle or gimbal pitch angle based on image deviations, and then continuously acquire and adjust the image to gradually bring the blade's imaging position closer to the image center.

[0005] However, existing methods still have shortcomings in complex inspection scenarios. Wind turbine blade inspection images often contain background objects such as grasslands, forests, road boundaries, and tower structures. After edge detection, these background objects easily form a large number of non-blade line segments, interfering with blade edge line selection and centerline calculation. Existing line selection, clustering, and deviation calculation processes typically use fixed line parameter representations. When the blade is in a near-vertical orientation in the image, the slope of the blade edge line increases sharply, easily causing unstable slope calculations, invalidation of selection conditions, imbalance of cluster feature space, and abnormal pixel deviation calculations. Furthermore, proportional control or proportional-derivative control methods usually require repeated image detection and attitude adjustment, which can easily lead to response lag when blades sway, UAV attitude disturbances occur, or the camera's field of view is narrow. Therefore, visual feedback attitude correction in UAV inspection of wind turbine blades still needs improvement in background line segment suppression, line parameter representation under different blade imaging attitudes, and the efficiency of converting pixel deviation to attitude adjustment. Summary of the Invention

[0006] To address the issues of interference from non-blade line segments in complex backgrounds for blade centerline extraction, parameter instability caused by fixed slope representation in vertical imaging attitude, and insufficient efficiency in converting pixel deviation to attitude adjustment, this invention provides a UAV attitude correction method and system based on visual feedback. Through grass background grayscale suppression, dominance mode determination based on line segment angle distribution, adaptive determination of straight line parameters under horizontal and vertical dominance modes, and attitude correction conversion based on blade centerline deviation, this method reduces interference from complex background line segments for blade centerline extraction, minimizes parameter instability caused by fixed slope representation in vertical imaging attitude, and directly obtains the UAV yaw angle adjustment and gimbal pitch angle adjustment from the current image deviation.

[0007] To achieve the above objectives, the present invention provides the following solution: A method for correcting the attitude of a drone based on visual feedback, comprising: The original color image of the wind turbine blades is acquired by the airborne camera of the UAV. Based on the green threshold range of grass in the HSV color space, a grass binary mask is generated for the original color image. Then, the grass grayscale is suppressed on the grayscale original color image according to the grass binary mask to obtain the grass suppressed grayscale image. An edge map is generated based on the grass suppression grayscale image, and a probabilistic Hough transform is performed on the edge map to detect line segments, resulting in a set of line segments. The line segment angle is calculated based on the endpoint pixel coordinates of each line segment in the line segment set, and the dominant mode of the current image is determined based on the distribution of the line segment angles. The dominant mode includes the horizontal dominant mode and the vertical dominant mode. When the dominant mode is horizontal, the slope and intercept of each line segment are determined with the image abscissa as the independent variable and the image ordinate as the dependent variable, and the line segment is filtered based on the slope of the blade principal axis of the currently detected blade; when the dominant mode is vertical, the inverse slope and inverse intercept of each line segment are determined with the image ordinate as the independent variable and the image abscissa as the dependent variable, and the line segment is filtered based on the inverse slope of the blade principal axis of the currently detected blade, resulting in a set of filtered line parameters. K-means clustering was performed on the filtered set of straight line parameters to obtain two cluster centers corresponding to the two edge lines of the blade, and the blade centerline equation was determined based on the two cluster centers. Based on the image center point coordinates, the blade centerline equation, and the dominant mode, calculate the horizontal and vertical movement components of the image center point relative to the blade centerline. Based on the camera focal length information, the horizontal movement component is converted into the UAV yaw angle adjustment, and the vertical movement component is converted into the gimbal pitch angle adjustment. Attitude correction is then performed according to the UAV yaw angle adjustment and the gimbal pitch angle adjustment.

[0008] Preferably, a grass binary mask is generated from the original color image based on the grass green threshold range in the HSV color space, and grass grayscale suppression is performed on the grayscaled original color image according to the grass binary mask to obtain a grass-suppressed grayscale image, including: The original color image is converted from the BGR color space to the HSV color space to obtain an HSV image; The HSV image is thresholded based on the green threshold range of the grassland to generate a binary grassland mask. The original color image is converted to a grayscale image, and the grassland and non-grass regions are determined in the grayscale image based on the grassland binary mask. The grayscale value of the grassland area is set to the average grayscale value of the non-grassland area or the average grayscale value of the grassland, while the original grayscale value of the non-grassland area is retained, resulting in a grassland suppressed grayscale image.

[0009] Preferably, an edge map is generated based on the grassland suppression grayscale image, and probabilistic Hough transform line segment detection is performed on the edge map to obtain a set of line segments, including: The Otsu algorithm was used to adaptively binarize the grass suppression grayscale image to obtain a binary image. The Canny operator is used to perform edge detection on a binary image to obtain an edge map. Probabilistic Hough transform is performed on the edge map to detect line segments, resulting in a set of line segments. Each line segment in the set is represented by the pixel coordinates of its two endpoints.

[0010] Preferably, the line segment angles are calculated based on the endpoint pixel coordinates of each line segment in the line segment set, and the dominant mode of the current image is determined based on the distribution of the line segment angles. The dominant mode includes a horizontal dominant mode and a vertical dominant mode, including: Calculate the corresponding line segment angle based on the pixel coordinates of the two endpoints of each line segment; The distribution of line segment angles in a set of line segments; When the number of line segments whose angles fall within the preset horizontal angle range exceeds half the total number of line segments in the line segment set, the dominant mode is determined to be the horizontal dominant mode. When the number of line segments whose angles fall within the preset vertical angle range exceeds half the total number of line segments in the line segment set, the dominant mode is determined to be the vertical dominant mode.

[0011] Preferably, when the dominant mode is the horizontal dominant mode, the slope and intercept of each line segment are determined with the image's horizontal coordinate as the independent variable and the image's vertical coordinate as the dependent variable, and line segment selection is performed based on the slope of the blade's principal axis currently being detected, including: When the dominant mode is horizontal, the slope and intercept of each line segment are determined according to the pixel coordinates of the endpoints of each line segment and the correspondence between the horizontal and vertical coordinates of the image. Obtain the blade principal axis slope of the currently detected blade; Calculate the slope difference between the slope of each line segment and the slope of the blade main shaft; Line segments with slope differences less than a preset screening threshold are retained as line segments, and the slopes and intercepts of the retained line segments form the set of filtered line parameters.

[0012] Preferably, when the dominant mode is the vertical dominant mode, the inverse slope and inverse intercept of each line segment are determined with the image ordinate as the independent variable and the image abscissa as the dependent variable, and line segment selection is performed based on the blade principal axis inverse slope of the currently detected blade, including: When the dominant mode is vertical, the inverse slope and inverse intercept of each line segment are determined according to the pixel coordinates of the endpoints of each line segment and the correspondence between the image's vertical coordinate and the image's horizontal coordinate. Obtain the blade principal axis anti-slope of the currently detected blade; Calculate the difference in reverse slope between the reverse slope of each line segment and the reverse slope of the blade main shaft; Line segments with a back slope difference less than a preset screening threshold are retained as line segments, and the back slope and back intercept of the retained line segments form a set of filtered line parameters.

[0013] Preferably, K-means clustering is performed on the filtered set of straight line parameters to obtain two cluster centers corresponding to the two edge lines of the blade, and the blade centerline equation is determined based on the two cluster centers, including: Use the line parameters in the filtered set of line parameters as cluster objects, and set the number of clusters to 2; K-means clustering was performed on the clustered objects to obtain two cluster centers; The two cluster centers are respectively determined as the parameter centers corresponding to the two edge lines of the leaf; When the dominant mode is horizontal, the arithmetic mean of the slopes in the two cluster centers and the arithmetic mean of the intercepts in the two cluster centers are taken to obtain the blade centerline equation. When the dominant mode is vertical, the arithmetic mean of the inverse slopes of the two cluster centers and the arithmetic mean of the inverse intercepts of the two cluster centers are taken to obtain the blade centerline equation.

[0014] Preferably, based on the image center point coordinates, the blade centerline equation, and the dominant mode, the horizontal and vertical movement components of the image center point relative to the blade centerline are calculated, including: Determine the coordinates of the image center point based on the image width and image height; When the dominant mode is horizontal, the horizontal and vertical distances of the image center point relative to the blade centerline are determined based on the coordinates of the image center point and the slope and intercept in the blade centerline equation, and the horizontal and vertical movement components are calculated based on the horizontal and vertical distances. When the dominant mode is vertical, the horizontal and vertical distances of the image center point relative to the blade centerline are determined based on the coordinates of the image center point and the inverse slope and inverse intercept in the blade centerline equation. The horizontal and vertical movement components are then calculated based on the horizontal and vertical distances.

[0015] Preferably, the horizontal movement component is converted into a UAV yaw angle adjustment amount based on the camera focal length information, and the vertical movement component is converted into a gimbal pitch angle adjustment amount. Attitude correction is then performed according to the UAV yaw angle adjustment amount and the gimbal pitch angle adjustment amount, including: The horizontal movement component is converted into the yaw angle adjustment of the UAV based on the horizontal focal length of the camera. The vertical movement component is converted into a gimbal pitch angle adjustment based on the camera's vertical focal length. Adjust the drone's yaw angle according to the drone's yaw angle adjustment amount, and adjust the gimbal's pitch angle according to the gimbal's pitch angle adjustment amount, so that the image position of the wind turbine blades in the subsequently acquired images is closer to the image center.

[0016] A visual feedback-based drone attitude correction system includes: The grassland suppression grayscale image generation unit is used to acquire the original color image of the wind turbine blades captured by the UAV's onboard camera, generate a grassland binary mask based on the grassland green threshold range in the HSV color space, and suppress the grassland grayscale of the grayscale original color image according to the grassland binary mask to obtain the grassland suppression grayscale image. The line segment set detection unit is used to generate an edge map based on the grass suppression grayscale map, and to perform probabilistic Hough transform line segment detection on the edge map to obtain a line segment set; The dominant mode determination unit is used to calculate the line segment angle based on the endpoint pixel coordinates of each line segment in the line segment set, and determine the dominant mode of the current image based on the distribution of the line segment angles. The dominant mode includes the horizontal dominant mode and the vertical dominant mode. The line parameter filtering unit is used to determine the slope and intercept of each line segment when the dominant mode is horizontal, using the image abscissa as the independent variable and the image ordinate as the dependent variable, and to filter line segments based on the slope of the blade principal axis of the currently detected blade; when the dominant mode is vertical, it determines the inverse slope and inverse intercept of each line segment using the image ordinate as the independent variable and the image abscissa as the dependent variable, and to filter line segments based on the inverse slope of the blade principal axis of the currently detected blade, thus obtaining the filtered line parameter set. The blade centerline determination unit is used to perform K-means clustering on the filtered set of straight line parameters to obtain two cluster centers corresponding to the two edge lines of the blade, and to determine the blade centerline equation based on the two cluster centers. The pixel movement component calculation unit is used to calculate the horizontal and vertical movement components of the image center point relative to the blade center line based on the image center point coordinates, the blade center line equation, and the dominant mode. The attitude correction conversion unit is used to convert the horizontal movement component into the UAV yaw angle adjustment amount and the vertical movement component into the gimbal pitch angle adjustment amount based on the camera focal length information, and to perform attitude correction according to the UAV yaw angle adjustment amount and the gimbal pitch angle adjustment amount.

[0017] The present invention discloses the following beneficial effects: This invention generates a binary grassland mask based on the green threshold range of the HSV color space, and uses this mask to suppress the grayscale of the original color image after grayscale conversion. This weakens structured background areas such as grassland before they enter the edge map generation and probabilistic Hough transform line segment detection. Compared with methods that only perform general grayscale conversion, filtering, and edge detection, the number of line segments formed by non-leaf background textures can be limited to a lower level. The data objects entering subsequent line segment angle statistics, line segment filtering, and clustering processing are more concentrated in the leaf edge region. Based on this processing, the line segments corresponding to the two edge lines of the leaf have a higher proportion in the filtered set of straight line parameters, which can reduce the interference of background objects such as grassland, forest, and road boundaries on the process of determining the leaf centerline equation.

[0018] This invention calculates the angle of each line segment based on the endpoint pixel coordinates of the line segments in the line segment set, and determines the horizontal or vertical dominant mode based on the distribution of line segment angles. Under different dominant modes, slope and intercept or inverse slope and inverse intercept are used for line segment selection and K-means clustering, respectively. This processing allows the blade to no longer rely on a single slope form for line selection, clustering, and deviation calculation when the blade is in a near-vertical imaging attitude, reducing parameter instability caused by excessively large slope values. Furthermore, this invention calculates the horizontal and vertical movement components based on the image center point coordinates, the blade centerline equation, and the dominant mode, and then converts them into UAV yaw angle adjustment and gimbal pitch angle adjustment based on the camera focal length information, so that the attitude correction is directly obtained from the current image deviation; when the image detection result is valid, it can reduce the reliance on multiple rounds of detection and multiple approximation adjustments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of the method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of image preprocessing provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the line detection results provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the linear clustering results and centerline provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the system structure provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The purpose of this invention is to provide a method and system for UAV attitude correction based on visual feedback. This method uses a grass binary mask to suppress the grayscale of the original color image after grayscale conversion. It also switches between a horizontally dominant mode and a vertically dominant mode based on the distribution of line segment angles, ensuring that the selected set of line parameters more stably reflects the two edge lines of the blade. Furthermore, it calculates the horizontal and vertical movement components based on the blade centerline equation and the coordinates of the image center point, converting them into UAV yaw angle adjustment and gimbal pitch angle adjustment, thereby reducing the impact of false background line segments, abnormal vertical line slopes, and multiple iterative adjustments on the attitude correction process.

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Figure 1 The method flowchart provided in the embodiments of the present invention is as follows: Figure 1 As shown, this invention provides a method for correcting the attitude of a UAV based on visual feedback, comprising: Step 100: Obtain the original color image of the wind turbine blades captured by the UAV's onboard camera, generate a grass binary mask based on the grass green threshold range in the HSV color space, and perform grass grayscale suppression on the grayscaled original color image according to the grass binary mask to obtain a grass suppressed grayscale image. Step 200: Generate an edge map based on the grassland suppression grayscale image, and perform probabilistic Hough transform line segment detection on the edge map to obtain a set of line segments; Step 300: Calculate the line segment angle based on the endpoint pixel coordinates of each line segment in the line segment set, and determine the dominant mode of the current image based on the distribution of line segment angles; the dominant mode includes horizontal dominant mode and vertical dominant mode. Step 400: When the dominant mode is horizontal, determine the slope and intercept of each line segment with the image abscissa as the independent variable and the image ordinate as the dependent variable, and perform line segment filtering based on the slope of the blade principal axis of the currently detected blade; when the dominant mode is vertical, determine the inverse slope and inverse intercept of each line segment with the image ordinate as the independent variable and the image abscissa as the dependent variable, and perform line segment filtering based on the inverse slope of the blade principal axis of the currently detected blade, to obtain the set of filtered line parameters; Step 500: Perform K-means clustering on the filtered set of straight line parameters to obtain two cluster centers corresponding to the two edge lines of the blade, and determine the blade centerline equation based on the two cluster centers; Step 600: Based on the coordinates of the image center point, the equation of the blade centerline, and the dominant mode, calculate the horizontal and vertical movement components of the image center point relative to the blade centerline. Step 700: Convert the horizontal movement component into the UAV yaw angle adjustment amount based on the camera focal length information, convert the vertical movement component into the gimbal pitch angle adjustment amount, and perform attitude correction according to the UAV yaw angle adjustment amount and the gimbal pitch angle adjustment amount.

[0025] In step 100, the UAV's onboard camera acquires raw color images of the wind turbine blades. The raw color images are denoted as... The image data acquired by the airborne camera uses the BGR color space. The original color image... The image is converted to the HSV color space to obtain an HSV image. A grass green threshold range is used to identify grass background areas in the HSV image. As one implementation, the grass green threshold range includes hues... lie in to saturation Located between 43 and 255, brightness Located between 46 and 255. HSV images are thresholded based on the grass green threshold range to generate a grass binary mask. Grassland binary mask middle, Indicates pixel position It belongs to the grassland area. Indicates pixel position It belongs to a non-grassland area.

[0026] Original color image Convert to grayscale image, denoted as: Based on the binary masking of grassland In grayscale image The text describes the process of defining grassland and non-grassland areas. It then mentions conditions for satisfying certain criteria. The pixel position is set to the average gray value of the non-grassland area, or to a predetermined average gray value of the grassland area. For those satisfying... The pixel position, retain the grayscale image The original grayscale values ​​in the image. After grayscale replacement, the grass suppression grayscale image is obtained. .

[0027] Figure 2 The image preprocessing results are shown. Figure 2In the image, the grayscale texture of the grass background area is weakened, the grass edge responds less in subsequent edge detection, while the leaf edge remains in the grayscale image.

[0028] In step 200, the Otsu algorithm is used to suppress the grayscale image of the grassland. Adaptive binarization is performed to obtain a binary image. The Otsu algorithm determines the binarization threshold based on the grayscale distribution in the grassland suppression grayscale image. Grayscale value greater than the threshold The pixel value is set to 255, and the grayscale value is less than or equal to the threshold. The pixels are set to 0. Binary image. This is used to preserve strong edge information such as the blade edge and reduce the impact of background grayscale fluctuations on subsequent edge extraction.

[0029] For binary images Edge detection is performed using the Canny operator to obtain an edge map. The Canny operator performs gradient calculation, non-maximum suppression, and double-thresholding on a binary image, outputting edge lines with a single pixel width. (For edge maps...) Perform probabilistic Hough transform line segment detection to obtain a set of line segments. The probabilistic Hough transform extracts edge points participating in the voting from the edge point set and detects local peaks in the parameter space, outputting line segments with endpoints. (Line segment set) The first in The line segment is denoted as: In the formula, Set of line segments The first in Line segment; and The first The pixel coordinates of the two endpoints of the line segment.

[0030] Figure 3 The results of line detection obtained by probabilistic Hough transform are shown. Figure 3 The line segments in the model are used to provide the basis for filtering blade edge lines. Subsequent steps further filter the set of line segments based on the dominant mode and prior information about the blade main axis.

[0031] In step 300, based on the set of line segments The angle of a line segment is calculated using the pixel coordinates of its endpoints. The angle of a line segment Determined according to the following formula: In the formula, For the first The angle of a line segment; The arctangent function is used to determine the quadrant based on the difference between the vertical and horizontal coordinates. and The first The pixel coordinates of the two endpoints of the line segment. The angle of the line segment. The range of values ​​is line segment angle Represented by angle.

[0032] Statistical set of line segments The distribution of line segment angles in the set. If the number of line segments whose angles fall within a preset horizontal angle range accounts for a certain percentage of the total number of line segments... The dominant mode of the current image is determined by the majority of the total number of line segments. The majority of the total number of line segments determines the dominant mode of the current image as the vertical dominant mode. As one implementation method, the preset horizontal angle range includes... and The preset vertical angle range includes and In another implementation, the line segment angles can be equivalently normalized according to a 180° cycle, and then the number of line segments within the preset horizontal angle interval and the preset vertical angle interval can be counted. The dominant mode is used to determine the expression method of subsequent straight line parameters, avoiding the use of a single slope form when the blade is close to the vertical imaging posture.

[0033] In step 400, the line segment set is processed according to the dominant pattern. The parameters of the line segments are determined and filtered. The prior information of the blade principal axis of the currently detected blade can come from the blade design model or from the detection results of the previous frame image. The prior information of the blade principal axis includes the blade principal axis slope or the blade principal axis reverse slope.

[0034] When the dominant mode is horizontal, the line segment is represented by the horizontal coordinate of the image as the independent variable and the vertical coordinate as the dependent variable. For the first... line segment slope and intercept Determined according to the following formula: In the formula, For the first The slope of a line segment in the horizontally dominant mode; For the first The intercept of a line segment in the horizontally dominant mode; and The first The pixel coordinates of the two endpoints of the line segment.

[0035] Obtain the blade principal axis slope of the currently detected blade. Calculate the slope of each line segment. With the slope of the blade principal axis The difference in slope between them. If the following conditions are met: Then the first One line segment is retained as the line segment. In the formula, The preset filtering threshold is determined by the slope of the retained line segments. and intercept This forms the set of filtered line parameters.

[0036] Exemplary, preset filtering threshold It can be preset according to the image resolution, the imaging width of the blade in the image, or the fluctuation range of the line segment detection results of the previous frame.

[0037] When the dominant mode is vertical, the line segment is represented by the image's vertical coordinate as the independent variable and the image's horizontal coordinate as the dependent variable. For the first... line segment inverse slope and inverse intercept Determined according to the following formula: In the formula, For the first The reverse slope of a line segment in the vertical dominance mode; For the first The inverse intercept of a line segment in the vertical dominance mode.

[0038] Obtain the blade principal axis reverse slope of the currently detected blade. Calculate the reverse slope of each line segment. Anti-slope of blade main axis The difference in the antislope between them. If the following conditions are met: Then the first The line segment is selected as the retained line segment. The reverse slope corresponding to the retained line segment is determined by... and inverse intercept A set of filtered line parameters is formed. The parameter representation in the vertical-dominant mode uses the image's ordinate as the independent variable, which can reduce the problem of excessively large values ​​in the slope calculation of blade edge lines that are close to the vertical direction.

[0039] In step 500, K-means clustering is performed on the filtered set of line parameters. The number of clusters is set to 2. Each line parameter in the filtered set of line parameters is treated as a cluster object. In the horizontally dominant mode, the cluster objects are... In the vertically dominant pattern, the clustered objects are Two elements are selected from the filtered set of line parameters as initial cluster centers. Euclidean distance is used to measure the distance between the clustered objects and the cluster centers, and the two cluster centers are iteratively updated. As one implementation, the maximum number of iterations can be set to 10, and the termination precision can be set to 1.0. The above values ​​are used to illustrate one feasible clustering setup and do not limit other equivalent parameter configurations for K-means clustering.

[0040] After clustering converges, two cluster centers are obtained. The two cluster centers correspond to the parameter centers of the two edge lines of the blade, respectively. Figure 4 The results of linear clustering and the blade centerline are shown. Figure 4 In the diagram, the two clustering results correspond to the edge lines on both sides of the blade, while the center line of the blade is located between the two edge lines.

[0041] In the horizontally dominant model, the two cluster centers are denoted as follows: and The arithmetic mean of the slopes and intercepts at the two cluster centers is used to obtain the slope of the centerline corresponding to the blade centerline equation. Intercept of center line : In the horizontally dominant mode, the blade centerline equation is expressed as: In the formula, This represents the slope of the blade centerline in the horizontally dominant mode. This is the centerline intercept of the blade centerline in the horizontally dominant mode.

[0042] In the vertically dominant model, the two cluster centers are denoted as follows: and The arithmetic mean of the backslope and back intercept in the two cluster centers is taken to obtain the centerline backslope corresponding to the blade centerline equation. and centerline reverse intercept : In the vertical dominance mode, the blade centerline equation is expressed as: In the formula, This represents the anti-slope of the blade centerline in the vertically dominant mode. It is the inverse intercept of the blade centerline in the vertically dominant mode.

[0043] In step 600, based on the image center point coordinates, the blade centerline equation, and the dominant mode, the horizontal and vertical movement components of the image center point relative to the blade centerline are calculated. Let the image width be... Image height is The coordinates of the image center point are: The coordinates of the image center point are determined according to the following formula: In the formula, Image width; Image height; The x-coordinate of the image center point; The ordinate is the y-coordinate of the image center point.

[0044] In the horizontally dominant mode, the blade centerline equation is: Image center point vertical distance to the centerline of the blade It can be expressed as follows: when At this point, the horizontal distance between the image center point and the blade centerline is 0, and the vertical distance is used to generate the vertical movement component. Let... and according to Determine the vertical translation component .when At that time, the horizontal distance of the image center point relative to the blade centerline and vertical distance Determined according to the following formula: In the formula, The horizontal distance between the image center point and the blade centerline in the horizontally dominant mode; This represents the vertical distance between the image center point and the blade centerline in the horizontally dominant mode. The horizontal shift component is determined based on the projective relation. and vertical movement component Determined according to the following formula: In the formula, This represents the horizontal displacement component of the image center point relative to the blade centerline. This represents the vertical movement component of the image center point relative to the blade centerline.

[0045] In the vertical dominance mode, the blade centerline equation is: .when At this point, the vertical distance between the image center point and the blade centerline is 0, and the horizontal distance is used to generate the horizontal movement component. Let... and according to Determine the horizontal translation component .when At that time, the horizontal distance of the image center point relative to the blade centerline and vertical distance Determined according to the following formula: In the formula, The horizontal distance between the image center point and the blade centerline in the vertical dominance mode; This represents the vertical distance between the image center point and the blade centerline in the vertically dominant mode. The horizontal shift component is determined based on the projective relation. and vertical movement component The following formula shall still be used to determine: when and At that time, the center point of the image is located on the center line of the blade, let , .

[0046] In step 700, the horizontal movement component is determined based on the camera focal length information. Convert to UAV yaw angle adjustment and convert the vertical movement component Convert to gimbal tilt angle adjustment. Camera focal length information includes the camera's horizontal focal length. Focal length perpendicular to the camera Camera's horizontal focal length Focal length perpendicular to the camera It can be approximated based on image size and camera field of view: In the formula, The horizontal focal length of the camera; The vertical focal length of the camera; This refers to the horizontal field of view of the camera; The vertical field of view of the camera; Image width; This represents the image height.

[0047] Horizontal translation component Corresponding yaw angle adjustment amount for drones Vertical movement component Corresponding gimbal tilt angle adjustment amount The yaw angle adjustment and gimbal pitch angle adjustment of the UAV are determined according to the following formula: In the formula, This is the yaw angle adjustment amount for the drone; For the gimbal tilt angle adjustment; This is the horizontal translation component; This represents the vertical movement component.

[0048] If the small-angle conditions corresponding to the horizontal and vertical translation components hold, the following formula can be used for approximate calculation: The drone is adjusted according to the drone's yaw angle. Adjust the yaw angle, and the gimbal should be adjusted according to the gimbal pitch angle. Adjust the pitch angle. After attitude correction, the wind turbine blades will appear closer to the image center in subsequent images.

[0049] This embodiment uses a grassland binary mask to suppress the grayscale of the original color image after grayscale conversion. It switches between horizontal and vertical dominance modes to determine the straight line parameters based on the distribution of line segment angles, making the set of filtered straight line parameters more stably reflect the two edge lines of the blade. Furthermore, it calculates the horizontal and vertical movement components based on the blade centerline equation and the coordinates of the image center point, and converts them into the yaw angle adjustment of the UAV and the pitch angle adjustment of the gimbal, thereby reducing the impact of false background line segments, abnormal vertical line slopes, and multiple iterative adjustments on the attitude correction process.

[0050] Figure 5 This is a schematic diagram of the system structure provided in an embodiment of the present invention, such as... Figure 5 As shown, corresponding to the above method, the present invention also provides a UAV attitude correction system based on visual feedback, comprising: The grassland suppression grayscale image generation unit is used to acquire the original color image of the wind turbine blades captured by the UAV's onboard camera, generate a grassland binary mask based on the grassland green threshold range in the HSV color space, and suppress the grassland grayscale of the grayscale original color image according to the grassland binary mask to obtain the grassland suppression grayscale image. The line segment set detection unit is used to generate an edge map based on the grass suppression grayscale map, and to perform probabilistic Hough transform line segment detection on the edge map to obtain a line segment set; The dominant mode determination unit is used to calculate the line segment angle based on the endpoint pixel coordinates of each line segment in the line segment set, and determine the dominant mode of the current image based on the distribution of the line segment angles. The dominant mode includes the horizontal dominant mode and the vertical dominant mode. The line parameter filtering unit is used to determine the slope and intercept of each line segment when the dominant mode is horizontal, using the image abscissa as the independent variable and the image ordinate as the dependent variable, and to filter line segments based on the slope of the blade principal axis of the currently detected blade; when the dominant mode is vertical, it determines the inverse slope and inverse intercept of each line segment using the image ordinate as the independent variable and the image abscissa as the dependent variable, and to filter line segments based on the inverse slope of the blade principal axis of the currently detected blade, thus obtaining the filtered line parameter set. The blade centerline determination unit is used to perform K-means clustering on the filtered set of straight line parameters to obtain two cluster centers corresponding to the two edge lines of the blade, and to determine the blade centerline equation based on the two cluster centers. The pixel movement component calculation unit is used to calculate the horizontal and vertical movement components of the image center point relative to the blade center line based on the image center point coordinates, the blade center line equation, and the dominant mode. The attitude correction conversion unit is used to convert the horizontal movement component into the UAV yaw angle adjustment amount and the vertical movement component into the gimbal pitch angle adjustment amount based on the camera focal length information, and to perform attitude correction according to the UAV yaw angle adjustment amount and the gimbal pitch angle adjustment amount.

[0051] The beneficial effects of this invention are as follows: (1) This invention generates a grassland binary mask based on the green threshold range of grassland in the HSV color space, and then performs grassland grayscale suppression on the original grayscale image based on the grassland binary mask, so that structured background areas such as grassland are grayscale averaged before generating edge maps and performing probabilistic Hough transform line segment detection. As a result, the number of false line segments formed by non-leaf background objects such as grassland textures, forest textures, and road boundaries after edge detection is reduced, and the data objects entering the subsequent line segment filtering, line parameter clustering, and leaf centerline determination processes are more concentrated in the leaf edge region. Compared with the method of only using general grayscale, filtering, and line detection, this invention can reduce the interference of complex backgrounds on the detection and extraction of real leaf edges and improve the stability of the leaf centerline equation determination process under complex field backgrounds.

[0052] (2) This invention calculates the angle of each line segment based on the endpoint pixel coordinates of each line segment in the line segment set, and determines the dominant mode of the current image based on the distribution of line segment angles. When the dominant mode is horizontal, the slope and intercept of each line segment are determined with the image abscissa as the independent variable and the image ordinate as the dependent variable; when the dominant mode is vertical, the inverse slope and inverse intercept of each line segment are determined with the image ordinate as the independent variable and the image abscissa as the dependent variable. The above processing can switch the parameterization form of the line according to the overall direction of the line in the image, so that the process of line segment screening, K-means clustering and determination of the blade centerline equation no longer depends on a single slope representation. When the blade is close to the vertical direction in the image, the inverse slope and inverse intercept participate in the subsequent calculation, which can reduce the instability of slope calculation, failure of screening conditions, imbalance of cluster feature space and abnormal calculation of pixel deviation caused by the slope approaching infinity.

[0053] (3) This invention performs K-means clustering on the filtered set of straight line parameters to obtain two cluster centers corresponding to the two edge lines of the blade, and determines the blade centerline equation based on the two cluster centers. This process can obtain the blade centerline while preserving the geometric features of the blade's double edge lines, making the centerline result independent of a single detected straight line. For images of blades in different imaging postures such as horizontal, tilted, or vertical, the parameter determination methods in both the horizontal and vertical dominant modes can provide the corresponding straight line parameter basis for centerline extraction, thereby improving the consistency of centerline calculation results under different blade postures.

[0054] (4) This invention calculates the horizontal and vertical movement components of the image center point relative to the blade centerline based on the image center point coordinates, the blade centerline equation, and the dominant mode. It then converts the horizontal movement component into a UAV yaw angle adjustment and the vertical movement component into a gimbal pitch angle adjustment based on the camera focal length information. This processing, based on a pinhole camera model and projective geometry, directly converts pixel deviations in the current image into attitude adjustment values, reducing the need for multiple rounds of detection and approximation adjustments relying on proportional control or proportional-derivative control. In inspection scenarios involving blade swaying, UAV attitude disturbances, or narrow camera fields of view, this method improves attitude correction response efficiency, allowing the wind turbine blades to be positioned closer to the image center region in subsequent acquired images more quickly.

[0055] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0056] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for correcting the attitude of a UAV based on visual feedback, characterized in that, include: The original color image of the wind turbine blades is acquired by the airborne camera of the UAV. A grass binary mask is generated based on the grass green threshold range in the HSV color space. Grass grayscale suppression is performed on the grayscale original color image based on the grass binary mask to obtain a grass suppressed grayscale image. An edge map is generated based on the grassland suppression grayscale image, and a probabilistic Hough transform is performed on the edge map to detect line segments, resulting in a set of line segments. The line segment angle is calculated based on the endpoint pixel coordinates of each line segment in the line segment set, and the dominant mode of the current image is determined based on the distribution of the line segment angles. The dominant mode includes a horizontal dominant mode and a vertical dominant mode. When the dominant mode is horizontal, the slope and intercept of each line segment are determined with the image horizontal coordinate as the independent variable and the image vertical coordinate as the dependent variable, and the line segment is filtered based on the slope of the blade principal axis of the currently detected blade; when the dominant mode is vertical, the reverse slope and reverse intercept of each line segment are determined with the image vertical coordinate as the independent variable and the image horizontal coordinate as the dependent variable, and the line segment is filtered based on the reverse slope of the blade principal axis of the currently detected blade, thus obtaining a set of filtered line parameters. K-means clustering is performed on the filtered set of straight line parameters to obtain two cluster centers corresponding to the two edge lines of the blade, and the blade centerline equation is determined based on the two cluster centers. Based on the image center point coordinates, the blade centerline equation, and the dominant mode, calculate the horizontal and vertical movement components of the image center point relative to the blade centerline. The horizontal movement component is converted into a yaw angle adjustment amount for the UAV based on the camera focal length information, and the vertical movement component is converted into a gimbal pitch angle adjustment amount. Attitude correction is then performed according to the UAV yaw angle adjustment amount and the gimbal pitch angle adjustment amount.

2. The UAV attitude correction method based on visual feedback according to claim 1, characterized in that, The method involves generating a grass binary mask based on the grass green threshold range in the HSV color space, and then performing grass grayscale suppression on the grayscaled original color image using the grass binary mask to obtain a grass-suppressed grayscale image, including: The original color image is converted from the BGR color space to the HSV color space to obtain an HSV image; The HSV image is thresholded according to the green threshold range of the grassland to generate the grassland binary mask; The original color image is converted into a grayscale image, and the grassland and non-grass regions are determined in the grayscale image based on the grassland binary mask. The grayscale value of the grassland area is set to the average grayscale value of the non-grassland area or the average grayscale value of the grassland, while retaining the original grayscale value of the non-grassland area, to obtain the grassland suppressed grayscale image.

3. The UAV attitude correction method based on visual feedback according to claim 1, characterized in that, The step involves generating an edge map based on the grassland suppression grayscale image, and performing probabilistic Hough transform line segment detection on the edge map to obtain a set of line segments, including: The Otsu algorithm is used to adaptively binarize the grassland suppression grayscale image to obtain a binary image; The edge map is obtained by performing edge detection on the binary image using the Canny operator. The edge map is subjected to probabilistic Hough transform line segment detection to obtain the line segment set, and each line segment in the line segment set is represented by the pixel coordinates of its two endpoints.

4. The UAV attitude correction method based on visual feedback according to claim 1, characterized in that, The process involves calculating the line segment angles based on the endpoint pixel coordinates of each line segment in the line segment set, and determining the dominant mode of the current image based on the distribution of the line segment angles. The dominant mode includes a horizontal dominant mode and a vertical dominant mode, comprising: Calculate the corresponding line segment angle based on the pixel coordinates of the two endpoints of each line segment; Statistically analyze the distribution of line segment angles for each line segment in the given set; When the number of line segments whose angles fall within the preset horizontal angle range exceeds half the total number of line segments in the line segment set, the dominant mode is determined to be the horizontal dominant mode. When the number of line segments whose angles fall within the preset vertical angle range exceeds half the total number of line segments in the line segment set, the dominant mode is determined to be the vertical dominant mode.

5. The UAV attitude correction method based on visual feedback according to claim 1, characterized in that, When the dominant mode is horizontal, the slope and intercept of each line segment are determined using the horizontal coordinate of the image as the independent variable and the vertical coordinate of the image as the dependent variable. Line segment selection is then performed based on the slope of the blade's principal axis at the currently detected blade, including: When the dominant mode is the horizontal dominant mode, the slope and intercept of each line segment are determined according to the pixel coordinates of the endpoints of each line segment and the correspondence between the horizontal and vertical coordinates of the image. Obtain the blade principal axis slope of the currently detected blade; Calculate the slope difference between the slope of each line segment and the slope of the blade main shaft; Line segments with slope differences less than a preset screening threshold are retained as line segments, and the slopes and intercepts corresponding to the retained line segments form the set of filtered line parameters.

6. The UAV attitude correction method based on visual feedback according to claim 1, characterized in that, When the dominant mode is vertical, the reverse slope and reverse intercept of each line segment are determined using the image's vertical coordinate as the independent variable and the image's horizontal coordinate as the dependent variable. Line segment selection is then performed based on the reverse slope of the blade's principal axis at the currently detected blade, including: When the dominant mode is the vertical dominant mode, the inverse slope and inverse intercept of each line segment are determined according to the pixel coordinates of the endpoints of each line segment and the correspondence between the vertical coordinate and the horizontal coordinate of the image. Obtain the blade principal axis anti-slope of the currently detected blade; Calculate the difference in reverse slope between the reverse slope of each line segment and the reverse slope of the blade main shaft; Line segments with a backslope difference less than a preset screening threshold are retained as line segments, and the backslope and back intercept corresponding to the retained line segments form the set of line parameters after screening.

7. The UAV attitude correction method based on visual feedback according to claim 1, characterized in that, The step of performing K-means clustering on the filtered set of straight line parameters to obtain two cluster centers corresponding to the two edge lines of the blade, and determining the blade centerline equation based on the two cluster centers, includes: The line parameters in the filtered set of line parameters are used as cluster objects, and the number of clusters is set to 2. K-means clustering was performed on the clustered objects to obtain two cluster centers; The two cluster centers are respectively determined as the parameter centers corresponding to the two edge lines of the leaf; When the dominant mode is the horizontal dominant mode, the arithmetic mean of the slopes in the two cluster centers and the arithmetic mean of the intercepts in the two cluster centers are taken to obtain the blade centerline equation. When the dominant mode is the vertical dominant mode, the arithmetic mean of the inverse slopes in the two cluster centers and the arithmetic mean of the inverse intercepts in the two cluster centers are taken to obtain the blade centerline equation.

8. The UAV attitude correction method based on visual feedback according to claim 1, characterized in that, The step of calculating the horizontal and vertical movement components of the image center point relative to the blade centerline based on the image center point coordinates, the blade centerline equation, and the dominant mode includes: The coordinates of the image center point are determined based on the image width and image height; When the dominant mode is the horizontal dominant mode, the horizontal and vertical distances of the image center point relative to the blade centerline are determined based on the coordinates of the image center point and the slope and intercept in the blade centerline equation, and the horizontal and vertical movement components are calculated based on the horizontal and vertical distances. When the dominant mode is the vertical dominant mode, the horizontal and vertical distances of the image center point relative to the blade centerline are determined based on the coordinates of the image center point and the inverse slope and inverse intercept in the blade centerline equation, and the horizontal and vertical movement components are calculated based on the horizontal and vertical distances.

9. The UAV attitude correction method based on visual feedback according to claim 1, characterized in that, The step of converting the horizontal movement component into a UAV yaw angle adjustment based on the camera focal length information, converting the vertical movement component into a gimbal pitch angle adjustment, and performing attitude correction according to the UAV yaw angle adjustment and the gimbal pitch angle adjustment includes: The horizontal movement component is converted into the yaw angle adjustment of the UAV based on the horizontal focal length of the camera. The vertical movement component is converted into the gimbal pitch angle adjustment amount based on the camera's vertical focal length. Adjust the yaw angle of the UAV according to the yaw angle adjustment amount, and adjust the gimbal pitch angle according to the gimbal pitch angle adjustment amount, so that the imaging position of the wind turbine blades in the subsequent acquired images is closer to the center of the image.

10. A visual feedback-based unmanned aerial vehicle (UAV) attitude correction system, characterized in that, include: The grassland suppression grayscale image generation unit is used to acquire the original color image of the wind turbine blades captured by the UAV airborne camera, generate a grassland binary mask based on the grassland green threshold range in the HSV color space, and perform grassland grayscale suppression on the grayscale original color image according to the grassland binary mask to obtain the grassland suppression grayscale image. The line segment set detection unit is used to generate an edge map based on the grassland suppression grayscale map, and to perform probabilistic Hough transform line segment detection on the edge map to obtain a line segment set; The dominant mode determination unit is used to calculate the line segment angle based on the endpoint pixel coordinates of each line segment in the line segment set, and determine the dominant mode of the current image based on the distribution of the line segment angles. The dominant mode includes a horizontal dominant mode and a vertical dominant mode. The line parameter filtering unit is used to determine the slope and intercept of each line segment with the image horizontal coordinate as the independent variable and the image vertical coordinate as the dependent variable when the dominant mode is horizontal dominant mode, and to filter line segments based on the slope of the blade principal axis of the currently detected blade; when the dominant mode is vertical dominant mode, it determines the inverse slope and inverse intercept of each line segment with the image vertical coordinate as the independent variable and the image horizontal coordinate as the dependent variable, and to filter line segments based on the inverse slope of the blade principal axis of the currently detected blade, thereby obtaining a set of filtered line parameters. The blade centerline determination unit is used to perform K-means clustering on the filtered set of straight line parameters to obtain two cluster centers corresponding to the two edge lines of the blade, and to determine the blade centerline equation based on the two cluster centers. The pixel movement component calculation unit is used to calculate the horizontal and vertical movement components of the image center point relative to the blade center line based on the coordinates of the image center point, the equation of the blade center line, and the dominant mode. The attitude correction conversion unit is used to convert the horizontal movement component into a UAV yaw angle adjustment amount and the vertical movement component into a gimbal pitch angle adjustment amount based on the camera focal length information, and to perform attitude correction according to the UAV yaw angle adjustment amount and the gimbal pitch angle adjustment amount.

Images