An unmanned aerial vehicle optimal path planning method and digital system fusing image recognition, M-RRT and APF algorithm

Abstract

The application provides an unmanned aerial vehicle optimal track planning method and system fusing image recognition and M-RRT and APF algorithms. Wind power generator point cloud and visual data are acquired in real time through a multi-modal sensor, key components of blades are recognized based on deep learning, a geometric mapping model is constructed, tip point is corrected in combination with DBSCAN clustering, and a shutdown posture is predicted. A global path adopts an M-RRT algorithm improved by a direction heuristic factor, and a key area of the blade is guided to search; a local track is optimized through a dynamic weight APF algorithm, and a repulsive force field is adjusted in real time to cope with posture changes. An online replanning mechanism is introduced, NSGA-III multi-objective optimization is triggered when an environment mutates or a tracking error exceeds a threshold, and optimal tracks are generated by comprehensively considering energy consumption, time and safety. The system integrates high-precision perception, adaptive planning and dynamic optimization, and significantly improves inspection coverage and track safety under a complex shutdown posture.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of unmanned aerial vehicle autonomous navigation and intelligent control technology, and particularly relates to an unmanned aerial vehicle optimal path planning method and system fusing image recognition and improved rapid extended random tree (M-RRT) and artificial potential field (APF) algorithm, and is particularly suitable for autonomous inspection operation in a complex environment with uncertain shutdown posture of a wind turbine. BACKGROUND

[0002] With the rapid growth of global wind power installed capacity, regular inspection and maintenance of wind turbines has become a key link to ensure safe and efficient operation of wind farms. The traditional manual inspection method has inherent defects such as low efficiency, high cost and high risk, while the unmanned aerial vehicle inspection technology is gradually becoming the mainstream solution for wind turbine inspection due to its advantages of strong flexibility, low cost and wide view angle.

[0003] Current unmanned aerial vehicle wind power inspection mainly adopts a preset path automatic flight mode, which plans a fixed inspection path by pre-inputting geometric parameters and spatial positions of wind turbines. This method can better complete the inspection task when the wind turbine is in a standard shutdown posture (the blades are arranged in the shape of "Y" or "1"). However, in actual operation, when the wind turbine is shut down due to fault repair, power grid scheduling or adverse weather, etc., its blades often present various non-standard postures, including: single or multiple blades at any yaw angle, abnormal blade pitch angle, blade twist state change, random offset of the nacelle orientation, etc.

[0004] The fixed flight path planning method proposed by the existing research cannot cope with the dynamic changes of the attitude of the wind turbine, mainly showing that: (1) the adaptability defects of fixed flight path planning, such as the preset safety margin may be broken through by the blade of abnormal attitude, resulting in collision risk; the key detection points (such as blade bolts and lightning strike points) may be out of the camera field of view due to attitude changes; sudden gusts may cause the actual blade position to deviate from the expected trajectory; the actual measurement data shows that when the blade yaw angle exceeds ± 30°, the detection integrity rate of the traditional method decreases by more than 40%; (2) the limitations of path planning algorithms, such as in the rapid expansion random tree (RRT) algorithm: the traditional RRT (such as CN109917826A) has slow convergence speed in three-dimensional space, the planning time often exceeds 500ms, the generated path has redundant nodes, the smoothness is poor, and it does not meet the dynamic constraints of the unmanned aerial vehicle; the response to dynamic obstacles is lagging, and the re-planning efficiency is low. In the artificial potential field (APF) algorithm: the traditional APF is easy to fall into local minimum value in the dense area of wind turbines, the potential field parameters are fixed, and it cannot adapt to the size difference of wind turbines of different capacities; the processing effect of the blade tip moving quickly is poor, and path oscillation may occur; (3) the deficiencies of environment perception and modeling, the current modeling technology relies on the pre-input static wind turbine model, lacks real-time attitude perception ability, the dynamic update frequency of the blade scanning area is insufficient, the influence of wind speed on the actual position of the blade is not considered, and the feature extraction of different types of wind turbines lacks generality; (4) the defects of system integration, the current solutions are mostly independent modules, which have different time delays in the perception-planning-control links; the response mechanism to sudden situations is single, lacks multi-objective optimization for inspection quality, and is difficult to support multi-machine cooperative inspection.

[0005] The core contradictions in the current technical development are: the contradiction between the uncertainty of the attitude of the wind turbine and the requirement of the determination of the path planning, the contradiction between the algorithm calculation complexity and the real-time requirement of the system, the contradiction between the local obstacle avoidance safety and the global inspection integrity, and the contradiction between the single machine type special algorithm and the compatibility of wind farms of different types. These contradictions lead to the problems of response lag, path suboptimality and limited adaptability of the existing unmanned aerial vehicle inspection system when responding to sudden attitude changes of the wind turbine, and a new generation of intelligent planning method needs to be developed. SUMMARY

[0006] Therefore, it is necessary to provide an unmanned aerial vehicle optimal flight path planning method and system integrating image recognition and M-RRT and APF algorithms, aiming to solve the real-time flight path planning problem of unmanned aerial vehicle autonomous inspection under the condition of uncertain attitude of wind turbine, and overcome the defects of poor adaptability, low planning efficiency and non-smooth path in the prior art.

[0007] In a first aspect, the embodiments of the present application provide a UAV optimal path planning method combining image recognition and M-RRT and APF algorithms, which comprises:

[0008] Real-time acquisition of three-dimensional point cloud data and visual information of the wind turbine through a multi-modal sensor array;

[0009] Identification of key components of the wind turbine and estimation of the shutdown attitude parameters based on a deep learning model, and establishment of a geometric mapping model of the spatial position of the wind turbine blade and the image taken by the UAV;

[0010] Design of a key point detection model to identify the blade tip point, and automatic measurement of the current shutdown attitude of the wind turbine by using DBSCAN clustering to filter the wrong blade tip point;

[0011] Inputting the current shutdown attitude into the geometric mapping model to obtain the predicted shutdown attitude of the wind turbine at the next time;

[0012] Based on the current attitude of the wind turbine, an improved M-RRT algorithm combining a direction heuristic factor is used to generate a global inspection path;

[0013] Based on the predicted shutdown attitude of the wind turbine, an improved APF algorithm with dynamic weight distribution is used for local trajectory optimization;

[0014] Triggering online re-planning according to environmental changes, task requirements and equipment states;

[0015] Optimal path selection based on multi-objective optimization of energy consumption, time, and safety factor.

[0016] Optionally, in an implementation manner of the first aspect of the present application, the identification of the key components of the wind turbine and the estimation of the shutdown attitude parameters based on the deep learning model, and the establishment of the geometric mapping model of the spatial position of the wind turbine blade and the image taken by the UAV, comprises:

[0017] Detection and positioning of the wind turbine blade by training a deep learning model;

[0018] Extraction of blade edge information through Canny edge detection and Hough transformation;

[0019] Calculation of the pixel coordinates of the blade tip according to the blade tip pixel coordinates and the UAV attitude information, and through coordinate system transformation;

[0020] Calculation of the GPS coordinates of the blade tip in combination with the attitude information of the UAV, and determination of the rotation angle of the blade;

[0021] According to the mapping relationship between the movement trend of the blade, the environmental changes, and the rotation angle and yaw angle at different time points, a geometric mapping model of the spatial position of the wind turbine blade and the image taken by the UAV is constructed.

[0022] Optionally, in an implementation form of the first aspect of the application, the design key point detection model identifies the blade tip point, adopts DBSCAN clustering to filter the wrong blade tip point, and automatically measures the current shutdown posture of the wind turbine, comprising:

[0023] identifying the blade tip pixel based on the deep learning model;

[0024] correcting the blade tip pixel through the Hough straight line, and identifying the point in the low-density area through the DBSCAN clustering algorithm to eliminate the data point deviating from the abnormal angle;

[0025] determining the blade inclination parameter according to the reference relationship between the coordinate axis in the coordinate system by analyzing the distribution of the blade tip point;

[0026] automatically measuring the current shutdown posture of the wind turbine.

[0027] Optionally, in an implementation form of the first aspect of the application, the current shutdown posture is input into the geometric mapping model to obtain the predicted shutdown posture of the wind turbine at the next moment, comprising:

[0028] obtaining the arrival time of the next wind turbine;

[0029] inputting the arrival time and the current shutdown posture into the geometric mapping model to predict the shutdown posture at the next moment;

[0030] taking the shutdown posture at the next moment as the predicted shutdown posture of the next wind turbine.

[0031] Optionally, in an implementation form of the first aspect of the application, the global inspection path is generated based on the current posture of the wind turbine by using the improved M-RRT algorithm with the fusion of the direction heuristic factor, comprising:

[0032] environment perception and posture modeling: the current three-dimensional posture and the blade space position of the wind turbine are obtained in real time by using the high-precision IMU and the laser radar, and the obstacle point cloud map including the tower drum and the blade surface is constructed;

[0033] direction heuristic factor fusion: the direction weight function is introduced in the traditional RRT random sampling :

[0034] ,

[0035] wherein, is the direction gain coefficient, is the direction vector from the starting point to the target point, is the direction vector from the starting point to the random point, represents the length of the module between the target point and the random point;

[0036] The guiding search is biased to the key detection area of the blade, including the blade root and the leading edge area, and the invalid extended nodes are reduced.

[0037] Dynamic step adjustment: adaptively adjust the expansion step according to the curvature of the blade, increase the step in the flat area, and reduce the step in the curved area.

[0038] Optionally, in an implementation form of the first aspect of the present application, the predicted shutdown posture of the wind turbine is used for local trajectory optimization by using an improved APF algorithm with dynamic weight distribution, including:

[0039] Posture prediction and constraint modeling: combining the operation data of the wind turbine, including the speed, the wind speed, and the historical shutdown posture, the swing range of the blade in the shutdown state is predicted through an LSTM network, and the dynamic forbidden area of the maximum swing envelope of the blade is determined;

[0040] Reconstruction of repulsive force field reconstruction, real-time adjustment of repulsive force intensity of forbidden area, wherein the attractive force generated by the repulsive force field is :

[0041] ,

[0042] wherein, is the repulsive force weight, , is the time decay factor, is the state deviation, , are the corresponding deviation coefficients, is the repulsive force coefficient, is the distance between the current point and the target obstacle, is the distance between the starting point and the target obstacle, and the repulsive force intensity of the forbidden area is adjusted in real time;

[0043] Attractive force-repulsive force weight distribution: increase the repulsive force weight when close to the high-risk area of the blade tip, and strengthen the attractive force field in the safe path segment, balance the obstacle avoidance efficiency and path smoothness;

[0044] Judge whether a collision with an obstacle occurs;

[0045] If so, reselect the parent node and rewire;

[0046] Otherwise, form an obstacle avoidance trajectory path and perform pruning optimization on the trajectory path.

[0047] Optionally, in an implementation form of the first aspect of the present application, online re-planning is triggered according to environmental changes, task requirements and device states; and the optimal trajectory is selected based on multi-objective optimization of energy consumption, time and safety coefficient, including:

[0048] Setting online re-planning trigger conditions, including: image recognition-based wind turbine posture mutation detection trigger, path tracking error exceeding threshold trigger, environmental obstacle intrusion into safety area trigger;

[0049] Establishing a multi-objective optimization evaluation module, including: establishing a multi-objective function including path length, safety, energy consumption, and shooting angle, and using NSGA-III algorithm to select Pareto optimal solution, wherein, in the Pareto optimal solution set generated by the NSGA-III algorithm, the optimal solution is selected as the final re-planning path according to the preference of the decision maker or the weight of the expert system.

[0050] In the second aspect, the embodiments of the present application provide a UAV optimal path planning system fusing image recognition and M-RRT and APF algorithms, applied to the UAV optimal path planning method fusing image recognition and M-RRT and APF algorithms as described in the first aspect, comprising:

[0051] A multi-modal data acquisition module is configured to acquire wind turbine three-dimensional point cloud data and visual information in real time through a multi-modal sensor array;

[0052] A geometric mapping module is configured to identify key components of a wind turbine and estimate a shutdown posture parameter based on a deep learning model, and establish a geometric mapping model of wind turbine blade space position and UAV shooting image;

[0053] A posture recognition module is configured to design a key point detection model to identify blade tip points, and use DBSCAN clustering to filter false blade tip points, and automatically measure the current shutdown posture of the wind turbine;

[0054] A posture prediction module is configured to input the current shutdown posture into the geometric mapping model to obtain a predicted shutdown posture of the wind turbine at the next time;

[0055] A path planning module is configured to generate a global inspection path based on the current posture of the wind turbine using an improved M-RRT algorithm fusing a direction heuristic factor;

[0056] A path optimization module is configured to perform local trajectory optimization using an improved APF algorithm with dynamic weight distribution based on the predicted shutdown posture of the wind turbine;

[0057] A dynamic re-planning module is configured to trigger online re-planning according to environmental changes, task requirements, and equipment states;

[0058] A multi-objective optimization module is configured to select an optimal path based on multi-objective optimization of energy consumption, time, and safety factor.

[0059] In a third aspect, an electronic device is provided, comprising:

[0060] a processor;

[0061] a memory for storing processor-executable instructions;

[0062] wherein the processor is configured to implement the method for optimal UAV path planning by fusing image recognition with M-RRT and APF algorithm as described in the first aspect when executing the instructions.

[0063] In a fourth aspect, a computer-readable storage medium is provided, which stores a program instructing a device to execute the method for optimal UAV path planning by fusing image recognition with M-RRT and APF algorithm as described in the first aspect.

[0064] The present application proposes a method and system for optimal UAV path planning by fusing image recognition with M-RRT and APF algorithm. Wind turbine point cloud and vision data are acquired in real time by multi-modal sensors, key components of the blades are identified based on deep learning, and a geometric mapping model is constructed. The tip points are corrected by DBSCAN clustering, and the shutdown posture is predicted. The global path uses the M-RRT algorithm improved by the direction heuristic factor to guide the search in the key areas of the blades. The local trajectory is optimized by the dynamic weight APF algorithm, and the repulsive force field is adjusted in real time to respond to changes in posture. An online replanning mechanism is introduced, and NSGA-III multi-objective optimization is triggered when the environment mutates or the tracking error exceeds the threshold. The optimal path is generated by considering energy consumption, time, and safety. The system integrates high-precision perception, adaptive planning, and dynamic optimization, significantly improving the inspection coverage rate and trajectory safety under complex shutdown postures.

[0065] Advantages:

[0066] (1) High-precision perception and modeling: Combining multi-modal sensors and deep learning, key components of the wind turbine are accurately identified and the shutdown posture is predicted, improving the reliability of environmental modeling.

[0067] (2) Efficient global planning: The improved M-RRT algorithm adjusts the direction heuristic factor and dynamic step length, significantly reducing invalid searches and improving the coverage efficiency of key areas of the blades (such as the blade root and leading edge).

[0068] (3) Dynamic obstacle avoidance optimization: Based on the dynamic weight allocation strategy of the APF algorithm, the blade swing is responded in real time, the path smoothness and obstacle avoidance ability are balanced, and the collision risk is reduced.

[0069] (4) Adaptive replanning: Through multi-objective optimization (energy consumption, time, and safety) and online replanning mechanism, the path remains optimal in the event of environmental mutations or device abnormalities.

[0070] (5) Engineering practicality: The system integrates sensor fusion, real-time computing and lightweight deployment, and is suitable for wind turbine inspection under complex shutdown posture, taking into account both efficiency and safety. Attached Figure Description

[0071] Figure 1 This is a schematic flowchart of an embodiment of the UAV optimal trajectory planning method that integrates image recognition with M-RRT and APF algorithms.

[0072] Figure 2 This is a schematic diagram illustrating the principle of the RRT algorithm provided in an embodiment of this application.

[0073] Figure 3 This is a schematic diagram of a UAV optimal trajectory planning system module that integrates image recognition with M-RRT and APF algorithms, as provided in an embodiment of this application.

[0074] Figure 4 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0075] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0076] It should be noted that, in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0077] It should be noted that in the embodiments of this application, the terms "first," "second," etc., are used only for descriptive purposes and should not be construed as indicating or implying relative importance, nor as indicating or implying order. Features specified as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0078] Based on the embodiments described in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0079] Example 1

[0080] Figure 1 This is a schematic flowchart of an embodiment of the UAV optimal trajectory planning method that integrates image recognition with M-RRT and APF algorithms.

[0081] like Figure 1 As shown, a method for optimal UAV trajectory planning that integrates image recognition with M-RRT and APF algorithms includes:

[0082] Step S101: Real-time acquisition of three-dimensional point cloud data and visual information of wind turbine generators through a multi-modal sensor array.

[0083] Real-time acquisition of 3D point cloud data and visual information of wind turbines using a multimodal sensor array is one of the key technologies in the monitoring and maintenance of wind power equipment. This step aims to achieve high-precision and high-efficiency 3D modeling and real-time monitoring of the wind turbine structure through the collaborative work of multiple sensors, thereby providing data support for subsequent deformation monitoring, defect identification, and fault early warning.

[0084] Specifically, multimodal sensor arrays typically include the following sensors: LiDAR (Light Detection and Ranging): Used to acquire high-precision 3D point cloud data. LiDAR emits a laser beam and measures its reflection time to quickly and continuously acquire 3D point cloud data of the target object. High-definition camera: Used to acquire visual information of wind turbine blades. High-definition cameras can capture surface images of wind turbine blades, supporting subsequent defect identification and texture analysis. Millimeter-wave radar: Used to assist in acquiring 3D point cloud data of wind turbine blades. Millimeter-wave radar has strong penetration capabilities in complex environments and can provide relatively stable point cloud data. Six-dimensional force sensor: Used to monitor changes in contact force between the wind turbine blade and the end effector, providing feedback for robotic arm or robot operations. Temperature sensor: Used to monitor temperature changes of the wind turbine blade, providing data support for temperature compensation models.

[0085] 3D point cloud data is acquired by scanning wind turbine blades with sensors such as LiDAR to record the 3D coordinate information of their surface. The processing of 3D point cloud data includes the following steps: Point cloud preprocessing: including denoising, normalization, and registration to improve the quality of the point cloud data. Point cloud registration: unifying point cloud data collected by different sensors into the same coordinate system to improve the accuracy of subsequent analysis. Feature extraction: extracting key features from the point cloud data, such as blade contours and defect areas.

[0086] The acquisition of visual information primarily relies on equipment such as high-definition cameras to capture surface images of wind turbine blades. Visual information processing includes the following steps: Image preprocessing: This includes operations such as denoising, enhancement, and segmentation to improve image clarity and contrast. Feature extraction: Extracting key features from the image, such as blade edges and defect areas. Defect recognition: Identifying and classifying defects in the image using deep learning models.

[0087] Multimodal data fusion combines data collected by sensors such as LiDAR, high-definition cameras, and six-dimensional force sensors to improve the accuracy and real-time performance of monitoring. It can be uniformly identified using timestamps.

[0088] Step S102: Based on a deep learning model, identify key components of the wind turbine and estimate shutdown attitude parameters, and establish a geometric mapping model between the spatial position of the wind turbine blades and images captured by the UAV.

[0089] Specifically, the step of identifying key components of a wind turbine and estimating its shutdown attitude parameters based on a deep learning model, and establishing a geometric mapping model between the spatial position of the wind turbine blades and images captured by a UAV, includes:

[0090] The detection and localization of wind turbine blades are achieved by training a deep learning model.

[0091] Blade edge information was extracted using Canny edge detection and Hough transform.

[0092] The pixel coordinates of the leaf tip are calculated based on the leaf tip pixel coordinates and the UAV pose information, and through coordinate system transformation.

[0093] By combining the drone's pose information, the GPS coordinates of the blade tip are calculated, and the rotation angle of the blade is determined.

[0094] Based on the mapping relationship between the blade's motion trend, environmental changes, and rotation and yaw angles at different time points, a geometric mapping model of the spatial position of the wind turbine blade and images captured by UAVs is constructed.

[0095] In this embodiment, deep learning algorithms (such as YOLOv5, YOLOv7, Mask R-CNN, etc.) can be used to detect and locate wind turbine blades. These models can automatically identify blades in images and extract their key features, such as blade tip, blade root, and blade center. Based on the blade detection by the deep learning model, traditional image processing techniques (such as Canny edge detection and Hough transform) can be combined to further extract the edge information of the blades. Canny edge detection can effectively extract the contour of the blades, while Hough transform can be used to detect the straight line features of the blades, thereby assisting in the localization and attitude estimation of the blades. These methods maintain high robustness even under complex backgrounds or varying lighting conditions.

[0096] During flight, drones capture images of wind turbine blades. Deep learning models can then be used to obtain the pixel coordinates of the blade tips. Simultaneously, the drone's pose information (such as GPS coordinates and attitude angles) provides the blades' position in three-dimensional space. Through coordinate system transformation (such as from world coordinates to camera coordinates), the pixel coordinates of the blade tips can be mapped to three-dimensional space, allowing the calculation of the blade tip's GPS coordinates. This process typically involves camera calibration to determine intrinsic and extrinsic parameters, ensuring accurate mapping between the image and three-dimensional space.

[0097] By analyzing the UAV's pose information (such as yaw and pitch angles), the blade rotation angle can be further calculated. For example, by calculating the change in the blade tip's position in an image, the blade's rotation angle can be deduced. Furthermore, by combining the blade's motion trend with environmental changes, a mapping relationship between the blade rotation angle and time points can be established. This method can effectively estimate the blade's stationary attitude parameters, providing data support for subsequent inspections and maintenance.

[0098] Finally, by analyzing the blade's motion trend, environmental changes, and the mapping relationship between rotation and yaw angles at different time points, a geometric mapping model of the spatial position of wind turbine blades and UAV-captured images can be constructed. This model can associate the spatial position information of the blades with feature points in the UAV images, thereby enabling real-time monitoring and fault diagnosis of the blades. For example, through blade trajectory mesh diagrams and temporal shape analysis, blade offsets and angle changes can be further identified.

[0099] Step S102 combines deep learning models, traditional image processing techniques, and UAV pose information to achieve accurate identification of key components of wind turbine blades and estimation of shutdown attitude parameters, thereby establishing a geometric mapping model between the spatial position of wind turbine blades and images captured by UAVs.

[0100] Step S103: Design a key point detection model to identify blade tip points, use DBSCAN clustering to filter out erroneous blade tip points, and automatically measure the current shutdown attitude of the wind turbine.

[0101] Specifically, the design key point detection model identifies blade tip points, uses DBSCAN clustering to filter erroneous blade tip points, and automatically measures the current shutdown attitude of the wind turbine, including:

[0102] Identifying blade tip pixels using a deep learning model;

[0103] The blade tip pixels are corrected using Hough lines, and low-density areas are identified using the DBSCAN clustering algorithm to eliminate data points that deviate from abnormal angles.

[0104] By analyzing the distribution of the blade tip, and based on the reference relationship with the coordinate axes in the coordinate system, the blade tilt angle parameters are determined.

[0105] Automatically measures the current shutdown attitude of the wind turbine.

[0106] Specifically, deep learning models (such as object detection algorithms) are used to identify blades in images and extract the pixel positions of the blade tips. Deep learning models are typically based on pre-trained detection models and are trained on labeled training datasets to achieve accurate identification of blade tip positions. For example, by training on a self-made dataset, the pixel coordinates of the blade tips can be extracted. Deep learning object detection algorithms can then be used to obtain the coordinates of the center point of the blade tip. and width and height This allows us to determine the position of the leaf tip.

[0107] After identifying the leaf tip pixels, the distribution of these pixels may be inaccurate due to image noise or detection errors. Therefore, the Hough line transform is used to correct the leaf tip pixels and improve their accuracy. The Hough line transform can detect straight line structures in the image, thereby helping to correct the distribution of leaf tip pixels to better match the actual geometry.

[0108] Based on the corrected blade tip points, the DBSCAN clustering algorithm is used to perform cluster analysis on the blade tip points. DBSCAN is a density-based clustering algorithm that can automatically identify low-density regions in the dataset and treat points in these low-density regions as outliers. DBSCAN can discover clusters of arbitrary shapes and can identify noise points, thus effectively filtering out blade tip points that deviate from the normal angle. For example, when the blade angle is greater than 2°, these data points are usually considered outliers and need to be eliminated. The DBSCAN algorithm can also define concepts such as ε-neighborhood and density reachability of points, and can identify points in low-density regions and mark them as noise.

[0109] After filtering out abnormal blade tip points, the distribution of these points is analyzed, and the blade's tilt angle parameters are determined by combining this with the coordinate axis relationships in the coordinate system. By analyzing the distribution of blade tip points, the blade's tilt angle relative to the coordinate system can be determined, thus allowing the calculation of the blade's tilt angle parameters. Furthermore, by analyzing the distribution of blade tip points, the blade's rotation direction and position can be determined, further enabling the calculation of the blade's tilt angle.

[0110] Finally, based on the blade tilt parameters obtained from the above steps, the current shutdown attitude of the wind turbine is automatically measured. For example, by analyzing the distribution of the blade tips and the tilt parameters, it can be determined whether the wind turbine is in a shutdown state and its shutdown attitude can be calculated. Analyzing the distribution of the blade tips and the tilt parameters can also determine the operating status of the wind turbine and provide corresponding control information.

[0111] Step S104: Input the current shutdown attitude into the geometric mapping model to obtain the predicted shutdown attitude of the wind turbine at the next moment.

[0112] Specifically, inputting the current shutdown attitude into the geometric mapping model to obtain the predicted shutdown attitude of the wind turbine at the next moment includes:

[0113] Obtain the arrival time to the next wind turbine;

[0114] The arrival time and the current stopping attitude are input into the geometric mapping model to predict the stopping attitude at the next moment;

[0115] The shutdown attitude at the next moment is used as the predicted shutdown attitude of the next wind turbine.

[0116] Specifically, during wind turbine inspection or maintenance, it is necessary to obtain the arrival time of the next wind turbine based on the mission plan and flight path. This time is usually calculated based on the drone's flight speed, range, and mission scheduling strategy.

[0117] The current wind turbine's shutdown attitude parameters (such as blade tilt angle and yaw angle) and the estimated arrival time of the next wind turbine are input into the geometric mapping model. This model, based on visual SLAM technology, establishes a relationship between the observation model and the camera motion model through matching points and feature points, thereby predicting the shutdown attitude of the next wind turbine. The geometric mapping model, based on the input current shutdown attitude and arrival time, combined with the wind turbine's dynamic characteristics (such as blade rotation and wind speed changes), predicts the shutdown attitude at the next moment. This prediction result can be used for UAV path planning and attitude adjustment to improve inspection efficiency and safety.

[0118] The predicted results serve as the initial state for the next wind turbine, used for subsequent shutdown attitude analysis and control decisions. This process enables continuous tracking and prediction of wind turbine shutdown states, supporting automated operation and maintenance.

[0119] Step S105: Based on the current attitude of the wind turbine, the improved M-RRT algorithm with fused direction heuristic factors is used to generate a global inspection path.

[0120] Specifically, the step of generating a global inspection path based on the current attitude of the wind turbine using an improved M-RRT algorithm that incorporates directional heuristic factors includes:

[0121] Environmental perception and attitude modeling: High-precision IMU and lidar are used to acquire the current three-dimensional attitude of the wind turbine and the spatial position of the blades in real time, and to construct an obstacle point cloud map including the tower and blade surfaces;

[0122] Directional heuristic factor fusion: Introducing a directional weighting function into traditional RRT random sampling :

[0123] ,

[0124] in, This is the directional gain coefficient. Let be the direction vector from the starting point to the target point. Let be the direction vector from the starting point to the random point. This represents the magnitude between the target point and the random point;

[0125] The search is guided to focus on key detection areas of the leaf, including the leaf root and leading edge areas, reducing invalid expansion nodes;

[0126] Dynamic step size adjustment: The step size is adaptively adjusted according to the blade curvature, increasing the step size in flat areas and decreasing the step size in curved areas.

[0127] Specifically, a high-precision IMU (Inertial Measurement Unit) and LiDAR are used to acquire the three-dimensional attitude of the wind turbine and the spatial position of its blades in real time. These sensors are used to construct an obstacle point cloud map that includes key structures such as the tower and blade surfaces. This map provides accurate environmental information for subsequent path planning.

[0128] The Randomized Tracking (RRT) algorithm is a path planning algorithm based on a tree structure. Its core idea lies in using a strategy combining random sampling and tree structure expansion to explore and find feasible paths. The principle of the RRT algorithm is as follows: Figure 2 As shown. First, initialization is performed, setting the starting point. This is the root node of tree T. Next, random sampling is used to generate... Next, in tree T, the distance between nodes is calculated using the Euclidean distance metric, and the closest node is selected from the results. Finally, along arrive Directional expansion with fixed step size generation And perform collision detection. If In free space and with If a connection is found without collision, add it to tree T. Repeat this process until the target point is found. Or, by reaching the maximum number of iterations, a collision-free path can eventually be obtained.

[0129] In the random sampling process of the traditional RRT (Fast Random Tree) algorithm, a directional weight function is introduced to guide the search towards key detection regions of the leaf, such as the leaf root and leading edge. This directional weight function takes the form:

[0130] ,

[0131] in, This is the directional gain coefficient. Let be the direction vector from the starting point to the target point. Let be the direction vector from the starting point to the random point. This represents the magnitude between the target point and the random point.

[0132] The step size is adaptively adjusted based on the blade curvature. In flat areas, the step size can be increased to improve search efficiency; while in curved areas, the step size is decreased to ensure path accuracy and safety. This adaptive adjustment mechanism helps generate better global paths in complex terrain.

[0133] Combining the aforementioned directional heuristic factor and dynamic step size adjustment strategy, the improved M-RRT algorithm can generate a globally optimal path from the starting point to the target point in complex environments. This path not only considers the distribution of environmental obstacles but also incorporates the detection requirements of critical blade areas, thereby improving inspection efficiency and safety. Through the above steps, step S105 realizes global inspection path planning based on the current attitude of the wind turbine, providing efficient and accurate path support for the automated inspection of UAVs.

[0134] Step S106: Based on the predicted shutdown attitude of the wind turbine, local trajectory optimization is performed using the improved APF algorithm with dynamic weight allocation.

[0135] Specifically, the method of predicting the shutdown attitude of the wind turbine and optimizing the local trajectory using an improved APF algorithm with dynamic weight allocation includes:

[0136] Attitude prediction and constraint modeling: Combining wind turbine operating data, including rotational speed, wind speed, and historical shutdown attitude, the blade swing range under shutdown conditions is predicted through an LSTM network, and the dynamic no-go zone of the maximum blade yaw envelope is defined.

[0137] The repulsive field is reconstructed, and the repulsive force intensity of the restricted area is adjusted in real time. The gravitational force generated by the repulsive field is... :

[0138] ,

[0139] in, For repulsive force weight, , The time decay factor, For resource deviation, , These are the corresponding deviation coefficients. The repulsion coefficient is... The distance from the current point to the target obstacle. The repulsive force intensity of the restricted area is adjusted in real time based on the distance between the starting point and the target obstacle.

[0140] Gravity-repulsion weight allocation: Increase the repulsion weight when approaching the high-risk area near the blade tip, and strengthen the gravity field in the safe path section to balance obstacle avoidance efficiency and path smoothness;

[0141] Determine whether a collision has occurred with an obstacle;

[0142] If so, then select a new parent node and rewire.

[0143] Otherwise, an obstacle avoidance trajectory path is formed, and the trajectory path is trimmed and optimized.

[0144] Specifically, by combining wind turbine operating data (such as rotational speed and wind speed) with historical shutdown attitudes, an LSTM network is used to predict the blade oscillation range during shutdown. This prediction result is used to delineate the dynamic no-go zone of the maximum blade yaw envelope, thereby providing constraints for subsequent path planning.

[0145] Based on the predicted halting attitude, the repulsive field is reconstructed, and the repulsive force intensity of the restricted area is adjusted in real time. The gravitational function of the repulsive field is defined as: This function allows the repulsive field to be dynamically adjusted according to the real-time environment to ensure that the path avoids high-risk areas.

[0146] When approaching high-risk areas near the blade tip, the repulsive force weight is increased to enhance obstacle avoidance capability; in safe path segments, the gravitational field is strengthened to guide the path towards the target point. This weight allocation strategy achieves a balance between obstacle avoidance efficiency and path smoothness. During path planning, collisions with obstacles are detected in real time. If a collision occurs, a new parent node is selected and the wiring is rewired to generate a new obstacle avoidance path. If no collision occurs, an obstacle avoidance trajectory is formed, and the path is optimized using a pruning algorithm to improve its smoothness and feasibility. Through the above steps, step S106 achieves local trajectory optimization based on the predicted shutdown attitude of the wind turbine. Combined with the improved APF algorithm, it can effectively cope with obstacle avoidance requirements in complex environments and improve the intelligence and safety of UAV inspection paths.

[0147] Step S107: Trigger online replanning based on environmental changes, mission requirements, and equipment status. Step S108: Select the optimal trajectory based on multi-objective optimization considering energy consumption, time, and safety factors.

[0148] Specifically, online replanning trigger conditions are set, including: triggering by wind turbine attitude change detection based on image recognition, triggering by path tracking error exceeding a threshold, and triggering by environmental obstacles intruding into the safe area;

[0149] A multi-objective optimization evaluation module is established, including: establishing a multi-objective function that includes path length, safety, energy consumption, and shooting angle; using the NSGA-III algorithm to select the Pareto optimal solution; wherein, in the Pareto optimal solution set generated by the NSGA-III algorithm, the optimal solution is selected as the final replanning path according to the decision-maker's preference or the weight of the expert system.

[0150] Specifically, online replanning trigger conditions are set, including: Image recognition-based wind turbine attitude change detection: The wind turbine's shutdown attitude is monitored in real time using image recognition technology. If an attitude change (such as an abnormal change in blade angle) is detected, replanning is triggered. Path tracking error exceeding a threshold: If the UAV deviates from the preset flight path by more than a set threshold during flight, replanning is triggered. Environmental obstacle intrusion into the safe zone: If an obstacle (such as blades, towers, etc.) is detected entering the UAV's safe flight zone, replanning is triggered.

[0151] After online replanning, the optimal trajectory needs to be selected based on multi-objective optimization. This includes: establishing a multi-objective function: constructing a multi-objective function that includes objectives such as path length, safety, energy consumption, and camera angle. Path length reflects the economic efficiency of the trajectory; safety reflects obstacle avoidance capability; energy consumption reflects flight efficiency; and camera angle reflects mission completion quality.

[0152] Pareto optimal solution selection using the NSGA-III algorithm: The NSGA-III algorithm is used to optimize the multi-objective function and generate a Pareto optimal solution set, which contains multiple solutions that weigh different objectives.

[0153] Selecting the optimal solution as the final path: From the Pareto optimal solution set, the optimal solution is selected as the final replanning path based on the decision-maker's preferences or the weights of the expert system. For example, if safety is the priority of the task, the path with the highest safety is selected; if energy consumption is the priority, the path with the lowest energy consumption is selected.

[0154] Through the above steps, step S107 realizes online replanning based on environmental changes, mission requirements and equipment status, and combines multi-objective optimization methods to provide the UAV with the optimal trajectory selection, thereby improving the efficiency and safety of flight missions.

[0155] Example 2

[0156] like Figure 3 As shown, this application provides an optimal trajectory planning system for unmanned aerial vehicles (UAVs) that integrates image recognition with M-RRT and APF algorithms. It is applied to the optimal trajectory planning method for UAVs that integrates image recognition with M-RRT and APF algorithms as described in Embodiment 1. The system includes: a multimodal data acquisition module 11, a geometric mapping module 12, an attitude recognition module 13, an attitude prediction module 14, a path planning module 15, a path optimization module 16, a dynamic replanning module 17, and a multi-objective optimization module 18.

[0157] It is understood that in this embodiment, the multimodal data acquisition module 11 is used to acquire three-dimensional point cloud data and visual information of the wind turbine in real time through a multimodal sensor array.

[0158] It is understood that in this embodiment, the geometric mapping module 12 is used to identify key components of the wind turbine based on a deep learning model and estimate the shutdown attitude parameters, and to establish a geometric mapping model between the spatial position of the wind turbine blades and the images taken by the UAV.

[0159] It is understood that in this embodiment, the attitude recognition module 13 is used to design a key point detection model to identify the blade tip, use DBSCAN clustering to filter out erroneous blade tips, and automatically measure the current shutdown attitude of the wind turbine.

[0160] It is understood that in this embodiment, the attitude prediction module 14 is used to input the current shutdown attitude into the geometric mapping model to obtain the predicted shutdown attitude of the wind turbine at the next moment.

[0161] It is understood that in this embodiment, the path planning module 15 is used to generate a global inspection path based on the current attitude of the wind turbine and using an improved M-RRT algorithm that incorporates directional heuristic factors.

[0162] It is understood that in this embodiment, the path optimization module 16 is used to optimize the local trajectory based on the predicted shutdown attitude of the wind turbine and the improved APF algorithm with dynamic weight allocation.

[0163] It is understood that in this embodiment, the dynamic replanning module 17 is used to trigger online replanning based on environmental changes, task requirements, and equipment status.

[0164] It is understood that in this embodiment, the multi-objective optimization module 18 is used to select the optimal trajectory based on multi-objective optimization of energy consumption, time, and safety factor.

[0165] Figure 4 This is an electronic device provided in one embodiment of this application. For example... Figure 4 As shown, the electronic device includes at least the following components: processor 101 and memory 100, communication interface 103, and bus 102.

[0166] In this embodiment of the application, memory 100 is used to store executable instructions of processor 101, which, when configured to execute instructions, implements the method as described in the first aspect.

[0167] In embodiments of this application, a computer-readable storage medium includes instructions that instruct a device to perform the method as described in the first aspect. For example, the instructions instruct the device to perform... Figure 1 The method is shown in the process steps.

[0168] In one embodiment of this application, the program operating in the electronic device may be a program that controls a central processing unit (CPU) or similar device to achieve the functions of the above-described embodiments of the present invention (a program that enables the computer to function). Information processed by these devices is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (FlashROM) and hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0169] It should be noted that a portion of the electronic device described above can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0170] It should be noted that the term "computer" as used here refers to a computer built into an electronic device, employing hardware including an operating system and peripheral devices. Furthermore, "computer-readable recording media" refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard drives built into a computer.

[0171] Furthermore, a "computer-readable recording medium" can include: a medium that dynamically stores a program for a short period of time, such as a communication line used when transmitting a program via a network such as the Internet or a communication line such as a telephone line; or a medium that stores a program for a fixed period of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the aforementioned program can be a program used to implement the above-mentioned functions, or it can be a program that can implement the above-mentioned functions by combining with programs already recorded in the computer.

[0172] Furthermore, the electronic device in the above embodiments can also be implemented as an assembly (device group) composed of multiple devices. Each device constituting the device group can possess some or all of the functions or functional blocks of the electronic device in the above embodiments. As a device group, it is sufficient to have all the functions or functional blocks of the electronic device.

[0173] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. An unmanned aerial vehicle optimal path planning method fusing image recognition with M-RRT and APF algorithms, characterized in that, The method comprises: Real-time acquisition of wind turbine three-dimensional point cloud data and visual information through a multi-modal sensor array; Identification of key components of the wind turbine and estimation of the shutdown attitude parameters based on a deep learning model, establishment of a geometric mapping model of the spatial position of the wind turbine blade and the image taken by the unmanned aerial vehicle; Design of a key point detection model to identify the blade tip point, filtering of false blade tip points using DBSCAN clustering, and automatic measurement of the current shutdown attitude of the wind turbine; Input of the current shutdown attitude into the geometric mapping model to obtain the predicted shutdown attitude of the wind turbine at the next time; Generation of a global inspection path based on the current attitude of the wind turbine using an improved M-RRT algorithm with a fused direction heuristic factor; Local trajectory optimization based on the predicted shutdown attitude of the wind turbine using an improved APF algorithm with dynamic weight distribution; Online re-planning triggered according to environmental changes, task requirements, and equipment states; Optimal trajectory selection based on multi-objective optimization of energy consumption, time, and safety factor. 2.The optimal path planning method of UAV based on fusion of image recognition, M-RRT and APF algorithm according to claim 1, characterized in that, The identification of key components of the wind turbine and the estimation of the shutdown attitude parameters based on the deep learning model, and the establishment of the geometric mapping model of the spatial position of the wind turbine blade and the image taken by the unmanned aerial vehicle, comprise: Detection and positioning of the wind turbine blade through training of the deep learning model; Extraction of blade edge information through Canny edge detection and Hough transform; Calculation of the pixel coordinates of the blade tip through coordinate system transformation based on the blade tip pixel coordinates and the unmanned aerial vehicle attitude information; Calculation of the GPS coordinates of the blade tip and determination of the rotation angle of the blade in combination with the attitude information of the unmanned aerial vehicle; Construction of the geometric mapping model of the spatial position of the wind turbine blade and the image taken by the unmanned aerial vehicle based on the mapping relationship between the rotation angle and the yaw angle at different time points according to the motion trend of the blade and environmental changes.

3. The unmanned aerial vehicle optimal path planning method of fusing image recognition with M-RRT and APF algorithm according to claim 2, characterized in that, The design of the key point detection model to identify the blade tip point, the filtering of false blade tip points using DBSCAN clustering, and the automatic measurement of the current shutdown attitude of the wind turbine, comprise: Identification of the blade tip pixel based on the deep learning model; Correction of the blade tip pixel through Hough straight line and identification of the points in the low-density area through the DBSCAN clustering algorithm to eliminate data points deviating from the abnormal angle; Determination of the blade inclination parameter based on the reference relationship between the blade tip point and the coordinate axes through analysis of the distribution of the blade tip point; Automatic measurement of the current shutdown attitude of the wind turbine.

4. The unmanned aerial vehicle optimal path planning method of claim 3, wherein, The input of the current shutdown attitude into the geometric mapping model to obtain the predicted shutdown attitude of the wind turbine at the next time, comprise: Acquisition of the arrival time of the next wind turbine; Input of the arrival time and the current shutdown attitude into the geometric mapping model to predict the shutdown attitude at the next time; Taking the shutdown attitude at the next time as the predicted shutdown attitude of the next wind turbine.

5. The unmanned aerial vehicle optimal path planning method of claim 4, wherein, The generation of a global inspection path based on the current attitude of the wind turbine using an improved M-RRT algorithm with a fused direction heuristic factor, comprise: Environmental perception and attitude modeling: real-time acquisition of the current three-dimensional attitude and blade spatial position of the wind turbine using high-precision IMU and laser radar, and construction of an obstacle point cloud map containing the tower cylinder and blade surface; Directional heuristic fusion: introducing a directional weight function in traditional RRT random sampling : ， wherein, is a directional gain coefficient, is a start point to target point directional vector, is a start point to random point directional vector, denotes a module length between the target point and the random point; The guiding search is inclined to the blade key detection area, including the blade root and the leading edge area, and the invalid expansion nodes are reduced. Dynamic step adjustment: adaptively adjust the expansion step according to the blade curvature, increase the step in the flat area, and reduce the step in the curved area.

6. The unmanned aerial vehicle optimal path planning method of claim 5, wherein, The improved APF algorithm with dynamic weight distribution is used for local trajectory optimization based on the predicted shutdown posture of the wind turbine, including: Posture prediction and constraint modeling: combining the wind turbine operation data, including speed, wind speed, and historical shutdown posture, the blade swing range in the shutdown state is predicted through the LSTM network, and the dynamic forbidden area of the maximum blade swing envelope is drawn; The repulsive force field is reconstructed, and the repulsive force intensity of the forbidden area is adjusted in real time, wherein the attraction generated by the repulsive force field is : ， wherein, is a repulsive force weight, , is a time decay factor, is a state bias, , are respective bias coefficients, is a repulsive force coefficient, is a distance between the current point and the target obstacle, is a distance between the start point and the target obstacle, and the repulsive force strength of the forbidden zone is adjusted in real time. Attractive-repulsive weight distribution: increase the repulsive weight when close to the high-risk area of the blade tip, and strengthen the attractive field in the safe path segment to balance the obstacle avoidance efficiency and path smoothness; Judge whether the obstacle collides or not; If yes, reselect the parent node and rewire; Otherwise, form an obstacle avoidance trajectory path and optimize the trajectory path.

7. The unmanned aerial vehicle optimal path planning method of claim 6, wherein, Trigger online re-planning according to environmental changes, task requirements and equipment states; Multi-objective optimization is used to select the optimal trajectory based on energy consumption, time and safety factor, including: Set the online re-planning trigger condition, including: wind turbine posture mutation detection trigger based on image recognition, path tracking error exceeding threshold trigger, and environmental obstacle intrusion into safe area trigger; Establish a multi-objective optimization evaluation module, including: establish a multi-objective function including path length, safety, energy consumption and shooting angle, and select the optimal solution as the final re-planning path by using NSGA-III algorithm to generate Pareto optimal solution set.

8. An unmanned aerial vehicle optimal path planning system fusing image recognition with M-RRT and APF algorithm, applied to the unmanned aerial vehicle optimal path planning method fusing image recognition with M-RRT and APF algorithm according to any one of claims 1 to 7, characterized in that, It includes: A multi-modal data acquisition module is used to acquire wind turbine three-dimensional point cloud data and visual information in real time through a multi-modal sensor array; A geometric mapping module is used to identify key components of the wind turbine based on a deep learning model and estimate the shutdown posture parameters, and a geometric mapping model of the wind turbine blade space position and the unmanned aerial vehicle image is established; An attitude recognition module is used to design a key point detection model to identify the blade tip point, filter the wrong blade tip point by using DBSCAN clustering, and automatically measure the current shutdown posture of the wind turbine; An attitude prediction module is used to input the current shutdown posture into the geometric mapping model to obtain the predicted shutdown posture of the wind turbine at the next time; A path planning module is used to generate a global inspection path based on the current posture of the wind turbine by using an improved M-RRT algorithm with fused direction heuristic factors; A path optimization module is used to perform local trajectory optimization based on the predicted shutdown posture of the wind turbine by using an improved APF algorithm with dynamic weight distribution; A dynamic re-planning module is used to trigger online re-planning according to environmental changes, task requirements and equipment states; A multi-objective optimization module is used to select the optimal trajectory based on multi-objective optimization of energy consumption, time and safety factor.

9. An electronic device, comprising: It includes: A processor; A memory for storing processor-executable instructions; The processor is configured to implement the method of claim 1 to 7 when executing the instructions.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program instructing the device to implement the method of claim 1 to 7.

Images