Unmanned aerial vehicle optimal route planning method fusing image recognition, M-RRT and APF algorithms and digital intelligence system

Abstract

The invention provides an unmanned aerial vehicle optimal route planning method and system fusing image recognition and M-RRT and APF algorithms. The point cloud and visual data of a fan are obtained in real time through a multi-mode sensor, key parts of blades are recognized based on deep learning, a geometric mapping model is constructed, the blade tip points are corrected in a clustering mode in combination with DBSCAN, and the shutdown posture is predicted. A global path adopts an M-RRT algorithm improved by a direction heuristic factor to guide and search a key area of a blade; a local track is optimized through a dynamic weight APF algorithm, and a repulsive force field is adjusted in real time to cope with attitude changes. An online re-planning mechanism is introduced, NSGA-III multi-target optimization is triggered when the environment suddenly changes or the tracking error exceeds a threshold value, and the optimal track is generated by integrating energy consumption, time and safety. The system integrates high-precision sensing, self-adaptive planning and dynamic optimization, and the inspection coverage rate and the track safety under the complex shutdown attitude are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous navigation and intelligent control of unmanned aerial vehicles (UAVs), and specifically relates to a method and system for optimal trajectory planning of UAVs that integrates image recognition with an improved rapidly expanding random tree (M-RRT) and artificial potential field (APF) algorithm. The method and system are particularly suitable for autonomous inspection operations in complex environments where the shutdown posture of wind turbines is uncertain. Background Art

[0002] With the rapid growth of global wind power installed capacity, regular inspection and maintenance of wind turbines has become a critical component in ensuring the safe and efficient operation of wind farms. Traditional manual inspection methods have inherent drawbacks such as low efficiency, high cost, and high risk. However, drone inspection technology, with its advantages of flexibility, low cost, and wide viewing angle, is gradually becoming a mainstream solution for wind turbine inspection.

[0003] Current drone inspections of wind turbines primarily utilize automated flight modes with preset flight paths. By pre-entering the geometric parameters and spatial position of the wind turbine, a fixed inspection route is planned. This method effectively completes inspection tasks when the wind turbine is in a standard shutdown posture (with blades arranged in a "Y" or "S" shape). However, in actual operation, when wind turbines are shut down for reasons such as fault maintenance, grid dispatch, or inclement weather, their blades often exhibit various non-standard postures, including: arbitrary yaw angles of one or more blades, abnormal blade pitch angles, changes in blade torsion, and random deviations in the nacelle's orientation.

[0004] However, the fixed trajectory planning methods proposed in existing studies cannot cope with the dynamic changes in wind turbine attitude. The main manifestations are: (1) The adaptive defects of fixed trajectory planning, such as the preset safety margin may be broken by the abnormal posture of the blade, resulting in collision risks; key detection points (such as blade bolts, lightning strike points) may be out of the camera field of view due to attitude changes; sudden gusts of wind may cause the actual blade position to deviate from the expected trajectory; measured data show that when the blade yaw angle exceeds ±30°, the detection completeness rate of the traditional method drops by more than 40%. (2) The limitations of the path planning algorithm, such as the rapidly expanding random tree (RRT) algorithm: the traditional RRT (such as that described in CN109917826A) has a slow convergence speed in three-dimensional space, the planning time often exceeds 500ms, the generated path has redundant nodes, poor smoothness, and does not meet the dynamic constraints of the drone; the response to dynamic obstacles is delayed, and the re-planning efficiency is low. In terms of artificial potential field (APF) algorithms: Traditional APFs are prone to falling into local minima in areas with dense wind turbines. The potential field parameters are fixed and cannot adapt to the scale differences of wind turbines of different capacities. The processing effect on fast-moving blade tips is poor, which may cause path oscillation. (3) Insufficient environmental perception and modeling. Current modeling technology relies on pre-input static wind turbine models and lacks real-time posture perception capabilities. The dynamic update frequency of the blade sweep area is insufficient, and the impact of wind speed on the actual position of the blade is not considered. The feature extraction of different types of wind turbines lacks generalization. (4) System integration defects. Current solutions are mostly composed of independent modules. There are problems such as: the delay of each link of perception-planning-control is not synchronized; the emergency response mechanism is single, lacks multi-objective optimization for inspection quality, and is difficult to support multi-machine collaborative inspection.

[0005] The core contradictions facing current technological development lie in the following: the uncertainty of wind turbine attitude and the deterministic requirements of path planning; the contradiction between algorithm computational complexity and system real-time requirements; the contradiction between local obstacle avoidance safety and global inspection integrity; and the contradiction between algorithms dedicated to a single aircraft model and compatibility with multiple wind farm models. These contradictions lead to widespread problems in existing drone inspection systems when responding to sudden changes in wind turbine attitude, such as delayed response, suboptimal paths, and limited adaptability. The development of a new generation of intelligent planning methods is urgently needed. Summary of the Invention

[0006] In view of this, it is necessary to provide a UAV optimal trajectory planning method and system that integrates image recognition with M-RRT and APF algorithms, aiming to solve the real-time trajectory planning problem of UAV autonomous inspection when the wind turbine is shut down and the posture is uncertain, and to overcome the defects of existing technologies such as poor adaptability, low planning efficiency, and uneven path.

[0007] In a first aspect, an embodiment of the present application provides a method for optimal trajectory planning of a UAV by integrating image recognition with M-RRT and APF algorithms, the method comprising:

[0008] Acquire wind turbine 3D point cloud data and visual information in real time through a multimodal sensor array;

[0009] Using deep learning models to identify key wind turbine components and estimate shutdown posture parameters, a geometric mapping model was established between the spatial position of wind turbine blades and images taken by drones.

[0010] A key point detection model is designed to identify blade tips, and DBSCAN clustering is used to filter out incorrect blade tips, automatically measuring the current shutdown posture of the wind turbine.

[0011] Inputting the current shutdown posture into the geometric mapping model to obtain a predicted shutdown posture of the wind turbine at the next moment;

[0012] Based on the current posture of the wind turbine, the improved M-RRT algorithm integrating the directional heuristic factor is used to generate the global inspection path;

[0013] Based on the predicted shutdown posture of the wind turbine, the improved APF algorithm with dynamic weight allocation is used to perform local trajectory optimization;

[0014] Trigger online re-planning based on environmental changes, mission requirements, and equipment status;

[0015] The optimal trajectory is selected based on multi-objective optimization of energy consumption, time and safety factor.

[0016] Optionally, in an implementation of the first aspect of the present invention, identifying key components of a wind turbine and estimating shutdown posture parameters based on a deep learning model, and establishing a geometric mapping model between the spatial positions of wind turbine blades and images captured by a drone, include:

[0017] Detect and locate wind turbine blades by training deep learning models;

[0018] Extract blade edge information through Canny edge detection and Hough transform;

[0019] According to the pixel coordinates of the blade tip and the UAV posture information, the pixel coordinates of the blade tip are calculated through coordinate system transformation;

[0020] Combined with the UAV's position information, the GPS coordinates of the blade tip are calculated to determine the blade's rotation angle;

[0021] According to the mapping relationship between the blade's motion trend, environmental changes, and the rotation angle and yaw angle at different time points, a geometric mapping model between the spatial position of the wind turbine blade and the image taken by the drone is constructed.

[0022] Optionally, in an implementation of the first aspect of the present invention, the design key point detection model identifies blade tip points, uses DBSCAN clustering to filter out erroneous blade tip points, and automatically measures the current shutdown posture of the wind turbine, including:

[0023] Identify blade tip pixels based on a deep learning model;

[0024] The blade tip pixels are corrected using Hough straight lines, and points in low-density areas are identified using the DBSCAN clustering algorithm to eliminate data points that deviate from abnormal angles;

[0025] By analyzing the distribution of blade tip points and according to the reference relationship between the blade tip points and the coordinate axes in the coordinate system, the blade inclination parameters are determined.

[0026] Automatically measure the current shutdown posture of the wind turbine.

[0027] Optionally, in an implementation of the first aspect of the present invention, inputting the current shutdown posture into the geometric mapping model to obtain a predicted shutdown posture of the wind turbine at a next moment includes:

[0028] Get the arrival time to the next wind turbine;

[0029] Inputting the arrival time and the current parking posture into the geometric mapping model to predict the parking posture at the next moment;

[0030] The shutdown posture at the next moment is used as the predicted shutdown posture of the next wind turbine.

[0031] Optionally, in an implementation of the first aspect of the present invention, generating a global inspection path based on the current posture of the wind turbine using an improved M-RRT algorithm that incorporates a directional heuristic factor includes:

[0032] Environmental perception and attitude modeling: High-precision IMU and LiDAR are used to obtain the current 3D attitude of the wind turbine and the spatial position of the blades in real time, and to construct a point cloud map of obstacles including the tower and blade surfaces;

[0033] Directional heuristic factor fusion: Introducing the directional weight function w into the traditional RRT random sampling d :

[0034]

[0035] Where α is the directional gain coefficient, is the direction vector from the starting point to the target point, is the direction vector from the starting point to the random point, Represents the modulus length between the target point and the random point;

[0036] Guide the search towards the key detection areas of the blade, including the root and leading edge areas, to reduce invalid expansion nodes;

[0037] Dynamic step size adjustment: Adaptively adjust the extension step size according to the blade curvature, increasing the step size in flat areas and reducing the step size in curved areas.

[0038] Optionally, in an implementation of the first aspect of the present invention, the local trajectory optimization based on the predicted shutdown posture of the wind turbine using an improved APF algorithm with dynamic weight allocation includes:

[0039] Attitude prediction and constraint modeling: Combining wind turbine operating data, including rotational speed and wind speed, with historical shutdown attitudes, the LSTM network is used to predict the blade swing range in the shutdown state and define the dynamic restricted area of ​​the maximum blade deflection envelope.

[0040] Reconstruct the repulsive field and adjust the repulsive strength of the restricted area in real time. The gravitational force generated by the repulsive field is F rep :

[0041]

[0042] Among them, w t is the repulsive force weight, w t =k1·t+k2·Δθ, where t is the time attenuation factor, Δθ is the state deviation, k1 and k2 are the corresponding deviation coefficients, η is the repulsion coefficient, d is the distance between the current point and the target obstacle, and d0 is the distance between the starting point and the target obstacle. The repulsion strength of the restricted area is adjusted in real time;

[0043] Gravity-repulsion weight distribution: Increase the repulsion weight when approaching the high-risk area of ​​the blade tip, strengthen the gravitational field in the safe path section, and balance obstacle avoidance efficiency and path smoothness;

[0044] Determine whether a collision with an obstacle occurs;

[0045] If yes, reselect the parent node and rewire;

[0046] Otherwise, an obstacle avoidance trajectory path is formed and pruned to optimize the trajectory path.

[0047] Optionally, in an implementation of the first aspect of the present invention, online replanning is triggered according to environmental changes, mission requirements, and equipment status; and the optimal trajectory is selected based on multi-objective optimization of energy consumption, time, and safety factor, including:

[0048] Set online replanning trigger conditions, including: wind turbine attitude sudden change detection based on image recognition, path tracking error exceeding a threshold, and environmental obstacles invading the safety zone;

[0049] A multi-objective optimization evaluation module was established, including: establishing a multi-objective function including path length, safety, energy consumption, and shooting angle, and using the NSGA-III algorithm to select the Pareto optimal solution. Among the Pareto optimal solutions generated by the NSGA-III algorithm, the optimal solution was selected as the final replanning path based on the decision maker's preference or the weight of the expert system.

[0050] In a second aspect, an embodiment of the present application provides an optimal trajectory planning system for a UAV that integrates image recognition with M-RRT and APF algorithms, which is applied to the optimal trajectory planning method for a UAV that integrates image recognition with M-RRT and APF algorithms as described in the first aspect, including:

[0051] Multimodal data acquisition module, used to acquire wind turbine 3D point cloud data and visual information in real time through a multimodal sensor array;

[0052] A geometric mapping module is used to identify key components of wind turbines and estimate shutdown posture parameters based on a deep learning model, and to establish a geometric mapping model between the spatial position of wind turbine blades and images taken by drones;

[0053] The posture recognition module is used to design a key point detection model to identify the blade tip, use DBSCAN clustering to filter out incorrect blade tip points, and automatically measure the current shutdown posture of the wind turbine;

[0054] a posture prediction module, configured to input the current shutdown posture into the geometric mapping model to obtain a predicted shutdown posture of the wind turbine at the next moment;

[0055] The path planning module is used to generate a global inspection path based on the current posture of the wind turbine using an improved M-RRT algorithm that incorporates directional heuristic factors;

[0056] Path optimization module, which is used to perform local trajectory optimization based on the predicted shutdown posture of wind turbines using the improved APF algorithm with dynamic weight allocation;

[0057] Dynamic replanning module, used to trigger online replanning based on environmental changes, task requirements, and equipment status;

[0058] The multi-objective optimization module is used to select the optimal trajectory based on multi-objective optimization of energy consumption, time, and safety factor.

[0059] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0060] processor;

[0061] a memory for storing processor-executable instructions;

[0062] Wherein, the processor is configured to implement the optimal trajectory planning method for a UAV by integrating image recognition with M-RRT and APF algorithms as described in the first aspect when executing the instructions.

[0063] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a program, wherein the program instructs a device to execute the optimal trajectory planning method for a drone that integrates image recognition with M-RRT and APF algorithms as described in the first aspect.

[0064] The present invention proposes a method and system for optimal trajectory planning of unmanned aerial vehicles (UAVs) that integrates image recognition with M-RRT and APF algorithms. Wind turbine point cloud and visual data are acquired in real time through multimodal sensors. Key blade components are identified based on deep learning and a geometric mapping model is constructed. The blade tip is corrected using DBSCAN clustering to predict the shutdown posture. The global path uses an M-RRT algorithm improved with a directional heuristic factor to guide the search for key areas of the blade. Local trajectories are optimized using a dynamic weighted APF algorithm, which adjusts the repulsive field in real time to cope with posture changes. An online replanning mechanism is introduced. When the environment suddenly changes or the tracking error exceeds a threshold, the NSGA-III multi-objective optimization is triggered to generate the optimal trajectory based on energy consumption, time, and safety. The system integrates high-precision perception, adaptive planning, and dynamic optimization, significantly improving inspection coverage and trajectory safety under complex shutdown postures.

[0065] Beneficial effects:

[0066] (1) High-precision perception and modeling: Combining multimodal sensors with deep learning, accurately identifying key components of wind turbines and predicting shutdown postures, thereby improving the reliability of environmental modeling.

[0067] (2) Efficient global planning: The improved M-RRT algorithm significantly reduces invalid searches and improves the coverage efficiency of key areas of the blade (such as the root and leading edge) through directional heuristic factors and dynamic step size adjustment.

[0068] (3) Dynamic obstacle avoidance optimization: A dynamic weight allocation strategy based on the APF algorithm responds to blade swing in real time, balances path smoothness and obstacle avoidance capability, and reduces collision risk.

[0069] (4) Adaptive replanning: Through multi-objective optimization (energy consumption, time, safety) and online replanning mechanism, the trajectory is ensured to maintain optimal performance when the environment changes suddenly or the equipment is abnormal.

[0070] (5) Engineering practicality: The system integrates sensor fusion, real-time computing, and lightweight deployment, making it suitable for wind turbine inspections under complex shutdown conditions, while balancing efficiency and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1A flowchart of a method for optimal trajectory planning for a UAV that integrates image recognition with M-RRT and APF algorithms is provided in one embodiment of the present application.

[0072] Figure 2 A schematic diagram of the RRT algorithm principle provided in one embodiment of the present application.

[0073] Figure 3 A schematic diagram of the optimal trajectory planning system module for a drone that integrates image recognition, M-RRT, and APF algorithms, provided in one embodiment of the present application.

[0074] Figure 4 A schematic diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0075] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0076] It should be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art in the art to which this application relates. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0077] It should be noted that, in the embodiments of the present application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order. Features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0078] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0079] Example 1

[0080] Figure 1A flowchart of a method for optimal trajectory planning for a UAV that integrates image recognition with M-RRT and APF algorithms is provided in one embodiment of the present application.

[0081] like Figure 1 As shown in FIG, a method for optimal trajectory planning of a UAV that integrates image recognition with M-RRT and APF algorithms includes:

[0082] Step S101 : Acquire wind turbine three-dimensional point cloud data and visual information in real time through a multimodal sensor array.

[0083] Acquiring real-time 3D point cloud data and visual information from wind turbines through a multimodal sensor array is a key technology in wind turbine equipment monitoring and maintenance. This step aims to achieve high-precision, efficient 3D modeling and real-time monitoring of wind turbine structures through the coordinated operation of multiple sensors, providing data support for subsequent deformation monitoring, defect identification, and fault warning.

[0084] Specifically, the multimodal sensor array usually includes the following sensors: Laser Ranging and Detecting (LiDAR): used to obtain high-precision three-dimensional point cloud data. By emitting a laser beam and measuring its reflection time, the LiDAR can quickly and continuously obtain three-dimensional point cloud data of the target object. High-definition camera: used to obtain visual information of the wind turbine blades. The high-definition camera can capture the surface image of the wind turbine blades to provide support for subsequent defect identification and texture analysis. Millimeter-wave radar: used to assist in obtaining three-dimensional point cloud data of the wind turbine blades. Millimeter-wave radar has strong penetration ability in complex environments and can provide relatively stable point cloud data. Six-dimensional force sensor: used to monitor changes in contact force between the wind turbine blades and the end effector, and provide feedback for the operation of the robotic arm or robot. Temperature sensor: used to monitor temperature changes in the wind turbine blades and provide data support for the temperature compensation model.

[0085] 3D point cloud data is acquired by scanning wind turbine blades using sensors such as LiDAR, recording the 3D coordinate information of their surfaces. Processing 3D point cloud data involves the following steps: Point cloud preprocessing: This includes operations such as denoising, homogenization, and registration to improve the quality of the point cloud data. Point cloud registration: This unifies point cloud data collected by different sensors into a common coordinate system to improve the accuracy of subsequent analysis. Feature extraction: This extracts key features from the point cloud data, such as blade outlines and defect areas.

[0086] Acquiring visual information primarily relies on devices such as high-definition cameras to capture surface images of wind turbine blades. Visual information processing involves the following steps: Image preprocessing: This includes operations such as denoising, enhancement, and segmentation to improve image clarity and contrast. Feature extraction: This extracts key features from the image, such as blade edges and defective areas. Defect recognition: This uses deep learning models to identify and classify defects in the image.

[0087] Multimodal data fusion combines data collected by sensors such as lidar, high-definition cameras, and six-dimensional force sensors to improve monitoring accuracy and real-time performance. This data can be uniformly identified using timestamps.

[0088] Step S102: Identify key components of the wind turbine based on the deep learning model and estimate shutdown posture parameters, and establish a geometric mapping model between the spatial position of the wind turbine blades and the image taken by the drone.

[0089] Specifically, the method of identifying key components of a wind turbine and estimating shutdown posture parameters based on a deep learning model, and establishing a geometric mapping model between the spatial position of wind turbine blades and images taken by a drone, includes:

[0090] Detect and locate wind turbine blades by training deep learning models;

[0091] Extract blade edge information through Canny edge detection and Hough transform;

[0092] According to the pixel coordinates of the blade tip and the UAV posture information, the pixel coordinates of the blade tip are calculated through coordinate system transformation;

[0093] Combined with the UAV's position information, the GPS coordinates of the blade tip are calculated to determine the blade's rotation angle;

[0094] According to the mapping relationship between the blade's motion trend, environmental changes, and the rotation angle and yaw angle at different time points, a geometric mapping model between the spatial position of the wind turbine blade and the image taken by the drone is constructed.

[0095] In this embodiment, deep learning algorithms (such as YOLOv5, YOLOv7, MaskR-CNN, etc.) can be used to detect and locate wind turbine blades. These models can automatically identify blades in images and extract their key features, such as blade tips, blade roots, and blade centers. Based on the deep learning model used to detect blades, traditional image processing techniques (such as Canny edge detection and Hough transform) can be combined to further extract the edge information of the blades. Canny edge detection can effectively extract the outline of the blades, while Hough transform can be used to detect the straight line features of the blades, thereby assisting in the positioning and posture estimation of the blades. These methods can still maintain high robustness in the presence of complex backgrounds or changes in illumination.

[0096] During flight, drones capture images of wind turbine blades, and deep learning models can be used to obtain the pixel coordinates of the blade tips. Simultaneously, the drone's pose information (such as GPS coordinates and attitude angles) provides information about the blade's position in three-dimensional space. Through coordinate system transformations (such as from the world coordinate system to the camera coordinate system), the pixel coordinates of the blade tip can be mapped into three-dimensional space, allowing the GPS coordinates of the blade tip to be calculated. This process typically involves camera calibration to determine internal and external parameters to ensure accurate mapping of the image to three-dimensional space.

[0097] The blade's rotation angle can be further calculated using the drone's position information (such as yaw and pitch angles). For example, by calculating the change in the blade tip's position within the image, the blade's rotation angle can be inferred. Furthermore, a mapping relationship between blade rotation angle and time can be established by combining the blade's motion trends and environmental changes. This method effectively estimates the blade's idle attitude parameters, providing data support for subsequent inspections and maintenance.

[0098] Finally, by analyzing the mapping relationship between blade motion trends, environmental changes, and rotation and yaw angles at different time points, a geometric mapping model can be constructed that correlates the spatial position of wind turbine blades with drone-captured images. This model can correlate the blade's spatial position information with feature points in drone images, enabling real-time monitoring of blade status and fault diagnosis. For example, blade trajectory grids and time-series shape analysis can further identify blade offsets and angle changes.

[0099] Step S102 achieves accurate identification of key components of wind turbine blades and estimation of shutdown posture parameters by combining deep learning models, traditional image processing technology and drone posture information, thereby establishing a geometric mapping model between the spatial position of wind turbine blades and images taken by drones.

[0100] Step S103 : Design a key point detection model to identify blade tips, use DBSCAN clustering to filter out erroneous blade tips, and automatically measure the current shutdown posture of the wind turbine.

[0101] Specifically, the design key point detection model identifies blade tip points, uses DBSCAN clustering to filter out erroneous blade tip points, and automatically measures the current shutdown posture of the wind turbine, including:

[0102] Identify blade tip pixels based on a deep learning model;

[0103] Correcting the blade tip pixels using Hough straight lines, identifying points in low-density areas using the DBSCAN clustering algorithm, and eliminating data points that deviate from abnormal angles;

[0104] By analyzing the distribution of blade tip points and according to the reference relationship between the blade tip points and the coordinate axes in the coordinate system, the blade inclination parameters are determined.

[0105] Automatically measure the current shutdown posture of the wind turbine.

[0106] Specifically, a deep learning model (such as an object detection algorithm) is used to identify the blades in the image and extract the pixel locations of the blade tips. Deep learning models are typically based on a trained detection model and trained using a labeled training dataset to accurately identify the blade tip locations. For example, training a custom dataset can extract the pixel coordinates of the blade tip. Using a deep learning object detection algorithm, the center point coordinates (x, y) and width and height (w, h) of the blade tip can be obtained to determine the blade tip location.

[0107] After identifying leaf tip pixels, the distribution of these points may be inaccurate due to image noise or detection errors. Therefore, the Hough transform is used to correct these leaf tip pixels to improve the accuracy of the leaf tip points. The Hough transform can detect straight line structures in the image, which helps correct the distribution of leaf tip points to make them more consistent with the actual geometry.

[0108] Based on the corrected blade tip points, the DBSCAN clustering algorithm is used to perform cluster analysis on the blade tip points. DBSCAN is a density-based clustering algorithm that can automatically identify low-density areas in a data set and treat points in these low-density areas as outliers. DBSCAN can discover clusters of any shape and can identify noise points, thereby effectively filtering out blade tip points that deviate from normal angles. For example, when the blade angle is greater than 2°, these data points are generally considered outliers and need to be eliminated. The DBSCAN algorithm can also define concepts such as the ε-neighborhood and density reachability of points, which can identify points in low-density areas and mark them as noise.

[0109] After filtering out abnormal blade tip points, the blade tip point distribution is analyzed and combined with the coordinate axis relationships in the coordinate system to determine the blade's inclination parameters. By analyzing the tip point distribution, the blade's inclination angle relative to the coordinate system can be determined, and thus the blade's inclination parameters can be calculated. By analyzing the tip point distribution, the blade's rotational direction and position can be determined, further allowing the blade's inclination angle to be calculated.

[0110] Finally, the wind turbine's current shutdown posture is automatically measured based on the blade pitch parameters obtained in the above steps. For example, by analyzing the blade tip distribution and pitch parameters, it is possible to determine whether the wind turbine is in a shutdown state and calculate its shutdown posture. By analyzing the blade tip distribution and pitch parameters, the wind turbine's operating status can be determined and corresponding control information can be provided.

[0111] Step S104 : inputting the current shutdown posture into the geometric mapping model to obtain a predicted shutdown posture of the wind turbine at the next moment.

[0112] Specifically, the step of inputting the current shutdown posture into the geometric mapping model to obtain the predicted shutdown posture of the wind turbine at the next moment includes:

[0113] Get the arrival time to the next wind turbine;

[0114] Inputting the arrival time and the current parking posture into the geometric mapping model to predict the parking posture at the next moment;

[0115] The shutdown posture at the next moment is used as the predicted shutdown posture of the next wind turbine.

[0116] Specifically, during wind turbine inspection or maintenance, it is necessary to determine the arrival time for the next wind turbine based on the mission plan and flight path. This time is typically calculated based on the drone's flight speed, range, and mission scheduling strategy.

[0117] The current wind turbine's shutdown attitude parameters (such as blade pitch and yaw angle) and the estimated time to reach the next wind turbine are input into the geometric mapping model. Based on visual SLAM technology, this model establishes a relationship between the observation model and the camera motion model by matching points and feature points, thereby predicting the next wind turbine's shutdown attitude. The geometric mapping model uses the input current shutdown attitude and arrival time, combined with the wind turbine's dynamic characteristics (such as blade rotation and wind speed changes), to predict the next shutdown attitude. This prediction can be used for drone path planning and attitude adjustment to improve inspection efficiency and safety.

[0118] The prediction results serve as the initial state of the next wind turbine and are used for subsequent shutdown posture analysis and control decisions. This process enables continuous tracking and prediction of wind turbine shutdown states, supporting automated operation and maintenance.

[0119] Step S105 : Based on the current posture of the wind turbine, a global inspection path is generated by adopting an improved M-RRT algorithm that incorporates directional heuristic factors.

[0120] Specifically, the global inspection path is generated based on the current posture of the wind turbine using the improved M-RRT algorithm that integrates the directional heuristic factor, including:

[0121] Environmental perception and attitude modeling: High-precision IMU and LiDAR are used to obtain the current 3D attitude of the wind turbine and the spatial position of the blades in real time, and to construct a point cloud map of obstacles including the tower and blade surfaces;

[0122] Directional heuristic factor fusion: Introducing the directional weight function w into the traditional RRT random sampling d :

[0123]

[0124] Where α is the directional gain coefficient, is the direction vector from the starting point to the target point, is the direction vector from the starting point to the random point, Represents the modulus length between the target point and the random point;

[0125] Guide the search towards the key detection areas of the blade, including the root and leading edge areas, to reduce invalid expansion nodes;

[0126] Dynamic step size adjustment: Adaptively adjust the extension step size according to the blade curvature, increasing the step size in flat areas and reducing the step size in curved areas.

[0127] Specifically, a high-precision IMU (Inertial Measurement Unit) and LiDAR are used to obtain the wind turbine's 3D attitude and blade position in real time. These sensors construct a point cloud map of obstacles, including key structures like the tower and blade surfaces. This map provides precise environmental information for subsequent path planning.

[0128] The RRT algorithm is a path planning algorithm based on a tree structure. Its core idea is to explore and find feasible paths by combining random sampling with tree structure expansion. Figure 2 As shown. First, initialize and set the starting point Q start is the root node of the tree T. Secondly, random sampling generates Q rand Again, in the tree T, the distance between nodes is calculated by the Euclidean distance metric method, and the node Q with the closest distance is selected. near Finally, along Q near to Q rand Direction expansion fixed step step to generate Q new , and perform collision detection. If Q new In free space and with Q near If there is no collision between the lines, add them to the tree T. Repeat this process until the target point Q is found. goal Or the maximum number of iterations is reached and a collision-free path is finally obtained.

[0129] A directional weight function is introduced into the random sampling process of the traditional RRT (Rapid Random Tree) algorithm to guide the search towards key detection areas of the blade, such as the root and leading edge areas. The form of the directional weight function is:

[0130]

[0131] Where α is the directional gain coefficient, is the direction vector from the starting point to the target point, is the direction vector from the starting point to the random point, Indicates the modulus length between the target point and the random point.

[0132] The extended step size is adaptively adjusted based on the curvature of the blade. In flat areas, the step size is increased to improve search efficiency; in curved areas, the step size is reduced to ensure path accuracy and safety. This adaptive adjustment mechanism helps generate a more optimal global path in complex terrain.

[0133] Combining the aforementioned directional heuristics and dynamic step-size adjustment strategy, the improved M-RRT algorithm can generate a globally optimal path from the starting point to the target point in complex environments. This path not only takes into account the distribution of environmental obstacles but also incorporates the inspection requirements of key blade areas, thereby improving inspection efficiency and safety. Through the above steps, step S105 implements global inspection path planning based on the current posture of the wind turbine, providing efficient and accurate path support for automated drone inspections.

[0134] Step S106 , based on the predicted shutdown posture of the wind turbine, the improved APF algorithm with dynamic weight allocation is used to perform local trajectory optimization.

[0135] Specifically, the local trajectory optimization is performed based on the predicted shutdown posture of the wind turbine using the improved APF algorithm with dynamic weight allocation, including:

[0136] Attitude prediction and constraint modeling: Combining wind turbine operating data, including rotational speed and wind speed, with historical shutdown attitudes, the LSTM network is used to predict the blade swing range in the shutdown state and define the dynamic restricted area of ​​the maximum blade deflection envelope.

[0137] Reconstruct the repulsive field and adjust the repulsive strength of the restricted area in real time. The gravitational force generated by the repulsive field is F rep :

[0138]

[0139] Among them, w t is the repulsive force weight, w t =k1·t+k2·Δθ, where t is the time attenuation factor, Δθ is the state deviation, k1 and k2 are the corresponding deviation coefficients, η is the repulsion coefficient, d is the distance between the current point and the target obstacle, and d0 is the distance between the starting point and the target obstacle. The repulsion strength of the restricted area is adjusted in real time;

[0140] Gravity-repulsion weight distribution: Increase the repulsion weight when approaching the high-risk area of ​​the blade tip, strengthen the gravitational field in the safe path section, and balance obstacle avoidance efficiency and path smoothness;

[0141] Determine whether a collision with an obstacle occurs;

[0142] If yes, reselect the parent node and rewire;

[0143] Otherwise, an obstacle avoidance trajectory path is formed and pruned to optimize the trajectory path.

[0144] Specifically, the system combines wind turbine operating data (such as rotational speed and wind speed) with historical shutdown postures to predict the blade swing range in the shutdown state using an LSTM network. This prediction is used to define the dynamic restricted area of ​​the maximum blade deflection envelope, thus providing constraints for subsequent path planning.

[0145] Based on the predicted parking posture, the repulsive field is reconstructed and the repulsive strength of the restricted area is adjusted in real time. The gravitational function of the repulsive field is defined as: Through this function, the repulsive field can be dynamically adjusted according to the real-time environment to ensure that the path avoids high-risk areas.

[0146] When approaching the high-risk area of ​​the blade tip, the repulsive weight is increased to enhance the obstacle avoidance capability; in the safe path section, the gravitational field is strengthened to guide the path closer to the target point. This weight distribution strategy can strike a balance between obstacle avoidance efficiency and path smoothness. During the path planning process, it is detected in real time whether there is a collision with an obstacle. If a collision occurs, the parent node is reselected and the wiring is re-arranged to generate a new obstacle avoidance path. If no collision occurs, an obstacle avoidance trajectory path is formed, and the path is optimized through a pruning algorithm to improve the smoothness and feasibility of the path. Through the above steps, step S106 realizes the local trajectory optimization based on the predicted shutdown posture of the wind turbine. Combined with the improved APF algorithm, it can effectively respond to the obstacle avoidance needs in complex environments and improve the intelligence and safety of the drone inspection path.

[0147] Step S107: trigger online re-planning based on environmental changes, mission requirements, and equipment status. Step S108: select the optimal trajectory based on multi-objective optimization of energy consumption, time, and safety factor.

[0148] Specifically, online replanning trigger conditions are set, including: triggering by wind turbine attitude sudden change detection based on image recognition, triggering by path tracking error exceeding a threshold, and triggering by environmental obstacles invading the safety area;

[0149] A multi-objective optimization evaluation module was established, including: establishing a multi-objective function including path length, safety, energy consumption, and shooting angle, and using the NSGA-III algorithm to select the Pareto optimal solution. Among the Pareto optimal solutions generated by the NSGA-III algorithm, the optimal solution was selected as the final replanning path based on the decision maker's preference or the weight of the expert system.

[0150] Specifically, online re-planning trigger conditions are set. The trigger conditions for online re-planning include: Wind turbine attitude mutation detection based on image recognition: The shutdown attitude of the wind turbine is monitored in real time through image recognition technology. If a sudden attitude change is detected (such as an abnormal change in the blade angle), re-planning is triggered. Path tracking error exceeds the threshold: If the UAV deviates from the preset track during flight and exceeds the set threshold, re-planning is triggered. Environmental obstacles intrude into the safety area: If obstacles (such as blades, towers, etc.) are detected entering the UAV's flight safety area, re-planning is triggered.

[0151] After online replanning, the optimal trajectory must be selected based on multi-objective optimization. This involves establishing a multi-objective function that incorporates objectives such as path length, safety, energy consumption, and camera angle. Path length reflects the economic efficiency of the trajectory; safety reflects obstacle avoidance capabilities; energy consumption reflects flight efficiency; and camera angle reflects mission completion quality.

[0152] Use NSGA-III algorithm for Pareto optimal solution selection: Use NSGA-III algorithm to optimize the multi-objective function and generate a Pareto optimal solution set, which contains multiple solutions that balance different objectives.

[0153] Select the optimal solution as the final path: From the Pareto optimal solution set, the optimal solution is selected as the final replanning path based on the decision maker's preferences or the weights of the expert system. For example, if the task prioritizes safety, the path with the highest safety is selected; if energy consumption is the priority, the path with the lowest energy consumption is selected.

[0154] Through the above steps, step S107 realizes online replanning based on environmental changes, mission requirements and equipment status, and combines multi-objective optimization methods to provide the optimal trajectory selection for the UAV, thereby improving the efficiency and safety of the flight mission.

[0155] Example 2

[0156] like Figure 3As shown, the present application provides an optimal trajectory planning system for a UAV that integrates image recognition with M-RRT and APF algorithms, which is applied to the optimal trajectory planning method for a UAV that integrates image recognition with M-RRT and APF algorithms as described in Example 1, including: a multimodal data acquisition module 11, a geometric mapping module 12, a posture recognition module 13, a posture prediction module 14, a path planning module 15, a path optimization module 16, a dynamic replanning module 17, and a multi-objective optimization module 18.

[0157] It can be understood that, in this embodiment, the multimodal data acquisition module 11 is used to acquire the three-dimensional point cloud data and visual information of the wind turbine in real time through the multimodal sensor array.

[0158] It can be understood that in this embodiment, the geometric mapping module 12 is used to identify key components of the wind turbine and estimate shutdown posture parameters based on the deep learning model, and establish a geometric mapping model between the spatial position of the wind turbine blades and the image taken by the drone.

[0159] It is understandable that, in this embodiment, the posture recognition module 13 is used to design a key point detection model to identify blade tips, use DBSCAN clustering to filter out erroneous blade tips, and automatically measure the current shutdown posture of the wind turbine.

[0160] It can be understood that, in this embodiment, the posture prediction module 14 is used to input the current shutdown posture into the geometric mapping model to obtain the predicted shutdown posture of the wind turbine at the next moment.

[0161] It can be understood that, in this embodiment, the path planning module 15 is configured to generate a global inspection path based on the current posture of the wind turbine by adopting the improved M-RRT algorithm that incorporates the directional heuristic factor.

[0162] It can be understood that, in this embodiment, the path optimization module 16 is configured to perform local trajectory optimization based on the predicted shutdown posture of the wind turbine using an improved APF algorithm with dynamic weight allocation.

[0163] It is understandable that, in this embodiment, the dynamic replanning module 17 is used to trigger online replanning according to environmental changes, task requirements and equipment status.

[0164] It can be understood that, in this embodiment, the multi-objective optimization module 18 is used to select the optimal trajectory based on multi-objective optimization of energy consumption, time, and safety factor.

[0165] Figure 4 This is an electronic device provided by an embodiment of the present application. Figure 4 As shown, the electronic device includes at least the following parts: a processor 101 and a memory 100 , a communication interface 103 , and a bus 102 .

[0166] In an embodiment of the present application, the memory 100 is used to store instructions executable by the processor 101, and the processor 101 is configured to implement the method of the first aspect when executing the instructions.

[0167] In an embodiment of the present application, a computer-readable storage medium includes instructions, and the instructions instruct a device to execute the method of the first aspect. For example, the instructions instruct the device to execute Figure 1 The method is shown in the process steps.

[0168] The program running in the electronic device involved in one embodiment of the present application can be a program that controls a central processing unit (CPU) and the like to realize the functions of the above-mentioned embodiment involved in one embodiment of the present invention (a program that enables a computer to function). Then, the information processed by these devices is temporarily stored in a random access memory (RAM) while being processed, and then stored in various ROMs such as read-only memory (Flash ROM) and hard disk drive (HDD), and is read, modified, and written by the CPU as needed.

[0169] It should be noted that a portion of the electronic device of the above embodiment may also be implemented by a computer. In this case, a program for implementing the control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer and executed.

[0170] It should be noted that the "computer" mentioned here refers to a computer built into an electronic device, employing hardware including an operating system (OS) and peripheral devices. Furthermore, "computer-readable recording medium" refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computers.

[0171] Furthermore, "computer-readable recording media" may include: media that dynamically store programs for a short period of time, such as communication lines when transmitting programs via networks such as the Internet or communication lines such as telephone lines; and media that store programs for a fixed period of time, such as volatile memory within computers acting as servers or clients in this context. Furthermore, the aforementioned program may be a program for implementing a portion of the aforementioned functions, or a program that can achieve the aforementioned functions by combining with a program already stored in a computer.

[0172] Furthermore, the electronic device in the above-described embodiments can also be implemented as a collection (device group) consisting of multiple devices. Each device constituting the device group may have a portion or all of the functions or functional blocks of the electronic device in the above-described embodiments. A device group only needs to have all the functions or functional blocks of the electronic device.

[0173] Those skilled in the art should recognize that the above embodiments are merely intended to illustrate the present application and are not intended to limit the present application. As long as they are within the spirit of the present application, appropriate changes and modifications to the above embodiments are within the scope of protection claimed in the present application.

Claims

1. A method for optimal trajectory planning of unmanned aerial vehicles (UAVs) that integrates image recognition with M-RRT and APF algorithms, characterized by: The method comprises: Acquire wind turbine 3D point cloud data and visual information in real time through a multimodal sensor array; Using deep learning models to identify key wind turbine components and estimate shutdown posture parameters, a geometric mapping model was established between the spatial position of wind turbine blades and images taken by drones. A key point detection model is designed to identify blade tips, and DBSCAN clustering is used to filter out incorrect blade tips, automatically measuring the current shutdown posture of the wind turbine. Inputting the current shutdown posture into the geometric mapping model to obtain a predicted shutdown posture of the wind turbine at the next moment; Based on the current posture of the wind turbine, the improved M-RRT algorithm integrating the directional heuristic factor is used to generate the global inspection path; Based on the predicted shutdown posture of the wind turbine, the improved APF algorithm with dynamic weight allocation is used to perform local trajectory optimization; Trigger online re-planning based on environmental changes, mission requirements, and equipment status; The optimal trajectory is selected based on multi-objective optimization of energy consumption, time and safety factor.

2. The optimal trajectory planning method for unmanned aerial vehicle (UAV) integrating image recognition, M-RRT and APF algorithm according to claim 1 is characterized in that: The method of identifying key components of a wind turbine based on a deep learning model and estimating shutdown posture parameters, and establishing a geometric mapping model between the spatial position of wind turbine blades and images taken by a drone, includes: Detect and locate wind turbine blades by training deep learning models; Extract blade edge information through Canny edge detection and Hough transform; According to the pixel coordinates of the blade tip and the UAV posture information, the pixel coordinates of the blade tip are calculated through coordinate system transformation; Combined with the UAV's position information, the GPS coordinates of the blade tip are calculated to determine the blade's rotation angle; According to the mapping relationship between the blade's motion trend, environmental changes, and the rotation angle and yaw angle at different time points, a geometric mapping model between the spatial position of the wind turbine blade and the image taken by the drone is constructed.

3. The optimal trajectory planning method for unmanned aerial vehicle (UAV) integrating image recognition, M-RRT and APF algorithm according to claim 2 is characterized in that: The design key point detection model identifies blade tip points, uses DBSCAN clustering to filter out erroneous blade tip points, and automatically measures the current shutdown posture of the wind turbine, including: Identify blade tip pixels based on a deep learning model; Correcting the blade tip pixels using Hough straight lines, identifying points in low-density areas using the DBSCAN clustering algorithm, and eliminating data points that deviate from abnormal angles; By analyzing the distribution of blade tip points and according to the reference relationship between the blade tip points and the coordinate axes in the coordinate system, the blade inclination parameters are determined. Automatically measure the current shutdown posture of the wind turbine.

4. The optimal trajectory planning method for unmanned aerial vehicle (UAV) integrating image recognition, M-RRT and APF algorithm according to claim 3 is characterized in that: The step of inputting the current shutdown posture into the geometric mapping model to obtain a predicted shutdown posture of the wind turbine at the next moment includes: Get the arrival time to the next wind turbine; Inputting the arrival time and the current parking posture into the geometric mapping model to predict the parking posture at the next moment; The shutdown posture at the next moment is used as the predicted shutdown posture of the next wind turbine.

5. The optimal trajectory planning method for unmanned aerial vehicle (UAV) integrating image recognition, M-RRT and APF algorithm according to claim 4 is characterized in that: The global inspection path is generated based on the current posture of the wind turbine using the improved M-RRT algorithm that integrates the directional heuristic factor, including: Environmental perception and attitude modeling: High-precision IMU and LiDAR are used to obtain the current 3D attitude of the wind turbine and the spatial position of the blades in real time, and to construct a point cloud map of obstacles including the tower and blade surfaces; Directional heuristic factor fusion: Introducing the directional weight function w into the traditional RRT random sampling d : Where α is the directional gain coefficient, is the direction vector from the starting point to the target point, is the direction vector from the starting point to the random point, Represents the modulus length between the target point and the random point; Guide the search towards the key detection areas of the blade, including the root and leading edge areas, to reduce invalid expansion nodes; Dynamic step size adjustment: Adaptively adjust the extension step size according to the blade curvature, increasing the step size in flat areas and reducing the step size in curved areas.

6. The optimal trajectory planning method for unmanned aerial vehicle (UAV) integrating image recognition, M-RRT and APF algorithm according to claim 5, characterized in that: The method of performing local trajectory optimization based on the predicted shutdown posture of the wind turbine using the improved APF algorithm with dynamic weight allocation includes: Attitude prediction and constraint modeling: Combining wind turbine operating data, including rotational speed and wind speed, with historical shutdown attitudes, the LSTM network is used to predict the blade swing range in the shutdown state and define the dynamic restricted area of ​​the maximum blade deflection envelope. Reconstruct the repulsive field and adjust the repulsive strength of the restricted area in real time. The gravitational force generated by the repulsive field is F rep : Among them, w t is the repulsive force weight, w t =k1·t+k2·Δθ, where t is the time attenuation factor, Δθ is the state deviation, k1 and k2 are the corresponding deviation coefficients, η is the repulsion coefficient, d is the distance between the current point and the target obstacle, and d0 is the distance between the starting point and the target obstacle. The repulsion strength of the restricted area is adjusted in real time; Gravity-repulsion weight distribution: Increase the repulsion weight when approaching the high-risk area of ​​the blade tip, strengthen the gravitational field in the safe path section, and balance obstacle avoidance efficiency and path smoothness; Determine whether a collision with an obstacle occurs; If yes, reselect the parent node and rewire; Otherwise, an obstacle avoidance trajectory path is formed and pruned to optimize the trajectory path.

7. The optimal trajectory planning method for unmanned aerial vehicle (UAV) integrating image recognition, M-RRT and APF algorithms according to claim 6, characterized in that: Trigger online re-planning based on environmental changes, mission requirements, and equipment status; The optimal trajectory is selected based on multi-objective optimization of energy consumption, time, and safety factor, including: Set online replanning trigger conditions, including: wind turbine attitude sudden change detection based on image recognition, path tracking error exceeding a threshold, and environmental obstacles invading the safety zone; A multi-objective optimization evaluation module was established, including: establishing a multi-objective function including path length, safety, energy consumption, and shooting angle, and using the NSGA-III algorithm to select the Pareto optimal solution. Among the Pareto optimal solutions generated by the NSGA-III algorithm, the optimal solution was selected as the final replanning path based on the decision maker's preference or the weight of the expert system.

8. An optimal trajectory planning system for unmanned aerial vehicles (UAVs) that integrates image recognition with M-RRT and APF algorithms, applied to the optimal trajectory planning method for unmanned aerial vehicles (UAVs) that integrates image recognition with M-RRT and APF algorithms as described in any one of claims 1 to 7, characterized in that: include: Multimodal data acquisition module, used to acquire wind turbine 3D point cloud data and visual information in real time through a multimodal sensor array; A geometric mapping module is used to identify key components of wind turbines and estimate shutdown posture parameters based on a deep learning model, and to establish a geometric mapping model between the spatial position of wind turbine blades and images taken by drones; The posture recognition module is used to design a key point detection model to identify the blade tip, use DBSCAN clustering to filter out incorrect blade tip points, and automatically measure the current shutdown posture of the wind turbine; a posture prediction module, configured to input the current shutdown posture into the geometric mapping model to obtain a predicted shutdown posture of the wind turbine at the next moment; The path planning module is used to generate a global inspection path based on the current posture of the wind turbine using an improved M-RRT algorithm that incorporates directional heuristic factors; Path optimization module, which is used to perform local trajectory optimization based on the predicted shutdown posture of wind turbines using the improved APF algorithm with dynamic weight allocation; Dynamic replanning module, used to trigger online replanning based on environmental changes, task requirements, and equipment status; The multi-objective optimization module is used to select the optimal trajectory based on multi-objective optimization of energy consumption, time, and safety factor.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the optimal trajectory planning method for a UAV that integrates image recognition with M-RRT and APF algorithms as described in any one of claims 1 to 7 when executing the instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, and the program instructs the device to execute the optimal trajectory planning method for a drone that integrates image recognition with M-RRT and APF algorithms as described in any one of claims 1 to 7.

Images