A method for real-time detection of defects of curved surface components by adaptive flying of unmanned aerial vehicle

Abstract

The application provides a kind of unmanned plane adaptive fly-around's curved surface component defect real-time detection method, belong to unmanned plane intelligent inspection technical field, method includes: step S1: obtain laser point cloud data and image data;Step S2: reconstruct local geometric model;Step S3: construct unmanned plane expected pose sequence;Step S4: execute adaptive fly-around;Step S5: defect identification;Step S6: obtain the three-dimensional coordinates of defect on the surface of curved surface component.The application solves the problems of poor observation consistency, weak trajectory adaptive ability, imaging distortion and task and flight control decoupling in the detection of large curvature and irregular continuous curved surface components by the geometric reconstruction mechanism of multi-source perception fusion, combined with local curved surface geometry reconstruction, dynamic weight observation consistency deviation, NMPC adaptive adjustment mechanism and incremental scanning and closed-loop detection scheme of three-dimensional mapping.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection technology for unmanned aerial vehicles (UAVs), and in particular to a real-time defect detection method for curved surface components using an adaptive UAV flight path. Background Technology

[0002] With the advancement of modern industrial automation and intelligent monitoring technologies, health monitoring of large infrastructure, aerospace components, and energy equipment has become a core aspect of ensuring the safety of major projects. In applications such as wind turbine blades, large storage tanks, aircraft fuselages, and irregularly shaped building curtain walls, curved components are prone to defects such as fatigue cracks, corrosion pits, and coating peeling due to their complex geometry and harsh environments. Traditional inspection methods, such as manual visual inspection and large-scale hoisting equipment operations, are labor-intensive, high-risk, and time-consuming, and cannot cover complex on-site conditions.

[0003] Against this backdrop, UAV platforms equipped with visual perception systems have gradually become the mainstream technology for real-time defect detection of complex components due to their high mobility and strong environmental adaptability. However, existing technologies mainly rely on preset waypoint planning, and offline trajectory planning methods are effective for regular geometric surfaces, but cannot correct trajectories in real time on surfaces with large curvature or irregular curvature, resulting in fluctuations in imaging quality and incomplete coverage.

[0004] The local normal vectors, radii of curvature, and occlusion relationships of curved components change continuously in space. If a UAV maintains fixed flight parameters, it can easily lead to problems such as perspective distortion, local blurring, and insufficient field-of-view overlap. Current methods lack online reconstruction capabilities for local geometric changes, and the detection task is often decoupled from flight control, making it impossible to dynamically adjust the flight strategy. Especially in edge computing environments, without a unified feedback chain, data redundancy and missed detections are likely to occur.

[0005] Therefore, how to organically combine online surface geometry perception, observation consistency assessment, adaptive update of control parameters, and defect-driven incremental scanning has become a key technical issue for improving the intelligent operation and maintenance level of complex components and ensuring the safety of major projects. Summary of the Invention

[0006] This invention provides a real-time defect detection method for curved surface components in UAV adaptive flight. By using a geometric reconstruction mechanism based on multi-source perception fusion, combined with local surface geometric reconstruction, dynamic weighted observation consistency deviation, NMPC adaptive adjustment mechanism, and a closed-loop detection scheme of incremental scanning and 3D mapping, it solves the problems of poor observation consistency, weak trajectory adaptation capability, imaging distortion, and decoupling between mission and flight control in the detection of UAV components with large curvature and irregular continuous curved surfaces.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A real-time defect detection method for curved surface components using an adaptive UAV flyaround technique includes: Step S1: Perform time synchronization and spatial calibration on the lidar, camera and inertial measurement unit mounted on the UAV to acquire spatiotemporally synchronized lidar point cloud data and image data; Step S2: Based on the laser point cloud data, reconstruct the local geometric model of the inspected curved surface component in real time, and calculate the unit normal vector and principal curvature information at each sampling point on the target surface according to the local geometric model; Step S3: Based on the unit normal vector and principal curvature information, construct the UAV expected pose sequence that conforms to the optical axis alignment constraint, observation distance constraint and field of view overlap constraint, and generate the observation consistency deviation amount according to the optical axis angle deviation term, observation distance deviation term and field of view overlap rate deviation term driven by the principal curvature information dynamic weight. Step S4: Use the principal curvature information and the observation consistency deviation as feedback inputs for nonlinear model predictive control, and use the desired pose sequence as a reference trajectory to control the UAV to perform adaptive flight around the surface of the curved component. The adaptive flight around adopts a conventional flight around step size preset according to the geometric complexity of the curved component. Step S5: During the flight around the camera, defect identification is performed on the image data, and when a suspected defect is identified, the UAV is controlled to perform incremental scanning with an incremental step size smaller than the conventional flight around step size. Step S6: Back-project the identified defect pixels as observation rays, and calculate the intersection of the observation rays with the local geometric model to obtain the three-dimensional coordinates of the defect on the surface of the curved component.

[0008] In this specification, the time synchronization and spatial calibration in step S1 use a unified clock source to synchronously trigger the lidar, camera and inertial measurement unit, and establish the spatial coordinate transformation relationship between the lidar, camera and inertial measurement unit through offline calibration; during flight, the data of the inertial measurement unit is used to perform motion compensation on the lidar point cloud data and image data.

[0009] In this specification, the real-time reconstruction of the local geometric model in step S2 includes: statistical filtering to remove outliers from the laser point cloud data; dynamically determining the voxel grid side length based on the principal curvature information; performing voxel grid downsampling on the point cloud after removing outliers; and using a moving least squares method with a Gaussian weighting function to perform local surface fitting on the downsampled point cloud to reconstruct the local geometric model and solve for the unit normal vector and the principal curvature information.

[0010] In this specification, the construction of the UAV desired pose sequence in step S3 satisfies the following constraints: the optical axis vector of the camera is antiparallel to the unit normal vector at the sampling point on the target surface; the distance from the optical center of the camera to the sampling point is within a preset object distance range; the image field of view overlap rate corresponding to adjacent desired poses is not lower than a preset threshold; the observation consistency deviation is obtained by weighting and normalizing the optical axis angle deviation term, the observation distance deviation term, and the field of view overlap rate deviation term according to the dynamic weight; the dynamic weight increases the weight of the optical axis angle deviation term and the observation distance deviation term when the local curvature represented by the principal curvature information increases, and increases the weight of the field of view overlap rate deviation term when the local curvature decreases.

[0011] In this specification, the nonlinear model predictive control in step S4 uses the desired pose sequence as a reference trajectory and the principal curvature information and the observation consistency deviation as feedback quantities to adaptively update the state tracking weight matrix and control weighting matrix in the nonlinear model predictive control; under the conditions of satisfying the UAV dynamics constraints, actuator amplitude limiting constraints and obstacle avoidance constraints constructed based on the local geometric model, the optimal control quantity is solved to drive the UAV to fly.

[0012] In this specification, the state tracking weight matrix increases according to the increase of the principal curvature information and the observation consistency deviation, so as to enhance the control weight for pose tracking and attitude alignment; the control weighting matrix increases according to the increase of the observation consistency deviation, so as to suppress drastic changes in the control quantity; and the prediction time domain of the nonlinear model predictive control is shortened according to the increase of the principal curvature information.

[0013] In this specification, after calculating the unit normal vector in step S2, normal vector consistency correction is also performed: the cosine value of the angle between the normal vectors of adjacent sampling points is checked. If the angle is greater than a preset angle threshold, the normal vector of the high signal-to-noise ratio region is used as the seed point, and the normal vector of the local discontinuous region is weighted, redirected and smoothed through the normal vector propagation algorithm to ensure the continuity of the normal vector field in space.

[0014] In this specification, the defect identification in step S5 is achieved by a deep convolutional neural network running in the edge computing module; the deep convolutional neural network outputs the defect category label, bounding box coordinates and confidence score; when the confidence score reaches a preset first-level threshold, it is determined that the suspected defect has been identified.

[0015] In this specification, the incremental scanning in step S5 includes: taking the center of the suspected defect as a reference, determining the local circumference radius, angular step size and sampling density according to the geometric envelope size of the suspected defect, generating a spiral sampling trajectory with a step size of one-third to one-half of the conventional circumference step size within the normal cone angle of the curved component surface, discretizing the spiral sampling trajectory into a dense observation pose point sequence, and controlling the UAV to sequentially execute the dense observation pose point sequence to complete the local detailed inspection.

[0016] In this specification, the generation of the observation ray in step S6 includes: back-projecting the center pixel coordinates of the bounding box of the defect into a three-dimensional ray originating from the optical center of the camera based on the intrinsic parameters and real-time pose of the camera; calculating the intersection point using a numerical intersection method to solve for the intersection point of the three-dimensional ray with the surface equation of the local geometric model, and using the spatial coordinates of the intersection point as the three-dimensional coordinates of the defect on the surface of the surface component; after obtaining the three-dimensional coordinates, associating the defect's category label, bounding box size, confidence score, and the three-dimensional coordinates with a dynamically constructed surface topology map, and transmitting them to the ground station via a wireless data transmission link to form a structured detection result.

[0017] In summary, the present invention has at least the following beneficial effects: Improve observation accuracy and consistency: By reconstructing the local geometric model in real time and combining normal vectors, distance and field of view overlap constraints, perspective distortion, local blur and coverage discontinuity problems are effectively reduced.

[0018] Geometric adaptive trajectory control: Based on the deviation between the principal curvature information and the observation consistency, the NMPC parameters are adaptively updated to achieve higher trajectory stability, imaging consistency and obstacle avoidance safety.

[0019] Closed-loop mission and flight control linkage: Introducing a mission-driven closed-loop feedback mechanism, dynamically adjusting the inspection mode and triggering incremental scanning, achieving an adaptive transition from global to local fine inspection, and improving the detection rate of key defects.

[0020] Precise defect localization: By finding the intersection between the image and the surface model, defects are accurately located in three-dimensional space, reducing location drift and providing high-confidence data support for subsequent decision-making and maintenance.

[0021] This invention forms a closed-loop intelligent inspection system through geometric perception, observation and evaluation, control adaptation, and task-driven detailed inspection, providing effective defect detection and flight control. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the real-time defect detection method for curved surface components using adaptive drone flight, as described in this invention.

[0024] Figure 2 This is a schematic diagram of the sensing data acquisition system involved in this invention.

[0025] Figure 3 This is a schematic diagram of the geometric constraint principle of the UAV flying around curved surface component involved in this invention. Detailed Implementation

[0026] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0027] The following disclosure provides many different implementations or examples for carrying out different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the embodiments of the present invention; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0028] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] like Figure 1 As shown, this embodiment provides a real-time defect detection method for curved surface components using an adaptive UAV flight path, including: Step S1: Perform time synchronization and spatial calibration on the lidar, camera and inertial measurement unit mounted on the UAV to acquire spatiotemporally synchronized lidar point cloud data and image data; Step S2: Based on the laser point cloud data, reconstruct the local geometric model of the inspected curved surface component in real time, and calculate the unit normal vector and principal curvature information at each sampling point on the target surface according to the local geometric model; Step S3: Based on the unit normal vector and principal curvature information, construct the UAV expected pose sequence that conforms to the optical axis alignment constraint, observation distance constraint and field of view overlap constraint, and generate the observation consistency deviation amount according to the optical axis angle deviation term, observation distance deviation term and field of view overlap rate deviation term driven by the principal curvature information dynamic weight. Step S4: Use the principal curvature information and the observation consistency deviation as feedback inputs for nonlinear model predictive control, and use the desired pose sequence as a reference trajectory to control the UAV to perform adaptive flight around the surface of the curved component. The adaptive flight around adopts a conventional flight around step size preset according to the geometric complexity of the curved component. Step S5: During the flight around the camera, defect identification is performed on the image data, and when a suspected defect is identified, the UAV is controlled to perform incremental scanning with an incremental step size smaller than the conventional flight around step size. Step S6: Back-project the identified defect pixels as observation rays, and calculate the intersection of the observation rays with the local geometric model to obtain the three-dimensional coordinates of the defect on the surface of the curved component.

[0030] In some embodiments, the time synchronization and spatial calibration in step S1 use a unified clock source to synchronously trigger the lidar, camera and inertial measurement unit, and establish the spatial coordinate transformation relationship between the lidar, camera and inertial measurement unit through offline calibration; during flight, the data of the inertial measurement unit is used to perform motion compensation on the lidar point cloud data and image data.

[0031] In some embodiments, the real-time reconstruction of the local geometric model in step S2 includes: statistically filtering outliers from the laser point cloud data; dynamically determining the voxel grid side length based on the principal curvature information; downsampling the point cloud after outlier removal using the voxel grid; and performing local surface fitting on the downsampled point cloud using a moving least squares method with a Gaussian weighting function to reconstruct the local geometric model and solve for the unit normal vector and the principal curvature information.

[0032] In some embodiments, the construction of the UAV desired pose sequence in step S3 satisfies the following constraints: the optical axis vector of the camera is antiparallel to the unit normal vector at the sampling point on the target surface; the distance from the optical center of the camera to the sampling point is within a preset object distance range; the image field of view overlap rate corresponding to adjacent desired poses is not lower than a preset threshold; the observation consistency deviation is obtained by weighting and normalizing the optical axis angle deviation term, the observation distance deviation term, and the field of view overlap rate deviation term according to the dynamic weight; the dynamic weight increases the weight of the optical axis angle deviation term and the observation distance deviation term when the local curvature represented by the principal curvature information increases, and increases the weight of the field of view overlap rate deviation term when the local curvature decreases.

[0033] In some embodiments, the nonlinear model predictive control in step S4 uses the desired pose sequence as a reference trajectory and the principal curvature information and the observation consistency deviation as feedback quantities to adaptively update the state tracking weight matrix and control weight matrix in the nonlinear model predictive control; under the conditions of satisfying the UAV dynamics constraints, actuator amplitude limiting constraints and obstacle avoidance constraints constructed based on the local geometric model, the optimal control quantity is solved to drive the UAV to fly.

[0034] In some embodiments, the state tracking weight matrix increases according to the increase of the principal curvature information and the observation consistency deviation to enhance the control weights for pose tracking and attitude alignment; the control weighting matrix increases according to the increase of the observation consistency deviation to suppress drastic changes in the control quantity; and the prediction time domain of the nonlinear model predictive control is shortened according to the increase of the principal curvature information.

[0035] In some embodiments, after calculating the unit normal vector in step S2, normal vector consistency correction is also performed: the cosine value of the angle between the normal vectors of adjacent sampling points is checked. If the angle is greater than a preset angle threshold, the normal vector of the high signal-to-noise ratio region is used as the seed point, and the normal vector of the local discontinuous region is weighted, redirected and smoothed through the normal vector propagation algorithm to ensure the continuity of the normal vector field in space.

[0036] In some embodiments, the defect identification in step S5 is achieved by a deep convolutional neural network running within the edge computing module; the deep convolutional neural network outputs the defect's category label, bounding box coordinates, and confidence score; when the confidence score reaches a preset first-level threshold, it is determined that the suspected defect has been identified.

[0037] In some embodiments, the incremental scanning in step S5 includes: using the center of the suspected defect as a reference, determining the local circumference radius, angular step size and sampling density according to the geometric envelope size of the suspected defect, generating a spiral sampling trajectory with a step size of one-third to one-half of the conventional circumference step size within the normal cone angle of the surface of the curved component, discretizing the spiral sampling trajectory into a dense observation pose point sequence, and controlling the UAV to sequentially execute the dense observation pose point sequence to complete the local detailed inspection.

[0038] In some embodiments, the generation of the observation ray in step S6 includes: back-projecting the center pixel coordinates of the bounding box of the defect into a three-dimensional ray originating from the optical center of the camera based on the intrinsic parameters and real-time pose of the camera; calculating the intersection point using a numerical intersection method to solve for the intersection point of the three-dimensional ray with the surface equation of the local geometric model, and using the spatial coordinates of the intersection point as the three-dimensional coordinates of the defect on the surface of the surface component; after obtaining the three-dimensional coordinates, associating the defect's category label, bounding box size, confidence score, and the three-dimensional coordinates with a dynamically constructed surface topology map, and transmitting them to the ground station via a wireless data transmission link to form a structured detection result.

[0039] The technical concept of this invention is as follows: A real-time defect detection method for curved components using UAV adaptive flyaround is proposed. This method acquires data spatiotemporally and synchronously from multiple sources of sensors, reconstructs the local geometric model of the inspected area in real time, and calculates the unit normal vector and principal curvature. Furthermore, the method uses the principal curvature to drive the dynamic weight allocation of optical axis angle deviation, distance deviation, and field-of-view overlap deviation, constructs an observation consistency deviation, and uses it as a nonlinear model predictive control feedback to achieve adaptive flyaround. The preset threshold is calibrated based on the camera imaging quality and defect detection accuracy requirements. The conventional flyaround step size is the basic scanning step size preset according to the geometric complexity of the curved component, and the incremental scanning step size is one-third to one-half of the conventional flyaround step size.

[0040] Step S1: Spatiotemporal synchronization and calibration of multi-source sensors and elimination of point cloud motion distortion.

[0041] like Figure 2 As shown, the perception data acquisition system is mounted on a UAV and includes a multi-line LiDAR, a global shutter CMOS camera, an inertial measurement unit (IMU), and an edge computing module. A unified clock source is used to synchronize the hardware triggering of the LiDAR, global shutter CMOS camera, and IMU, and a unified synchronization pulse signal is used to trigger the hardware to ensure the spatiotemporal synchronization of data acquisition, providing a consistent data foundation for subsequent processing.

[0042] In the spatial dimension, establishing the transformation matrix between sensors is a prerequisite for ensuring the fusion of multi-source information. The camera intrinsic parameter matrix and the distortion coefficient vector composed of radial and tangential distortions are obtained through an offline calibration algorithm. Simultaneously, the transformation matrix from the camera coordinate system to the LiDAR coordinate system is calculated using a checkerboard pattern or a specially designed calibration field. And the transformation matrix from the lidar coordinate system to the IMU coordinate system. .

[0043] In the actual operation process, the edge computing module receives high-frequency motion data transmitted back by the IMU in real time, and uses the strapdown inertial navigation algorithm or extended Kalman filter (EKF) to predict and compensate the instantaneous pose of the UAV within the interval between two laser samplings, thereby eliminating the point cloud motion distortion caused by the UAV's motion and performing motion compensation on the image data.

[0044] Step S2: Real-time reconstruction and curvature analysis of the local geometric model.

[0045] Step S2 includes the following specific processing flow: Sub-step 2.1: Statistical filtering for outlier removal; Statistical outlier removal is performed on the original point cloud from the lidar system. Noisy points are identified and removed based on the mean and standard deviation of their neighborhood distances. Let the original point cloud set be... For any point Calculate its local neighborhood

[0046] Average distance between the nearest neighbors: ; The operator for the Euclidean norm (modulus) of a vector; The preset number of neighboring points; For point The j-th neighboring point; Global average distance with standard deviation for: ; N is the total number of points in the original point cloud set; points that meet the following conditions are identified as outliers and removed: ; in The outlier determination coefficient ranges from 1.5 to 3.0.

[0047] As one embodiment of the present invention, the point cloud noise reduction processing further includes employing radius-based outlier filtering. A radius is set. If the number of neighboring points contained in a spherical region centered at a certain point is less than... If the point is not found to be noise due to multipath reflection or environmental interference, it will be removed.

[0048] Sub-step 2.2: Dynamic voxel grid downsampling; Based on principal curvature factor Dynamically determine the voxel grid side length: ; in: The side length of the basic voxel is taken as 0.05m to 0.1m; This is a dynamic adjustment coefficient, ranging from 0.3 to 0.8. It is the principal curvature factor.

[0049] The inspected area is divided into sections with sides of length [missing information]. Three-dimensional voxel blocks, points Mapping to voxel raster space index: ; This is the floor operator; The integer index coordinates of points in the original point cloud in a 3D voxel grid after spatial mapping. Perform centroid downsampling on the point cloud within a single voxel and output the centroid point. : ; in This represents the number of point clouds contained within a single voxel. Let m be the spatial coordinates of the m-th point within the voxel. The centroid coordinates (i.e., the average spatial position) of all original point cloud coordinates within the voxel grid. Within each voxel, the arithmetic mean of all original point cloud coordinates is calculated as the centroid, and all points within that voxel are replaced by this centroid coordinates, thereby transforming the irregularly distributed point cloud into a sparse set of points with topological association.

[0050] Sub-step 2.3: Moving Least Squares (MLS) local surface fitting; A Gaussian-weighted moving least squares method is used to perform local surface analytical fitting on the downsampled point cloud. For the downsampled points... Select radius Neighborhood point set Construct the Gaussian weight function: ; in The smoothing factor has a value of 0.01 to 0.03. Let be the Gaussian weight value corresponding to the j-th neighboring point; It is a natural exponential function; In the downsampled point cloud, located at the center point The three-dimensional coordinate vector of the j-th neighboring point within the local neighborhood radius r.

[0051] For each observation point in the voxelized point cloud, a second-order polynomial surface equation is defined within its local neighborhood: ; The coefficients of the local surface equation to be fitted are determined by solving the weighted least squares method. Minimize the error function using weighted least squares method. : ; Find the coefficient vector Where A is a column vector consisting of the six coefficients of the second-order polynomial surface equation, with superscript... This indicates transpose. The weight function is... Gaussian weighting function , The distance from a neighboring point to the center point. To influence the radius parameter, this process ensures that the fitted surface has extremely high smoothness and geometric accuracy locally. For point The actual height (observed value).

[0052] At the same time, plane equations can also be constructed. Construct a weighted least squares optimization objective function: ; Solve for the surface normal vector and normalize it: ; superscript Indicates transpose. For an unnormalized normal vector (3D vector), derived from the local surface equation The gradient is calculated directly, and its direction is the normal direction of the tangent plane at that point of the surface, but the modulus is not fixed (affected by the slope of the surface). The gradient operator represents the expression for the surface equation. Find the vector of first-order partial derivatives. This is the normalized unit normal vector (magnitude of 1), which is the final attitude reference vector used for subsequent optical axis alignment constraints. Representing a three-dimensional vector The Euclidean norm (i.e., the modulus) is calculated using the following formula: .

[0053] Sub-step 2.4: Real-time calculation and curvature analysis of the target surface normal vector field distribution; After the surface fitting is completed, the system calculates the normal vector field distribution of the target surface in real time and performs curvature analysis. Based on the fitted second-order polynomial equation, the curvature of any point on the surface is calculated. Partial derivative at and This leads to the derivation of the unit normal vector at that point: ; Simultaneously, the Hessian matrix of the local region is calculated. Then, its eigenvalues ​​are extracted to obtain the principal curvatures of the surface. and (i.e., maximum and minimum principal curvature). This curvature information is not only used to characterize the geometric complexity of the surface components, but also serves as a key parameter for subsequent dynamic adjustment of the UAV's fly-around step size and observation density. The principal curvature factor K is set to a default value of 0 during the first execution, and K is updated from the curvature analysis results of the previous frame in subsequent loops.

[0054] Sub-step 2.5: Normal vector consistency correction; To ensure the logical consistency of the normal vector field in spatial distribution, this embodiment further introduces consistency correction logic. By checking the cosine value of the angle between the normal vectors of adjacent sampling points, if abrupt changes with an angle greater than 45° are found, the normal vector propagation algorithm is used to redirect and smooth outwards from a high signal-to-noise ratio region as the seed point, thereby eliminating the influence of surface wrinkles or artifacts. ; Sampling points The final normal vector (unit vector) after normal vector consistency correction replaces the initial normal vector. Used for subsequent steps. Sampling points The initial normal vector (unit vector) is directly obtained by the moving least squares surface fitting in step S2.3. adjacent sampling points and The angle between the normal vectors (unit: degrees). The calculation formula is: ,in It is the dot product of vectors. Universal quantifier, meaning "for". All neighboring points in the set All satisfied When "", the first case is executed (keeping the original normal vector unchanged). : Existential quantifier, indicating "in". In the set, there exists at least one neighboring point. Make When the condition is met, the second case (weighted redirection smoothing) is executed. Sampling points of Neighborhood point set, i.e., distance Recent The set consisting of points. Neighborhood point set The first in A three-dimensional coordinate vector of a neighboring point. Neighborhood points For the center point The normal vector propagation weight coefficient. This weight is related to the spatial distance and normal vector consistency, and is usually taken as the Gaussian weight in step S2.3. Consistent, or additionally weighted based on the cosine of the angle between the normal vectors. Neighborhood points The initial normal vector (unit vector).

[0055] : For the neighborhood point set The weighted average normal vector (unnormalized) is obtained by summing the normal vectors of all neighboring points. The Euclidean norm (modulus) of the above weighted summation result.

[0056] Smoothing is applied to local discontinuous regions to ensure the continuity of the normal vector field in space.

[0057] Sub-step 2.6: Output the local geometric model; The local geometric model is output, including the surface equation, normal vector, and principal curvature factor, and the geometric model of the inspected region is continuously updated. In step S2, outliers are removed from the original point cloud, the point cloud is downsampled based on a dynamic voxel grid, and the moving least squares method is used to fit the local surface to the sampled points and their neighborhood point sets to reconstruct the local geometric model. Then, the unit normal vector and principal curvature information at the sampled points are calculated. The principal curvature information is used to characterize the local geometric complexity and serves as the basis for the adaptive update of subsequent observation density and control parameters.

[0058] Step S3: Observation consistency assessment and desired pose generation.

[0059] For mapping the desired pose of the UAV, embodiments of this invention construct a stringent observation geometry consistency constraint system. For example... Figure 3 The desired observation pose of the UAV must satisfy the following geometric constraints: 1. Optical axis alignment constraint: The optical axis vector of the camera must be opposite to and parallel to the normal vector at the target point on the inspected surface. This requires the optical axis vector of the global shutter high-resolution CMOS camera to be strictly opposite to and parallel to the normal vector at the target point on the inspected surface, thereby eliminating the projection stretching caused by oblique imaging and reducing the projection stretching caused by oblique imaging.

[0060] 2. Constant distance constraint: The Euclidean distance from the camera optical center to the surface under inspection must be maintained within the preset optimal object distance range. The optimal object distance is set according to the characteristics of the camera's optical modulation transfer function and the depth of field range, and the Euclidean distance from the camera optical center to the surface is required to meet the allowable deviation range. The optimal object distance is determined by the lens imaging quality evaluation index.

[0061] 3. Field of view overlap constraint: The image overlap rate between adjacent waypoints must be no less than 60%. This is achieved by calculating the coverage relationship between adjacent waypoints corresponding to the UAV's flight speed and camera sampling frequency, ensuring that the overlap rate is no less than 60% to guarantee the continuity of defect features.

[0062] Based on the above constraints, a mapping relationship from local surface geometry to the desired pose of the UAV is established. A dynamically weighted normalized observation consistency deviation driven by the principal curvature is constructed based on the optical axis angle deviation, distance deviation, and field-of-view overlap rate deviation, serving as feedback input for subsequent adaptive updates of control parameters. The principal curvature factor is normalized to obtain a normalized principal curvature factor, which is then used for the dynamic weighting of the observation consistency deviation.

[0063] Define the normalized principal curvature factor: ; in This represents the maximum absolute value of the principal curvature in the current local region. This is the curvature reference value.

[0064] Define the observation consistency deviation as: ; in This is the normalized value for the deviation of the optical axis angle. This is the normalized quantity for the observation distance deviation. This is the normalized value for the field of view overlap rate deviation. The dynamic weights for optical axis alignment constraints. For the dynamic weights of the observation distance constraint, The dynamic weights for the field-of-view overlap constraint.

[0065] The dynamic weights satisfy: ; ; ; in: ; in These are the basic weights for optical axis alignment constraints, observation distance constraints, and field-of-view overlap constraints, respectively. The weighted adjustment coefficient driven by the principal curvature. The normalization factor is the sum of the three unnormalized weights.

[0066] To achieve adaptive adjustment of inspection strategies under different curvature regions, a curvature-based approach is introduced. The dynamic weight allocation mechanism. With curvature... The increase, and Monotonically increasing, while Monotonic reduction automatically increases the importance of optical axis alignment and constant distance constraints in high curvature regions, and relatively increases the importance of field-of-view overlap constraints in flat regions.

[0067] Step S4: Generate adaptive flight trajectory based on NMPC.

[0068] To achieve high-precision trajectory tracking and flyaround control, a nonlinear model predictive control (NMPC) framework was adopted. A six-degree-of-freedom nonlinear dynamic model of the UAV was constructed, with its state vector encompassing position, velocity, attitude rotation matrix, and body angular velocity. The desired pose sequence generated in step S3 was used as the reference trajectory, and the principal curvature information and observation consistency deviation were used as control feedback variables.

[0069] In step S4, nonlinear model predictive control is employed, using the desired pose sequence generated in step S3 as the reference trajectory, and incorporating the principal curvature information and observation consistency deviation. As a control feedback quantity.

[0070] The objective function of the nonlinear model predictive control satisfy: ; in To predict the time domain, To control the time domain, This is the state tracking weight matrix. To control the weighting matrix. The predicted state vector (column vector) represents the state in... Always looking towards the future The predicted value of the state at any given time. The reference trajectory state vector (column vector) represents the state of the trajectory at the reference trajectory. Always looking towards the future Reference values ​​for the expected state at any given time. To control the input vector (column vector), representing in Always looking towards the future The solution value of the control quantity at any given time, where The square of the weighted Euclidean norm is expressed by the following formula: ; Similarly, .

[0071] The state tracking weight matrix The adaptive update satisfies: ; The control weighting matrix The adaptive update satisfies: ; in As the baseline weight matrix, It is a non-negative adjustment coefficient.

[0072] Preferably, the state tracking weight matrix is ​​arranged according to... Update and control the weighted matrix. .when or When it increases, This increases and prompts the controller to focus more on pose tracking and attitude alignment; at the same time, it improves flight stability and imaging consistency in high curvature regions and observation degradation regions by shortening the prediction time domain and reducing the track propulsion step size.

[0073] According to the above and The updated results are used to adaptively adjust the state tracking weight matrix, control weighting coefficients, and prediction time domain, and to solve for the optimal control quantity under the conditions of satisfying dynamic constraints, actuator constraints, and obstacle avoidance constraints.

[0074] 1. Dynamic constraints: The consistency between control input and state evolution is limited by the six-degree-of-freedom dynamic relationship of the UAV.

[0075] 2. Obstacle avoidance constraint: Construct a potential field function using the real-time point cloud map generated in step S2. When the distance between the UAV and the surface of the component is less than the safety threshold, the potential field generates a repulsive force that causes the trajectory to deviate from the danger zone.

[0076] 3. Actuator limitations: Limits the maximum speed, thrust, and lift rate of change of the motor. The optimal control variables are periodically solved by running numerical optimization algorithms in the edge computing module, driving the UAV to adaptively fly around the surface normal.

[0077] In the aforementioned nonlinear model predictive control (NMPC), the initial state prediction step size is set to a relatively long time, while the control interval is relatively short, and it can be adaptively fine-tuned based on changes in principal curvature and observation consistency deviations. During the solution process, a real-time iterative scheme is adopted, using the solution from the previous time step as the initial value for the current time step, ensuring real-time performance under limited computing power.

[0078] Step S5: Real-time defect detection and incremental scanning on the edge side.

[0079] During the flight maneuver, the global shutter camera continuously acquires high-resolution images and inputs them into a deep convolutional neural network built into the edge computing module. This deep convolutional neural network employs a lightweight backbone network and connects to a feature pyramid network (FPN) after the feature extraction layer to achieve multi-scale defect feature fusion. The network output includes defect category labels (such as cracks, spalling, corrosion, etc.), bounding box coordinates, and a confidence score S.

[0080] Furthermore, establish a task-level feedback mechanism: 1. Normal inspection mode: If the confidence score in the current image sequence is lower than the preset first-level threshold. The drone maintains a predetermined adaptive flight speed. .

[0081] 2. Suspected Defect Detailed Inspection Mode: If a defect is detected that reaches the first-level threshold... The suspected defect caused the system to automatically instruct the flight control module to reduce its circumnavigation speed to [a certain value]. Based on the suspected defect center, the local circumference radius, angular step size, and sampling density are determined according to the defect geometric envelope size. A spiral sampling trajectory with a step size smaller than the conventional fly-around step size is generated within the normal cone angle of the inspected surface. That is, the step size is one-third to one-half of the conventional fly-around step size. The trajectory is discretized to form a dense sequence of observation pose points with multiple angles and close distances to complete the local incremental fine inspection. The sub-pixel level identification of tiny defects is achieved by using the multi-view geometry principle.

[0082] In step S5, a task-level feedback mechanism is established based on the confidence score and geometric envelope information of defect identification. When a suspected defect is identified, the UAV is controlled to perform an incremental scan around the suspected defect area with a step size smaller than the normal fly-around step size.

[0083] Step S6: Defect spatial localization and 3D mapping.

[0084] Defect spatial localization and 3D mapping are performed. This is achieved using the camera intrinsic parameter matrix and real-time calculated UAV pose. The defect center pixel in the image coordinate system The back projection is a ray originating from the camera's optical center. The ray is then numerically solved to obtain the local second-order surface fitted in the preceding steps. By finding the spatial intersection point, the precise three-dimensional coordinates of the defect on the component surface can be obtained. .

[0085] In step S6, based on the camera intrinsic parameter matrix and real-time pose, the defect pixels in the image are back-projected into observation rays, and the intersection points of the observation rays and the local geometric model reconstructed in step S2 are solved to obtain the three-dimensional coordinates of the defect on the surface of the curved component.

[0086] Step S7: Output the structured detection results.

[0087] All detected defect attributes (location, geometry, confidence level, etc.) are correlated in real time to a dynamically constructed surface topology map to form structured detection results, which are then transmitted to the ground station via a wireless broadband data transmission link. Meanwhile, the original high-definition data is stored in an airborne high-reliability solid-state storage array for subsequent in-depth analysis.

[0088] This invention addresses the technical bottlenecks of UAVs in detecting complex curved surface components, such as poor observation consistency, weak trajectory adaptation capability, and disconnect between task and flight control, by constructing a real-time geometric reconstruction mechanism based on multi-source perception fusion, a dynamic pose mapping algorithm, and a task-driven closed-loop control strategy. The system exhibits high robustness and accuracy, significantly improving the reliability and positioning accuracy of defect detection. Simultaneously, NMPC control and real-time obstacle avoidance technology ensure the safe operation of UAVs in complex environments.

[0089] In practical implementation, rapid adaptation is achieved for different curved surface components (such as pressure vessels or aerospace panels) by adjusting parameters (such as voxel side length, curvature reference value, NMPC weight matrix, etc.). For example, when dealing with components with large curvature, attitude alignment and distance maintenance constraints are enhanced to improve the sampling frame rate; while when dealing with flat areas, field of view overlap constraints and inspection efficiency are adjusted.

[0090] The core logic of this invention, namely adaptive control based on real-time geometric perception and task-driven refined detection, is applicable to various special robot operation fields. Through multi-source sensor data fusion and real-time edge decision-making, it achieves an optimal balance between detection accuracy and operational efficiency.

[0091] The embodiments described above are for illustrative purposes only and are not intended to limit the invention. Therefore, any changes in numerical values ​​or substitutions of equivalent elements should still fall within the scope of this invention.

[0092] The above detailed description will enable those skilled in the art to understand that the present invention can indeed achieve the aforementioned objectives and has complied with the provisions of the Patent Law.

[0093] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention. The above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

[0094] It should be noted that the above description of the process is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to the process under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0095] The basic concepts have been described above. Obviously, for those skilled in the art who have read this application, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore, such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this application.

[0096] Furthermore, this application uses specific terms to describe its embodiments. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different positions in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.

[0097] Furthermore, those skilled in the art will understand that aspects of this application can be described and illustrated through several patentable types or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Therefore, aspects of this application can be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. All of the above hardware or software can be referred to as a “unit,” “module,” or “system.” Furthermore, aspects of this application can take the form of a computer program product embodied in one or more computer-readable media, wherein computer-readable program code is contained therein.

[0098] The computer program code required for the operation of each part of this application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C, VB.NET, and Python; general programming languages ​​such as C; Visual Basic, Fortran2103, Perl, COBOL2102, PHP, and ABAP; dynamic programming languages ​​such as Python, Ruby, and Groovy; or other programming languages. This program code can run entirely on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as Software as a Service (SaaS).

[0099] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this application are not intended to limit the order of the processes and methods of this application. Although some currently considered useful embodiments of the invention have been discussed in the foregoing disclosure by way of various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the substance and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a purely software solution, such as an installation on an existing server or mobile device.

[0100] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this approach of the present application should not be construed as reflecting an intention that the claimed subject matter requires more features than expressly recited in each claim. Rather, the subject of the invention should possess fewer features than in any single embodiment described above.

Claims

1. A method for real-time defect detection of curved surface components using adaptive UAV flight, characterized in that, include: Step S1: Perform time synchronization and spatial calibration on the lidar, camera and inertial measurement unit mounted on the UAV to acquire spatiotemporally synchronized lidar point cloud data and image data; Step S2: Based on the laser point cloud data, reconstruct the local geometric model of the inspected curved surface component in real time, and calculate the unit normal vector and principal curvature information at each sampling point on the target surface according to the local geometric model; Step S3: Based on the unit normal vector and principal curvature information, construct the UAV expected pose sequence that conforms to the optical axis alignment constraint, observation distance constraint and field of view overlap constraint, and generate the observation consistency deviation amount according to the optical axis angle deviation term, observation distance deviation term and field of view overlap rate deviation term driven by the principal curvature information dynamic weight. Step S4: Use the principal curvature information and the observation consistency deviation as feedback inputs for nonlinear model predictive control, and use the desired pose sequence as a reference trajectory to control the UAV to perform adaptive flight around the surface of the curved component. The adaptive flight around adopts a conventional flight around step size preset according to the geometric complexity of the curved component. Step S5: During the flight around the camera, defect identification is performed on the image data, and when a suspected defect is identified, the UAV is controlled to perform incremental scanning with an incremental step size smaller than the conventional flight around step size. Step S6: Back-project the identified defect pixels as observation rays, and calculate the intersection of the observation rays with the local geometric model to obtain the three-dimensional coordinates of the defect on the surface of the curved component.

2. The real-time defect detection method for curved surface components using UAV adaptive flight as described in claim 1, characterized in that, In step S1, time synchronization and spatial calibration are performed using a unified clock source to synchronize and trigger the lidar, camera, and inertial measurement unit, and the spatial coordinate transformation relationship between the lidar, camera, and inertial measurement unit is established through offline calibration. During flight, motion compensation is performed on the lidar point cloud data and image data using the data from the inertial measurement unit.

3. The real-time defect detection method for curved surface components using adaptive UAV flight as described in claim 1, characterized in that, The real-time reconstruction of the local geometric model in step S2 includes: statistical filtering to remove outliers from the laser point cloud data; dynamically determining the voxel grid side length based on the principal curvature information; performing voxel grid downsampling on the point cloud after removing outliers; and using a moving least squares method with a Gaussian weighting function to perform local surface fitting on the downsampled point cloud to reconstruct the local geometric model and solve for the unit normal vector and the principal curvature information.

4. The real-time defect detection method for curved surface components using adaptive UAV flight as described in claim 1, characterized in that, The construction of the UAV desired pose sequence in step S3 satisfies the following constraints: the optical axis vector of the camera is antiparallel to the unit normal vector at the sampling point on the target surface; the distance from the optical center of the camera to the sampling point is within a preset object distance range; the image field of view overlap rate corresponding to adjacent desired poses is not lower than a preset threshold; the observation consistency deviation is obtained by weighting and normalizing the optical axis angle deviation, the observation distance deviation, and the field of view overlap rate deviation according to the dynamic weight; the dynamic weight increases the weight of the optical axis angle deviation and the observation distance deviation when the local curvature represented by the principal curvature information increases, and increases the weight of the field of view overlap rate deviation when the local curvature decreases.

5. The real-time defect detection method for curved surface components using adaptive UAV flight as described in claim 1, characterized in that, In step S4, the nonlinear model predictive control uses the desired pose sequence as a reference trajectory and the principal curvature information and the observation consistency deviation as feedback quantities to adaptively update the state tracking weight matrix and control weight matrix in the nonlinear model predictive control. Under the conditions of satisfying the UAV dynamics constraints, actuator amplitude limiting constraints, and obstacle avoidance constraints constructed based on the local geometric model, the optimal control quantity is solved to drive the UAV to fly.

6. The real-time defect detection method for curved surface components in adaptive UAV flight according to claim 5, characterized in that, The state tracking weight matrix increases according to the increase of the principal curvature information and the observation consistency deviation, so as to enhance the control weight of pose tracking and attitude alignment. The control weighting matrix increases as the observation consistency deviation increases to suppress drastic changes in the control quantity; the prediction time domain of the nonlinear model predictive control is shortened as the principal curvature information increases.

7. The real-time defect detection method for curved surface components using adaptive UAV flight as described in claim 1, characterized in that, After calculating the unit normal vector in step S2, normal vector consistency correction is also performed: check the cosine value of the angle between the normal vectors of adjacent sampling points. If the angle is greater than the preset angle threshold, the normal vector of the high signal-to-noise ratio region is used as the seed point, and the normal vector of the local discontinuous region is weighted, redirected and smoothed through the normal vector propagation algorithm to ensure the continuity of the normal vector field in space.

8. The real-time defect detection method for curved surface components in adaptive UAV flight according to claim 1, characterized in that, The defect identification in step S5 is achieved by a deep convolutional neural network running in the edge computing module; the deep convolutional neural network outputs the defect category label, bounding box coordinates and confidence score; when the confidence score reaches a preset first-level threshold, it is determined that the suspected defect has been identified.

9. The real-time defect detection method for curved surface components using adaptive UAV flight as described in claim 8, characterized in that, The incremental scanning in step S5 includes: taking the center of the suspected defect as a reference, determining the local circumference radius, angular step size and sampling density according to the geometric envelope size of the suspected defect, generating a spiral sampling trajectory with a step size of one-third to one-half of the conventional circumference step size within the normal cone angle of the surface of the curved component, discretizing the spiral sampling trajectory into a dense observation pose point sequence, and controlling the UAV to execute the dense observation pose point sequence sequentially to complete the local detailed inspection.

10. The real-time defect detection method for curved surface components using adaptive UAV flight as described in claim 1, characterized in that, The generation of the observation ray in step S6 includes: back-projecting the center pixel coordinates of the bounding box of the defect into a three-dimensional ray originating from the optical center of the camera based on the intrinsic parameters and real-time pose of the camera; calculating the intersection point using a numerical intersection method to solve for the intersection point of the three-dimensional ray with the surface equation of the local geometric model, and using the spatial coordinates of the intersection point as the three-dimensional coordinates of the defect on the surface of the curved component; after obtaining the three-dimensional coordinates, associating the defect's category label, bounding box size, confidence score, and the three-dimensional coordinates with the dynamically constructed surface topology map, and transmitting it to the ground station via a wireless data transmission link to form a structured detection result.

Images