Unmanned aerial vehicle bridge coverage detection trajectory planning method
By constructing a trajectory prediction model and a time-optimized trajectory dynamic trajectory, the problems of low detection efficiency of UAV bridges in the prior art and inability to cope with complex three-dimensional structures are solved, and efficient and autonomous bridge coverage detection trajectory planning is achieved.
Patent Information
- Application Number
- CN202510174321.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-17
- Publication Date
- 2025-05-27
AI Technical Summary
The existing UAV bridge detection methods mainly rely on manual manipulation, have low detection efficiency and cannot effectively cope with the coverage path planning of complex three-dimensional structures, and cannot use local information collected by sensors for detection.
By constructing a trajectory prediction model, using the drone parameters and dynamic models, combining deep learning and self-attention modules, a time-optimized trajectory dynamic trajectory is constructed to realize the dynamic parameter planning of the drone during flight, and independently plan the coverage detection trajectory.
It realizes the coverage detection of complex three-dimensional structures without pre-determined three-dimensional information, improves trajectory solution efficiency, can plan trajectories online, and improves detection efficiency and coverage integrity.
Smart Images
Figure CN120044967A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of UAV trajectory planning, and particularly to a UAV bridge coverage detection trajectory planning method. Background Art
[0002] In recent years, the country has vigorously developed the low-altitude economy. As an agile flying vehicle, UAVs have shown great application potential in the fields of national economy and even military. At present, in the process of China's transformation from a large infrastructure country to a strong infrastructure country, a large number of in-service bridges need to be inspected and maintained. UAVs have been widely used in bridge inspection scenarios to reduce inspection costs and ensure the safety of inspection personnel. However, the existing UAV inspection methods mainly rely on pilots to manually control UAVs to fly to designated points for local inspection and evaluation. The inspection efficiency is low, highly dependent on manual experience, and has a high risk of missed inspection. Using UAVs for all-round and autonomous bridge condition inspection and evaluation is a current research hotspot.
[0003] The existing methods mainly use heuristic-based methods to predict the coverage path. However, the advantage of heuristic-based methods is simplicity and high efficiency, while the disadvantage is that due to the simplicity of manually set rules, they cannot plan suitable coverage paths for complex three-dimensional structures and cannot complete coverage detection planning based on local information collected by current sensors. Summary of the Invention
[0004] Aiming at the deficiencies in the prior art, the present invention provides a UAV bridge coverage detection trajectory planning method, which solves the problem that the heuristic-based method for predicting the coverage path in the prior art cannot plan a suitable coverage path for complex three-dimensional structures based on local information collected by current sensors.
[0005] According to an embodiment of the present invention, a UAV bridge coverage detection trajectory planning method includes:
[0006] Construct a trajectory prediction model and mount it on the UAV;
[0007] The UAV imports its own UAV parameters into the trajectory prediction model to obtain a flight trajectory profile;
[0008] According to the UAV dynamics parameters and the flight trajectory profile, construct a time-optimal trajectory dynamics trajectory, and then the UAV moves according to the flight trajectory profile and the time-optimal trajectory dynamics trajectory.
[0009] Preferably, the construction method of the trajectory prediction model is as follows:
[0010] S1: Obtain a bridge image set and the corresponding camera observation positions, and convert the bridge image set into a training point cloud set;
[0011] S2: Query the voxel occupancy information of all point clouds in the training point cloud set, perform inverse sampling on the voxel occupancy information of all point clouds to obtain a set of sampled points;
[0012] S3: Calculate the coverage gain of all sampled points in the set of sampled points, and take the sampled point with the largest coverage gain within a preset range from the current camera position as the new current camera position, then repeat steps S1 - S3 until the loss function is minimized to obtain the trajectory planning model.
[0013] Preferably, the method for obtaining the image set of the bridge is as follows:
[0014] Create the first 3D model of the bridge, set the current camera position, and observe the first 3D model from the current camera position to obtain the observed images;
[0015] Extract the second 3D model of the bridge from the ShapeNet dataset, and at the current camera position, observe the second 3D model from the current camera position to obtain the supplementary images;
[0016] Integrate the observed images and the supplementary images to obtain the bridge image set.
[0017] Preferably, the method for converting the bridge image set into a training point cloud set is as follows:
[0018] Use the residual network layer to extract the image features of all images in the observed image set and the supplementary image set, respectively obtaining the observed image features and the supplementary image features;
[0019] According to the pose of the current camera, transform the supplementary image features into the coordinate system of the observed image;
[0020] Take the L1 norm of the observed image features and the supplementary image features as the loss function, and use the network model of U - Net to predict the depth information of the observed image to obtain the predicted depth map of the observed image;
[0021] According to the pose of the current camera, project the predicted depth map into the 3D space and perform filtering fusion with the training point cloud set at the previous repetition to obtain the training point cloud set at the current moment.
[0022] Preferably, the method for querying the voxel occupancy information of all point clouds in the training point cloud set includes:
[0023] Perform multiple downsamplings on the training point cloud set, and input the samples after each downsampling into the self - attention module SAN for screening;
[0024] Integrate the screening results corresponding to each downsampling, and perform encoding mapping using the multi - layer perceptron MLP to obtain the voxel occupancy information of each point cloud.
[0025] Preferably, the occupancy information of the primal body Occup(p) ∈ [0, 1], where 0 indicates that the sampling point is not occupied and 1 indicates that the sampling point is occupied.
[0026] Preferably, the construction method of the time-optimal trajectory dynamic trajectory is as follows:
[0027] Set the optimization objective function with the goal of the fastest growth rate of the UAV's moving distance;
[0028] Construct the UAV dynamic parameters according to the performance parameters of the UAV;
[0029] Input the optimization objective function and the UAV dynamic parameters into the Acados optimizer for iterative optimization, obtain multiple time-optimal trajectory points, and fit all the time-optimal trajectory points to obtain the time-optimal trajectory dynamic trajectory.
[0030] Preferably, when the maximum number of iterative optimizations is reached or the error descent rate is less than the preset value, stop the iterative optimization.
[0031] Preferably, the optimization objective function is as follows:
[0032]
[0033] s.t.
[0034]
[0035] x ∈ Λ
[0036] u ∈ Ω
[0037] φ ∈ [0, l arc
[0038] where φ k is the distance between the trajectory point and the initial position, p is the sampling point, the constraint Ω of the input quantity and the feasible region Λ of the state quantity are given according to the actual situation of the UAV, and the system model is the dynamic differential model of the UAV, is the derivative of φ k K is the coefficient matrix, and {Q c , Q l , Q x , Q u , q, α} are the weight matrices and weight coefficients of the relevant loss terms.
[0039] Compared with the prior art, the present invention has the following beneficial effects:
[0040] By constructing a three-dimensional model of the bridge, taking images, and combining them with bridge images in the public database to augment the training data, a prediction contour is obtained by using depth prediction and calculating the coverage gain. Therefore, there is no need to pre-give three-dimensional information, it can handle the covered three-dimensional structure, no longer rely on the local information collected by sensors, the trajectory solving efficiency is relatively high, and the effect of online planning can be achieved. In addition, according to the dynamic parameters of the UAV, a time-optimal trajectory dynamic trajectory is constructed to plan the dynamic parameters of the UAV during flight, enabling it to quickly perform coverage detection on the bridge at the fastest and optimal speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 The flowchart of the trajectory planning according to the embodiment of the present invention.
[0042] Figure 2 The planned coverage path and the generated point cloud map according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0043] The technical solutions in the present invention will be further described below with reference to the drawings and embodiments.
[0044] As Figure 1 shown, the embodiment of the present invention proposes a method for trajectory planning of UAV bridge coverage detection, including:
[0045] Construct a trajectory prediction model and mount it on the UAV;
[0046] The construction method of the trajectory prediction model is as follows:
[0047] S1: Obtain a set of bridge images and the corresponding camera observation positions, and convert the set of bridge images into a training point cloud set;
[0048] Set up a virtual simulation scenario, import three-dimensional structure information such as bridges and buildings. In this simulation environment, the first three-dimensional model of the bridge mainly consists of a surface mesh Mesh and an appearance material. The appearance material is attached to the surface mesh. By setting the current camera position, RGB image data (i.e., the observation image) of the camera observing the bridge is generated at a specific perspective. And the Mesh can be used as the ground truth of the scene three-dimensional structure information after projection transformation, and is used as the supervision signal of the depth prediction module to train a depth prediction network model specifically for the bridge detection scenario.
[0049] Use the ShapeNet dataset as the public training dataset, extract the second three-dimensional model of the bridge therein, and observe the second three-dimensional model at the current camera position to obtain supplementary images to supplement the training data and improve the generalization ability of the network in different scenarios.
[0050] After preprocessing and integrating the observation image and the supplementary image, a bridge image set is obtained.
[0051] After that, a pre-trained residual network layer is used to extract the features of the observation image and the supplementary image, obtaining the observation image features and the supplementary image features respectively. Then, a series of depth planes perpendicular to the optical axis of the image are defined. The supplementary image features f j , j∈{1, 2,..., m} are transformed into the coordinate system of the observation image I t using the pose T t of the camera.
[0052] The L1 norm loss is calculated between the transformed supplementary image features and the observation image features, and this result is used as the loss function of the depth model. The depth information of the observation image is predicted using the depth network model of U-Net to obtain the observation depth map. Finally, the predicted observation depth map is projected into the three-dimensional space using the current camera pose and filtered and fused with the point cloud information S t-1 obtained in the previous repetition. The regions with large image gradients in the depth map are filtered out, and the blank regions are filled using smoothing, thereby establishing the reconstructed training point cloud set S t for this repetition, as Figure 2 shown.
[0053] S2: Query the voxel occupancy information of all point clouds in the training point cloud set, and perform inverse sampling on the voxel occupancy information of all point clouds to obtain a set of sampling points;
[0054] Construct a voxel occupancy prediction module, which can implicitly represent whether a certain point position in the scene is occupied or not, represented by a value in the range of [0, 1], where 1 means occupied and 0 means unoccupied. The voxel occupancy prediction module takes the training point cloud set S t , the camera pose T t , and the query point cloud p as the model input, and then predicts a scalar value in the range of [0, 1], indicating whether the queried 3D position p is occupied or not at the current camera position T t .
[0055] In the present invention, the training point cloud set S t is downsampled four times, and the samples after each downsampling are output to a self-attention module SAM and encoded and mapped using a multi-layer perceptron MLP to obtain the output result Occup(p)∈[0, 1]. The self-attention network collects the K-nearest neighbor point clouds of the current point position p in each downsampled sample As input, this network can encode voxel occupancy at different scales, thereby supporting voxel occupancy prediction for large-scale infrastructure such as long-span bridges, and further supporting coverage planning for large-scale scenarios. The supervision signal is a signal indicating whether the current point p is occupied, judged based on the true point cloud information observable at the current camera position.
[0056] S3: Calculate the coverage gain of all sampling points in the sampling point set, and take the sampling point with the maximum coverage gain within a preset range from the current camera position as the new current camera position. Then repeat steps S1 - S3 until the loss function is minimized to obtain the trajectory planning model.
[0057] First, inverse sample the sampling points with known voxel occupancy information to obtain a set p of N discrete sampling points k , and then for each sampling point, encode its historical camera observation position as spherical harmonic information Sphere(p k ), and input all the encoded information into a Transformer framework to predict the coverage gain at this position. The supervision information is still obtained from the true 3D mesh. Finally, the coverage gains at all sampling positions are aggregated into the overall coverage gain at the query camera position. After obtaining the coverage gain of each sampling point, each time M new camera positions to be moved are sampled near the current camera position, and then the position with the maximum coverage gain rate is selected, and the camera is moved to this position. Repeat steps S1 - S3, perform a large amount of training on the virtual environment and public datasets, and perform cross-validation until the loss function is minimized and the performance meets the requirements to obtain the trajectory planning model.
[0058] The drone imports its own drone parameters into the trajectory prediction model to obtain the flight trajectory profile;
[0059] After obtaining the trajectory planning model, import the trained trajectory planning model into the upper computer of the drone. The upper computer uses Nvidia Jetson Nx, which has a GPU that can support neural network inference, so it is suitable for running the trajectory planning model of the present invention. Then install a gimbal detection camera on the drone, calibrate the parameters of the drone and the detection camera and import them into the trajectory planning model. The drone pilot flies the drone to the bridge and adjusts the attitude of the gimbal camera to align it with the bridge surface.
[0060] Use the trajectory planning model to predict the next target camera position, and accumulate and update the local point cloud information during the process, filter out the depth prediction values with large gradients, and perform smoothing operations on the predicted depth information using smoothing filtering. Record the camera trajectory passed between the current camera position and the predicted camera position, and integrate it into the set of historical camera positions, so as to fit out a flight trajectory profile.
[0061] According to the dynamic parameters and flight trajectory profile of the UAV, a time-optimal trajectory dynamic trajectory is constructed, and then the UAV moves according to the flight trajectory profile and the time-optimal trajectory dynamic trajectory.
[0062] After obtaining the point with the maximum camera coverage gain rate each time, first, the pan-tilt camera adjusts its own attitude to meet the requirements of the target point for the camera attitude. Subsequently, the upper computer automatically converts the target camera position information into the UAV's own coordinate system according to the parameters of the UAV and the detection camera, and then guides the UAV to the new position, so that both the position and the attitude meet the requirements given by the coverage path planning algorithm.
[0063] During the guidance process, first, a polynomial trajectory is interpolated between the current UAV position and the next target UAV position. This trajectory mainly gives the flight trajectory profile that the UAV should follow. Since the flight trajectory profile only contains contour information and is not a time trajectory that the UAV can directly execute, and does not contain indicators such as the UAV's movement speed. In addition, during the UAV coverage detection process, in order to improve the detection efficiency, the UAV should move from the current position to the next position as fast as possible. Therefore, according to the dynamic model of the UAV, a time-optimal dynamic trajectory needs to be predicted for the UAV to execute.
[0064] According to the requirement of the optimal time, a nonlinear least squares optimization problem is constructed. First, we assume that the length of the reference trajectory already executed by the current UAV is Its value can be accurately obtained by projecting the current UAV position onto the reference path. However, in the optimization task, it cannot be directly calculated and an approximate value φ k needs to be given and iteratively updated during the optimization. Thus, the optimal time objective can actually be transformed into solving the dynamic trajectory that makes the value of φ k increase the fastest. For this purpose, we set the following optimization objective function:
[0065]
[0066] s.t.
[0067]
[0068] x ∈ Λ
[0069] u ∈ Ω
[0070] φ ∈ [0, l arc
[0071] where φ k is the distance between the trajectory point and the initial position, p is the sampling point, the constraint Ω of the input quantity and the feasible region Λ of the state quantity are given according to the actual situation of the UAV, and the system model This is the dynamic differential model of the UAV. This is the derivative of φ k , where K is the coefficient matrix, and {Q c , Q l , Q x , Q u , q, α} is the weight matrix and weight coefficients of the relevant loss terms.
[0072] From the form of the objective function, it can be seen that this optimization problem is a special non - linear least - squares optimization and can be efficiently solved by a standard optimizer.
[0073] For the differential equation of the UAV dynamic model in the example application scenario of the present invention, it can be expressed as follows:
[0074]
[0075] Among them, {p, q, v, w, T} are the position, attitude, linear velocity, angular velocity, and thrust of the UAV respectively, as shown in Table 1, and τ is the torque. The state variables are set as The input variables are set as
[0076] Table 1 Parameters of the UAV used
[0077]
[0078]
[0079] The constructed optimization problem is sent to the Acados optimizer for iterative solution. In each iteration inside the optimizer, the non - linear least - squares problem is locally linearized and solved in the form of a linear least - squares problem. HPIPM is selected as the solver in the Acados optimizer, and the Gauss - Newton method is used to approximate the Hessian matrix. The optimizer stops optimizing when the number of iterations reaches 20 times or when the error descent rate is less than 1e - 10. In this way, the state prediction values at N + 1 time steps and the input prediction values within N time steps are obtained. To ensure accuracy and numerical stability, we only take the input at the current moment and the state at the next moment as the time - optimal trajectory points for planning and record them until reaching the end point of the flight trajectory profile. Then, stop planning and output all the previously recorded time - optimal trajectory points in chronological order as the final time - optimal trajectory dynamic trajectory. Then, the UAV moves according to the flight trajectory profile and the time - optimal trajectory dynamic trajectory.
[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the purpose and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.
Claims
1. A UAV bridge coverage detection trajectory planning method, characterized by: include: Build a trajectory prediction model and put it on the drone; The UAV imports its own UAV parameters into the trajectory prediction model to obtain the flight trajectory profile; According to the UAV dynamic parameters and the flight trajectory profile, a time-optimal trajectory dynamic trajectory is constructed, and then the UAV moves according to the flight trajectory profile and the time-optimal trajectory dynamic trajectory.
2. A method for planning a trajectory for bridge coverage detection by an unmanned aerial vehicle according to claim 1, characterized in that: The trajectory prediction model is constructed as follows: S1: Obtain a bridge image set and the corresponding camera observation positions, and convert the bridge image set into a training point cloud set; S2: query the body occupancy information of all point clouds in the training point cloud set, perform inverse sampling on the body occupancy information of all point clouds, and obtain a set of sampling points; S3: Calculate the coverage gain of all sampling points in the sampling point set, and take the sampling point with the largest coverage gain within a preset range from the current camera position as the new current camera position, and then repeat steps S1-S3 until the loss function is minimized to obtain the trajectory planning model.
3. The method for planning a trajectory for bridge coverage detection by an unmanned aerial vehicle according to claim 1, characterized in that: The method for obtaining the image set of the bridge is as follows: Making a first three-dimensional model of the bridge, setting a current camera position, observing the first three-dimensional model at the current camera position, and obtaining an observed image; Extract a second 3D model of the bridge from the ShapeNet dataset, and observe the second 3D model at the current camera position to obtain a supplementary image; The observed images and supplementary images are integrated to obtain the bridge image set.
4. The method for planning a trajectory for bridge coverage detection by an unmanned aerial vehicle according to claim 1, characterized in that: The method of converting the bridge image set into a training point cloud set is as follows: The residual network layer is used to extract the image features of all images in the observation image and the supplementary image set, and the observation image features and the supplementary image features are obtained respectively; According to the current camera pose, the supplementary image features are converted to the coordinate system of the observed image; The L1 norm of the observed image features and the supplementary image features is used as the loss function, and the U-Net network model is used to predict the depth information of the observed image to obtain the predicted depth map of the observed image; According to the current camera pose, the predicted depth map is projected into three-dimensional space and filtered and fused with the training point cloud set at the last repetition to obtain the training point cloud set at the current moment.
5. The method for planning a trajectory for bridge coverage detection by an unmanned aerial vehicle according to claim 1, characterized in that: Methods for querying the body occupancy information of all point clouds in the training point cloud set include: The training point cloud set is downsampled multiple times, and the samples after each downsampling are input into the self-attention module SAN for screening; The screening results corresponding to each downsampling are integrated, and the multi-layer perceptron MLP is used for encoding and mapping to obtain the body occupancy information of each point cloud.
6. The method for planning a trajectory for bridge coverage detection by an unmanned aerial vehicle according to claim 1, characterized in that: The element body occupancy information Occup(p)∈[0,1], wherein 0 indicates that the sampling point is not occupied, and 1 indicates that the sampling point is occupied.
7. The method for planning a trajectory for bridge coverage detection by an unmanned aerial vehicle according to claim 1, characterized in that: The construction method of the time-optimal trajectory dynamics trajectory is as follows: The optimization objective function is set with the goal of increasing the UAV's moving distance as the fastest; Construct the UAV dynamic parameters based on the UAV performance parameters; The optimization objective function and UAV dynamics parameters are input into the Acados optimizer for iterative optimization to obtain multiple time-optimal trajectory points. All time-optimal trajectory points are fitted to obtain the time-optimal trajectory dynamics trajectory.
8. The method for planning a trajectory for bridge coverage detection by an unmanned aerial vehicle according to claim 1, characterized in that: When the maximum number of iterative optimization times is reached or the error decrease rate is less than the preset value, the iterative optimization is stopped.
9. The method for planning a trajectory for bridge coverage detection by an unmanned aerial vehicle according to claim 1, characterized in that: The optimization objective function is as follows: st x∈Λ u∈Ω φ∈[0,l arc ] Among them, φ k is the distance between the trajectory point and the initial position, p is the sampling point, the input constraint Ω and the feasible domain Λ of the state quantity are given according to the actual situation of the UAV, and the system model is the dynamic differential model of the UAV, Then φ k The derivative of , K is the coefficient matrix, and {Q c ,Q l ,Q x ,Q u ,q,α} are the weight matrix and weight coefficient of the relevant loss terms.