Unmanned aerial vehicle intelligent inspection path optimization method based on multi-source perception

Abstract

The present application relates to unmanned aerial vehicle intelligent inspection path optimization method, especially to unmanned aerial vehicle intelligent inspection path optimization method based on multi-source sensing, belongs to the technical field of power inspection.The present application includes obtaining the synchronous data stream of airborne sensor;Fusion laser point cloud generates occupation grid and Euclidean signed distance field ESDF, identifies obstacles, superimposes into semantic risk layer, obtains layered world model by tracking and predicting dynamic body;Obtain the initial inspection path;Adopt conditional generative adversarial network cGAN, generate candidate observation pose meeting the shooting requirements;Model the pose quality evaluation as a sequential decision problem, optimize and obtain the optimal pose combination;Map the path planning problem to the graph structure, optimize the path and obtain the globally optimal route.The present application solves the problems of data island and precision loss in inspection, realizes fast response through local update of the affected area, and continuously optimizes the planning and decision quality according to the inspection feedback.

Description

Technical Field

[0001] This invention relates to a method for optimizing intelligent inspection paths for unmanned aerial vehicles (UAVs), and more particularly to a method for optimizing intelligent inspection paths for UAVs based on multi-source perception, belonging to the field of power line inspection technology. Background Technology

[0002] With the rapid construction and large-scale deployment of large-scale infrastructure projects such as power lines, photovoltaic power stations, oil and gas pipelines, and transportation bridges and tunnels, inspection work is facing unprecedented new challenges. Inspection areas are constantly expanding, target equipment is becoming increasingly dense, environmental risks remain high, and the demands for timeliness are also rising. Traditional manual inspection methods are inefficient, costly in terms of manpower, and easily affected by external conditions such as rugged terrain and variable weather. More importantly, these methods often involve multiple safety risks, including working at heights, operating live equipment, and traversing complex terrain, posing significant personal safety hazards.

[0003] Therefore, drone inspection technology has gradually emerged and become a mainstream method in the industry in recent years. Drones enable efficient coverage of vast areas, complex terrains, and hazardous environments, significantly improving inspection speed and data acquisition capabilities. However, in practical application, especially in complex scenarios such as mountainous areas, coastal areas, densely populated urban areas, and strong electromagnetic environments, drone inspection still faces a series of systemic challenges, which can be summarized in four aspects: First, "safe flight," meaning ensuring flight stability and safety in adverse weather, environments with dense obstacles, and limited navigation signals; second, "clear images," meaning acquiring high-quality, high-definition inspection images and data under conditions of high-speed movement, changing lighting, and complex target backgrounds; third, "comprehensive inspection," meaning achieving comprehensive optimization of inspection paths to avoid missed inspections and cover blind spots that are difficult for humans to reach; and fourth, "timely calculation," meaning rapid processing, intelligent analysis, and real-time fault identification of massive amounts of inspection data to support closed-loop decision-making. These challenges collectively constitute the key bottleneck in the evolution of current drone inspection technology from "usable" to "easy to use, reliable, and intelligent."

[0004] Traditional UAV path planning employs a fragmented, sequential process of "flight path first, observation later." This involves first planning a flyable path that meets flight dynamics, obstacle avoidance, and endurance constraints, and then assigning appropriate observation angles (poses) to specific inspection targets along this path. This decoupled approach has significant drawbacks: the planned flight path may be optimal from a flight perspective, but it could lead to poor sensor visibility of critical parts of the target, severe obstructions (such as power lines being blocked by towers), or frequent hovering, adjustments, and reshoots due to unreasonable viewing angles. This severely hinders overall inspection efficiency and significantly reduces the effectiveness of "automated inspection." On the other hand, the multi-source sensors integrated into drones, such as RGB (Red Green Blue), IR (Infrared Spectroscopy), LiDAR (Light Detection and Ranging), depth, IMU (Inertial Measurement Unit), and RTK (Real-time kinematic), suffer from problems such as asynchrony, noise, drift, loss of lock, and changes in lighting in real-world scenarios. The external data they rely on, including DEM (Digital Elevation Model), 3D models, no-fly zones, and weather data, also suffer from issues of timeliness and accuracy. If these multi-source, heterogeneous, and multi-precision data from both internal and external sources cannot be deeply fused within a unified spatiotemporal coordinate system, and their reliability cannot be modeled, it will lead to a "fragmentation" in the drone's perception of itself and its environment: discontinuous positioning trajectories, inconsistent and unreliable environmental maps, easily causing discontinuous positioning, inconsistent maps, and planning failures.

[0005] In task and constraint modeling, existing methods are mostly based on two-dimensional maps or simplified three-dimensional obstacle models. Their optimization objectives are often limited to "shortest path" or "shortest time," which cannot fully express and understand the rich semantics and dynamic characteristics of inspection scenarios. For example, it is difficult to distinguish the huge differences in the risk levels of different targets such as power lines, insulators, towers, surrounding trees, and people below at the safety and task levels, and it cannot effectively incorporate the spatiotemporal coupling constraints brought about by dynamic obstacles such as vehicles, pedestrians, and birds. However, the core of inspection tasks is "clear and accurate observation," and its data quality is highly dependent on the observation geometry, including the relative viewpoint between the sensor and the target, observation distance, occlusion, image overlap rate, and the ground sample distance (GSD) that ultimately determines the resolution. Ignoring these semantic and geometric constraints that are directly related to the task will inevitably lead to planned paths that are "flyable" but "unusable." Therefore, it is urgent to solve the above problems and shortcomings. Summary of the Invention

[0006] To address the problems and shortcomings of the existing technologies, this invention provides a method for optimizing intelligent inspection paths for unmanned aerial vehicles (UAVs) based on multi-source perception. By constructing a hierarchical world model, a richer and more accurate environmental perception is formed. When generating flight paths, the system can simultaneously process multi-level optimization constraints, support incremental updates and local replanning, and continuously optimize decision quality based on inspection feedback, thereby achieving intelligent inspection path optimization for UAVs.

[0007] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0008] The method for optimizing the intelligent inspection path of UAVs based on multi-source perception includes the following steps:

[0009] Acquire airborne sensor data streams and external data, perform preprocessing, and obtain synchronized data streams in a unified coordinate system;

[0010] Based on synchronous data streams in a unified coordinate system, laser point clouds are fused, semantic segmentation networks are used to identify obstacles, and dynamic bodies are tracked and predicted to obtain a hierarchical world model.

[0011] Based on the hierarchical world model and task requirements, obtain the initial inspection path;

[0012] Based on the initial inspection path, a conditional generative adversarial network (cGAN) is used to generate candidate observation poses that meet the shooting requirements, based on the inspection target and environmental constraints.

[0013] By using a reinforcement learning filter, pose quality assessment is modeled as a sequential decision problem, and candidate observation poses are optimized under preset constraints to obtain the optimal pose combination.

[0014] Based on the initial inspection path and the optimal pose combination, the path planning problem is mapped to a graph structure, and the path is optimized to obtain the globally optimal route.

[0015] Furthermore, the acquisition of airborne sensor data streams and external data includes the following steps:

[0016] The system acquires data streams from airborne sensors, processes the data using a unified time reference, and corrects the timestamps of each sensor to compensate for fixed delays and jitter, using the time of the Global Navigation Satellite System as a reference.

[0017] Pull external data that matches the task area from external systems, and slice and cache low-frequency external data according to validity period and spatial range to ensure fast access during planning and replanning.

[0018] After performing the above steps, a raw multi-source dataset with time alignment and range matching is obtained.

[0019] Furthermore, the preprocessing to obtain the synchronized data stream in a unified coordinate system includes the following steps:

[0020] Implement intrinsic and extrinsic parameter management and coordinate transformation link fixing for each sensor;

[0021] Establish transformation relationships from various sensor coordinate systems to a unified world coordinate system;

[0022] The required external data is also resampled to a unified world coordinate system and topologically consistent with the no-fly zone boundary, and finally a synchronized data stream is output.

[0023] Furthermore, the process of fusing laser point clouds with synchronous data streams in a unified coordinate system, identifying obstacles using a semantic segmentation network, and tracking and predicting dynamic objects to obtain a hierarchical world model includes the following steps:

[0024] The laser point cloud at each moment is distorted and its coordinates are transformed in the corresponding pose, then projected onto the world coordinate system and fused across multiple frames.

[0025] Let the original coordinates of the i-th laser point in the sensor coordinate system be... The measurement time is t i The reference time is t. ref The expression is as follows:

[0026] ;

[0027] In the formula, The coordinates of the i-th laser point after motion compensation are denoted as . From time t i to reference time t ref The rotation transformation matrix; From time t i to reference time t ref Translation vector;

[0028] Where the rotation matrix Translation vector Obtained through linear interpolation:

[0029] ;

[0030] Among them, Slerp(R) ti ,R tref ,α) is a spherical linear interpolation function used for smooth interpolation between two rotation matrices; α is the interpolation parameter, ranging from [0,1], representing the relative position within the scanning time window at the current moment; t start t represents the start time of the current scan frame. end This represents the end time of the current scan frame;

[0031] The distortion-corrected laser point cloud is transformed from the sensor coordinate system to the world coordinate system, as shown in the following expression:

[0032] ;

[0033] in, Let be the final coordinates of the i-th laser point in the world coordinate system; This is the rotation matrix from the sensor coordinate system to the world coordinate system; This is the translation vector from the sensor coordinate system to the world coordinate system;

[0034] The fusion process employs a ray casting and probability update mechanism: the space between the sensor and the measurement point is updated to free, and the voxels hit by the measurement are updated to occupied. For each laser point... From sensor location All voxels on the ray leading to the target point are updated using the 3D Bresenham algorithm, as shown in the following expression:

[0035] ;

[0036] Among them, v j s represents the world coordinate position of the j-th voxel on the ray; world This indicates the sensor's position in the world coordinate system; j is the voxel index along the ray direction; voxel_size is the resolution of the voxel grid, i.e., the side length of each voxel;

[0037] This yields a three-dimensional occupancy grid for collision detection. Based on the occupancy grid, the Euclidean signed distance field (ESDF) is further calculated, enabling any point in space to quickly query the signed distance and gradient information of the nearest obstacle. This provides a continuous and differentiable geometric basis for subsequent safety distance constraints, risk cost calculations, and gradient-based trajectory optimization.

[0038] Run a semantic segmentation network on synchronized images to output pixel-level results for different categories of obstacles;

[0039] By utilizing the external pose of the camera, the semantic results are back-projected into the three-dimensional space and aligned with the occupying grid in the same voxel grid or grid map to form a static geometric layer.

[0040] To support differentiated risk avoidance in path planning, the system maps different categories to different risk weights and safety buffers, and combines the Euclidean symbolic distance field (ESDF distance) to continuously diffuse the risk, thus obtaining a semantic risk layer.

[0041] For dynamic targets, multi-target association is performed based on the segmentation results and laser point cloud clustering at each time step. Kalman filtering is used to estimate their state, and future trajectory distribution is generated within a short-time prediction window. The prediction results are converted into time-related spatiotemporal occupancy to obtain the dynamic prediction layer.

[0042] The dynamic prediction layer, static geometry layer, and semantic risk layer are superimposed to form a short-term feasible space for immediate avoidance and replanning. Finally, a hierarchical world model is output.

[0043] Furthermore, the process of obtaining the initial inspection path based on the hierarchical world model and task requirements includes the following steps:

[0044] Using a layered world model as the environmental basis, the task requirements are structured, clarifying the set of inspection objects, the mandatory content of each object, the timeliness and priority, and flight constraints;

[0045] Extract the environmental elements required for planning from the hierarchical world model, including geometrically feasible space, semantic risk cost field, and short-term infeasible spatiotemporal region;

[0046] We obtain a computationally usable definition of the planning problem, including the objective point, the set of hard constraints, the soft constraint cost terms and their weights;

[0047] After defining the planning problem, an initial feasible route network is constructed on the static geometry layer. The free space is voxelized at a fixed resolution or a route map is constructed. A safety boundary is set using the Euclidean symbolic distance field (ESDF) to eliminate nodes and edges close to obstacles.

[0048] Apply hard constraints to no-fly zones and semantically high-risk areas to keep paths away from sensitive objects;

[0049] The skeleton path is obtained by using classical graph search. If the world model contains short-term infeasible areas in the dynamic prediction layer, a coarse avoidance is performed in this stage using a time window to ensure that the initial path is executable in a short time. After obtaining the skeleton path, it is converted into an initial inspection path that can be inspected.

[0050] Furthermore, the step of generating candidate observation poses that meet the shooting requirements based on the initial inspection path and using a conditional generative adversarial network (cGAN) based on the inspection target and environmental constraints includes the following steps:

[0051] A conditional generative adversarial network (cGAN) is employed, where the generator G learns to generate the observed pose from noise z and condition c, the discriminator D distinguishes the generated pose from the true pose, and the adversarial loss function is used. The expression is as follows:

[0052] ;

[0053] Where x is the actual observed pose and c is the condition vector; is the expected value of the real data; D(x|c) is the score of the discriminator for the real sample under given conditions; Let z be the expected value of the generated data; let g be the random noise vector; let G(z|c) be the pose generated by the generator from the random noise vector z under condition c; and let D(G(z|c)|c) be the score of the discriminator on the generated sample.

[0054] For the i-th inspection target T i Construct the target feature vector The expression is as follows:

[0055] ;

[0056] in, The coordinates of the target center; The primary orientation of the target; Target size; Minimum resolution requirement; To meet the required field of view range; For overlap ratio requirements;

[0057] Environmental features extracted based on a hierarchical world model The expression is as follows:

[0058] ;

[0059] Among them, F ESDF Features of ESDF;

[0060] F semantic For semantic features, a 3D convolutional neural network (3D CNN) is used to extract the semantic risk distribution around the target, as shown in the following expression: ; Among them, Target center coordinates Let the inspection target be the center and the radius be R. roi Local semantic voxel blocks;

[0061] F dynamic For dynamic constraints, the dynamic risk encoding within the prediction time window is expressed as follows: ;

[0062] Among them, F weather Let w(t) be a weather feature, and R be a time weighting function. dynamic (p,t) is the dynamic risk function, representing the risk value generated by the dynamic body motion prediction at time t and position p;

[0063] A multi-scale generator is used to progressively refine the observation pose, with each candidate observation pose containing:

[0064] pose i =[p i ,q i ,v i ,f i ,t i ];

[0065] Where, p i For camera position; q i Quaternion pose; v i f represents the direction of flight speed; i t is the focal length factor; i For the timing of the shoot;

[0066] Generative constraints introduce hard constraints through a loss function, including:

[0067] line-of-sight constraints The expression is as follows:

[0068] ;

[0069] Field of view constraint The expression is as follows: ;

[0070] Safety distance constraints The expression is as follows: ;

[0071] Occlusion constraints The expression is as follows:

[0072] ;

[0073] Where, p i This is the i-th generated observation location; Let d be the center position of the i-th inspection target; max The maximum allowable observation distance; d min The minimum allowable observation distance; θ min The minimum required field of view; n i Let d be the unit vector of the camera orientation at the i-th observation position; safe The required minimum safe distance threshold ESDF(p) i ) represents position p i Euclidean sign distance field value at ; max(0,d) safe -ESDF(p i The occlusion indicator function returns 1 as a penalty for insufficient safe distance, starting from position p. i To the target Obstructed; return to 0, unobstructed view;

[0074] The overall loss function is expressed as follows:

[0075] ;

[0076] Where, λ adv To counteract loss weights; λ dist λ represents the distance constraint weights. fov λ represents the field-of-view constraint weights. safe Weights for safety constraints; λocc Weights for occlusion constraints;

[0077] The overall quality score is calculated for each generated pose, as shown in the following expression:

[0078] ;

[0079] Among them, Q visibility For target visibility and occlusion level; Q resolution For imaging resolution satisfaction; Q safety For safety distance and risk assessment; Q efficiency The connection cost is the cost of the path; w1, w2, w3, and w4 are weighting coefficients.

[0080] Choose N with the highest quality score. candidate Each pose as a candidate set The expression is as follows:

[0081] ;

[0082] Where K is the total number of poses generated by the generator.

[0083] Furthermore, the step of modeling pose quality evaluation as a sequential decision problem using a reinforcement learning filter, optimizing candidate observation poses under preset constraints, and obtaining the optimal pose combination includes the following steps:

[0084] Based on the candidate observation pose set generated for each inspection target, the agent then makes selections step by step according to the target order or the path segment progression time t: state s t It consists of the current inspection progress, the coverage of the selected pose set, the geometric relationship with the initial inspection path, the local risks of the layered world model, and the characteristics of the remaining range; Action a t To make a decision on whether to select one or skip the candidate poses for the current target;

[0085] After executing the action, the environment updates the selected set and remaining budget, and then proceeds to the next objective;

[0086] Under predefined constraints, reinforcement learning is embedded using a combination of hard constraints and soft penalties: unviolationable constraints are directly masked in the feasible action set, while trade-off constraints are incorporated into the reward function as penalty terms. The reward function r... t It consists of observational benefits, security and compliance costs, and efficiency costs, with cumulative returns through discounts. The agent learns to obtain the optimal strategy within the feasible domain that satisfies the constraints.

[0087] During the training phase, different scenarios are repeatedly sampled in historical tasks and simulation environments for policy learning, using the PPO policy gradient method; course learning is introduced during training to improve convergence stability.

[0088] During the inference phase, given a set of candidate poses for a specific task and the current world model, the reinforcement learning filter outputs the selection results in sequence and updates the coverage and budget state immediately after each selection, ultimately obtaining the optimal pose combination that satisfies the constraints.

[0089] Furthermore, the process of mapping the path planning problem to a graph structure based on the initial inspection path and the optimal pose combination, and optimizing the path to obtain the globally optimal route, includes the following steps:

[0090] A directed weighted graph G=(V,E) is constructed, where node V represents the observation pose that must be visited. Each node carries the geometric attributes of the pose, the observation reward features, and the risk features from the hierarchical world model. Edge E represents a feasible transition segment from node i to j. The edge features are obtained by reachability evaluation, including transition distance / time / energy consumption, minimum Euclidean symbolic distance field (ESDF) safety margin, cumulative cost of traversing semantically high-risk areas, spatiotemporal conflict risk with dynamic volume prediction, and kinematic feasibility. At the same time, hard constraints are embedded in the graph in the form of infeasible edge removal, so that subsequent learning and search naturally take place within the feasible region.

[0091] Based on the above graph structure, the Graph Attention Network (GAT) is used to learn the representation of nodes and edges: through the message passing mechanism, each node aggregates the information of its adjacent edges and neighboring nodes, and learns the implicit rules of which next steps are better from the current pose.

[0092] Using the initial feature vector h of the node i (0) and edge feature e ij As input, attention weights are calculated for each edge at layer l, and the updated node representation h is obtained by weighted aggregation of neighbor information. i (l+1) Attention simultaneously considers geometric cost, risk cost, and task benefit, enabling the network to learn a data-driven trade-off between shorter path and lower risk / higher benefit.

[0093] By leveraging the scoring results of nodes and edges using the Graph Attention Network (GAT), the combinatorial optimization problem is transformed into a learning-guided graph search. First, the GAT is used to reweight the edges, generating new edge costs. Then, prune obviously inferior or high-risk edges; then run a global solver on the pruned graph to minimize the total route cost and meet budget and time window constraints.

[0094] If the hierarchical world model is updated during online operation, the system can incrementally update the features of the affected edges and recompute the graph attention network (GAT) in the forward direction. It can quickly re-optimize only within the local subgraph, thereby maintaining real-time performance and ultimately outputting the globally optimal route: containing the complete waypoint sequence, the expected arrival time of each segment and the shooting trigger point, and achieving the optimal trade-off in a global sense between total flight time / energy consumption, risk and observation quality.

[0095] The UAV intelligent inspection path optimization device based on multi-source perception includes:

[0096] The acquisition module is used to acquire airborne sensor data streams and external data, perform preprocessing, and acquire synchronous data streams in a unified coordinate system.

[0097] The layered world module is used to fuse laser point clouds with synchronous data streams in a unified coordinate system, identify obstacles using a semantic segmentation network, and track and predict dynamic objects to obtain a layered world model.

[0098] The initial module is used to obtain the initial inspection path based on the hierarchical world model and task requirements;

[0099] The pose module is used to generate candidate observation poses that meet the shooting requirements based on the initial inspection path and the conditional generative adversarial network cGAN, based on the inspection target and environmental constraints.

[0100] The optimization module is used to model pose quality assessment as a sequential decision problem through a reinforcement learning filter, optimize candidate observation poses under preset constraints, and obtain the optimal pose combination.

[0101] The path planning module is used to map the path planning problem to a graph structure based on the initial inspection path and the optimal pose combination, and to optimize the path to obtain the globally optimal route.

[0102] Furthermore, the acquisition module is used to acquire airborne sensor data streams and external data, perform preprocessing, and acquire a synchronized data stream in a unified coordinate system, wherein the preprocessing includes:

[0103] Implement intrinsic and extrinsic parameter management and coordinate transformation link fixing for each sensor;

[0104] Establish transformation relationships from various sensor coordinate systems to a unified world coordinate system;

[0105] The required external data is also resampled to a unified world coordinate system and topologically consistent with the no-fly zone boundary, and finally a synchronized data stream is output.

[0106] A computer device includes a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the steps of a method for optimizing the intelligent inspection path of a drone based on multi-source perception.

[0107] A computer storage medium storing a computer program, wherein when the computer program is executed by a processor, the steps of a method for optimizing the intelligent inspection path of a drone based on multi-source perception are implemented.

[0108] Compared with existing technologies, its advantages are as follows:

[0109] By achieving spatiotemporal alignment and coordinate unification using multi-source heterogeneous sensors, the problems of data silos and accuracy loss in traditional inspections are solved. Based on this, a hierarchical world model is constructed, organically integrating static geometry (occupied grid / ESDF, Euclidean Signed Distance Functions), semantic risk (obstacle recognition and risk stratification), and dynamic prediction (moving object tracking and short-term feasible airspace). This results in a richer and more accurate environmental understanding than traditional single maps. When generating flight routes, it can simultaneously handle geometric constraints (full range, no-fly zones), task constraints (resolution, view coverage, overlap rate), regulatory constraints (flight time budget, return threshold), and dynamic constraints (real-time obstacle avoidance, weather changes). By embedding these multi-layered constraints into the optimization process, the generated flight routes are ensured to be both safe and feasible, as well as efficient and of high quality. Furthermore, the hierarchical world model supports incremental updates and local replanning. When the environment changes or task requirements are adjusted, there is no need for global recalculation; instead, rapid response is achieved through local updates of the affected areas. It can continuously optimize the quality of subsequent planning decisions based on inspection feedback. Attached Figure Description

[0110] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0111] Figure 1 This is a flowchart of a method for optimizing the intelligent inspection path of unmanned aerial vehicles (UAVs) based on multi-source perception. Detailed Implementation

[0112] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0113] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0114] Example 1

[0115] This invention provides an embodiment, such as Figure 1 The diagram shows a method for optimizing the intelligent inspection path of unmanned aerial vehicles (UAVs) based on multi-source perception, which includes the following steps:

[0116] Acquire airborne sensor data streams and external data, perform preprocessing, and obtain synchronized data streams in a unified coordinate system;

[0117] Based on synchronous data streams in a unified coordinate system, laser point clouds are fused to generate occupied grids and Euclidean symbolic distance fields (ESDF). At the same time, a semantic segmentation network is used to identify obstacles, which are superimposed to form a semantic risk layer. Dynamic bodies are tracked and predicted to form a short-term feasible spatial domain, resulting in a layered world model.

[0118] Based on the hierarchical world model and task requirements, obtain the initial inspection path;

[0119] Based on the initial inspection path, a conditional generative adversarial network (cGAN) is used to generate candidate observation poses that meet the shooting requirements, based on the inspection target and environmental constraints.

[0120] By using a reinforcement learning filter, pose quality assessment is modeled as a sequential decision problem, and candidate observation poses are optimized under preset constraints to obtain the optimal pose combination.

[0121] Based on the initial inspection path and the optimal pose combination, the path planning problem is mapped to a graph structure. A graph attention network is used to learn the feature representations of nodes and edges, and the path is optimized to obtain the globally optimal route.

[0122] Example 2

[0123] This invention provides another embodiment, which is a method for optimizing the intelligent inspection path of unmanned aerial vehicles (UAVs) based on multi-source perception, wherein: the acquisition of airborne sensor data streams and external data is specifically as follows:

[0124] Acquire airborne sensor data streams, including RGB images, infrared thermal images, laser point clouds, IMU inertial navigation, and RTK positioning data. Perform unified time reference processing on the data, using the Global Navigation Satellite System time as a reference, and correct the timestamps of each sensor to compensate for fixed delays and jitter. Pull DEM elevation, 3D route models, and the latest meteorological data matching the mission area from external systems, including wind speed and direction, rainfall, visibility, temperature and humidity, as well as no-fly zones and temporarily controlled airspace data. Slice and cache low-frequency external data according to validity period and spatial range to ensure fast access during planning and replanning. This results in a raw multi-source data set that is time-aligned and range-matched.

[0125] Example 3

[0126] This invention provides another embodiment, which is a method for optimizing the intelligent inspection path of unmanned aerial vehicles based on multi-source perception, wherein: the preprocessing to obtain a synchronous data stream in a unified coordinate system is as follows:

[0127] Camera intrinsic parameters are used for distortion correction and imaging geometric consistency, while extrinsic parameters between the LiDAR, camera, and IMU are used for cross-modal projection and fusion. The lever parameters from the RTK antenna phase center to the machine system are used for precise positioning correction. Subsequently, a transformation relationship is established from each sensor coordinate system to a unified world coordinate system, such as ENU / NED or the mission local map coordinate system. RGB frames are associated with their corresponding poses, the LiDAR point cloud is distorted and transformed to a unified coordinate system, the depth map is projected into a dense depth surface, and the IMU data is pre-integrated for subsequent fusion and positioning. The external DEM and 3D model are also resampled to the same coordinate reference and topologically consistent with the no-fly zone boundary. Finally, a synchronized data stream is output.

[0128] Example 4

[0129] This invention provides another embodiment, which is a method for optimizing the intelligent inspection path of UAVs based on multi-source perception, wherein: the method of generating an occupied grid and an Euclidean symbolic distance field (ESDF) by fusing synchronous data streams in a unified coordinate system with laser point clouds is as follows:

[0130] The laser point cloud at each moment is distorted and its coordinates are transformed in the corresponding pose, then projected onto the world coordinate system and fused across multiple frames.

[0131] Let the original coordinates of the i-th laser point in the sensor coordinate system be... The measurement time is t i The reference time is t. ref The expression is as follows:

[0132] ;

[0133] In the formula, The coordinates of the i-th laser point after motion compensation are denoted as . From time t i to reference time t ref The rotation transformation matrix; From time t i to reference time t ref Translation vector;

[0134] Where the rotation matrix Translation vector Obtained through linear interpolation:

[0135] ;

[0136] Among them, Slerp(R) ti ,R tref ,α) is a spherical linear interpolation function used for smooth interpolation between two rotation matrices; α is the interpolation parameter, ranging from [0,1], representing the relative position within the scanning time window at the current moment; t start t represents the start time of the current scan frame. end This represents the end time of the current scan frame;

[0137] The distortion-corrected laser point cloud is transformed from the sensor coordinate system to the world coordinate system, as shown in the following expression:

[0138] ;

[0139] in, Let be the final coordinates of the i-th laser point in the world coordinate system; This is the rotation matrix from the sensor coordinate system to the world coordinate system; This is the translation vector from the sensor coordinate system to the world coordinate system;

[0140] The fusion process employs a ray casting and probability update mechanism: the space between the sensor and the measurement point is updated to free, and the voxels hit by the measurement are updated to occupied. For each laser point... From sensor location All voxels on the ray leading to the target point are updated using the 3D Bresenham algorithm, as shown in the following expression:

[0141] ;

[0142] Among them, v j s represents the world coordinate position of the j-th voxel on the ray; world This indicates the sensor's position in the world coordinate system; j is the voxel index along the ray direction; voxel_size is the resolution of the voxel grid, i.e., the side length of each voxel;

[0143] This yields a three-dimensional occupancy grid for collision detection. Based on the occupancy grid, the Euclidean signed distance field (ESDF) is further calculated, enabling any point in space to quickly query the signed distance and gradient information of the nearest obstacle. This provides a continuous and differentiable geometric basis for subsequent safety distance constraints, risk cost calculations, and gradient-based trajectory optimization.

[0144] Example 5

[0145] This invention provides another embodiment, which is a method for optimizing intelligent inspection paths for unmanned aerial vehicles (UAVs) based on multi-source perception. In this method, a semantic segmentation network is used to identify obstacles, which are then superimposed to form a semantic risk layer. Dynamic objects are tracked and predicted to form a short-term feasible airspace, resulting in a hierarchical world model, as detailed below:

[0146] A semantic segmentation network is run on synchronized RGB images, outputting pixel-level results for categories such as power lines, poles, trees, vehicles, and crowds. Subsequently, using camera-external pose parameters, the semantic results are back-projected into 3D space and aligned with the occupying grid in the same voxel grid or raster map to form a semantically labeled spatial layer, resulting in a static geometry layer. To support differentiated risk avoidance in path planning, the system maps different categories to different risk weights and safety buffers. For example, "power lines / crowds" are assigned higher weights and larger buffer zones, while "trees / general static obstacles" are assigned medium weights. The system also uses Euclidean distance (ESDF) to continuously diffuse the risk, forming a directly queryable semantic risk cost field, resulting in a semantic risk layer. This semantic risk layer can be used for both hard constraints (prohibiting entry into high-risk areas) and soft constraints (guiding the path away from high-risk categories in the form of cost).

[0147] For dynamic targets such as vehicles, crowds, and mobile machinery, multi-target association is performed based on the segmentation results and laser point cloud clustering at each time step. Kalman filtering is used to estimate their state, which includes position, velocity, heading, and uncertainty. Future trajectory distribution is generated within a short-term prediction window. The prediction results are converted into time-related spatiotemporal occupancy to obtain the dynamic prediction layer.

[0148] The dynamic prediction layer, static geometry layer, and semantic risk layer are superimposed to form a short-term feasible spatial domain for immediate avoidance and replanning. Finally, a hierarchical world model is output, in which the dynamic prediction layer includes tracking state + short-term predicted feasible spatial domain, the static geometry layer includes occupied grid + Euclidean symbolic distance field ESDF, and the semantic risk layer includes semantic label + risk cost field.

[0149] Example 6

[0150] This invention provides another embodiment, which is a method for optimizing intelligent inspection paths for unmanned aerial vehicles (UAVs) based on multi-source perception, wherein: the initial inspection path is obtained based on a hierarchical world model and task requirements, as follows:

[0151] Using a layered world model as the environmental foundation, the task requirements are structured, clearly defining the set of inspection objects, such as tower numbers, line segments, and component rows / columns; utility tunnel mileage markers; mandatory image content for each object, including visible surface / angle range, minimum resolution GSD, overlap rate, and thermal imaging trigger conditions; timeliness and priority, including priority for high-risk points and fault re-inspection; and flight constraints, including minimum obstacle clearance distance, maximum altitude, no-fly zones and buffer zones, take-off and landing points, and return-to-base thresholds. Environmental elements required for planning are extracted from the layered world model, including geometrically feasible space: traversable voxels formed by occupying grids / Euclidean symbolic distance field (ESDF) thresholds; semantic risk cost field: risk weights for different categories; and short-term infeasible spatiotemporal regions given by the dynamic prediction layer. This yields a computationally usable planning problem definition, including target points, hard constraint sets, soft constraint cost terms, and their weights. After defining the planning problem, an initial feasible route network is constructed on the static geometric layer. The free space is voxelized at a fixed resolution or a route map is constructed, and safety boundaries are set using the Euclidean symbolic distance field (ESDF): ESDF requirements are defined. (x)≥dmin is used to remove nodes and edges close to obstacles; hard constraints are applied to no-fly zones and semantically high-risk zones to keep the path away from sensitive objects such as power lines and crowds; then, a classic graph search is used to obtain the skeleton path, for example, in the case of a single machine, the A* algorithm is used to generate connected paths from the starting point to each target area on a 3D grid; if the world model contains short-term infeasible areas in the dynamic prediction layer, a coarse avoidance is performed in this stage in the form of a time window, that is, high-risk voxels in the future are set as inaccessible or given high cost to ensure that the initial path is executable in the short term. After obtaining the skeleton path, it is converted into an initial inspection path that can be inspected and executed.

[0152] Example 7

[0153] This invention provides another embodiment, which is a method for optimizing intelligent inspection paths for unmanned aerial vehicles (UAVs) based on multi-source perception. Specifically, based on the initial inspection path, a conditional generative adversarial network (cGAN) is used to generate candidate observation poses that meet the shooting requirements, based on the inspection target and environmental constraints.

[0154] A conditional generative adversarial network (cGAN) is employed, where the generator G learns to generate the observed pose from noise z and condition c, the discriminator D distinguishes the generated pose from the true pose, and the adversarial loss function is used. The expression is as follows:

[0155] ;

[0156] Where x is the actual observed pose and c is the condition vector (inspection target + environmental constraints). is the expected value of the real data; D(x|c) is the score of the discriminator for the real sample under given conditions; Let z be the expected value of the generated data; let g be the random noise vector; let G(z|c) be the pose generated by the generator from the random noise vector z under condition c; and let D(G(z|c)|c) be the score of the discriminator on the generated sample.

[0157] For the i-th inspection target T i Construct the target feature vector The expression is as follows:

[0158] ;

[0159] in, The coordinates of the target center; The primary orientation of the target; Target size; Minimum resolution requirement; To meet the required field of view range; For overlap ratio requirements;

[0160] Environmental features extracted based on a hierarchical world model The expression is as follows:

[0161] ;

[0162] Among them, F ESDF Features of ESDF;

[0163] F semantic For semantic features, a 3D convolutional neural network (3D CNN) is used to extract the semantic risk distribution around the target, as shown in the following expression: ;in, The coordinates of the target center are Let the inspection target be the center and the radius be R. roi Local semantic voxel blocks;

[0164] F dynamic For dynamic constraints, the dynamic risk encoding within the prediction time window is expressed as follows: ;

[0165] Among them, F weather Let w(t) be a weather feature, and R be a time weighting function. dynamic (p,t) is the dynamic risk function, representing the risk value generated by the dynamic body motion prediction at time t and position p;

[0166] A multi-scale generator is used to progressively refine the observation pose, with each candidate observation pose containing:

[0167] pose i =[p i ,q i ,v i ,f i ,t i ];

[0168] Where, p i For camera position; q i Quaternion pose; v i f represents the direction of flight speed; i t is the focal length factor; i For the timing of the shoot;

[0169] Generative constraints introduce hard constraints through a loss function, including:

[0170] line-of-sight constraints The expression is as follows:

[0171] ;

[0172] Field of view constraint The expression is as follows:

[0173] ;

[0174] Safety distance constraints The expression is as follows:

[0175] ;

[0176] Occlusion constraints The expression is as follows:

[0177] ;

[0178] Where, p i This is the i-th generated observation location; Let d be the center position of the i-th inspection target; max The maximum allowable observation distance; d min The minimum allowable observation distance; θ min The minimum required field of view; n i Let d be the unit vector of the camera orientation at the i-th observation position; safe The required minimum safe distance threshold ESDF(p) i ) represents position p i Euclidean sign distance field value at ; max(0,d) safe -ESDF(p iThe occlusion indicator function returns 1 as a penalty for insufficient safe distance, starting from position p. i To the target Obstructed; return to 0, unobstructed view;

[0179] The overall loss function is expressed as follows:

[0180] ;

[0181] Where, λ adv To counteract loss weights; λ dist λ represents the distance constraint weights. fov λ represents the field-of-view constraint weights. safe Weights for safety constraints; λocc Weights for occlusion constraints;

[0182] The overall quality score is calculated for each generated pose, as shown in the following expression:

[0183] ;

[0184] Among them, Q visibility For target visibility and occlusion level; Q resolution For imaging resolution satisfaction; Q safety For safety distance and risk assessment; Q efficiency The connection cost is the cost of the path; w1, w2, w3, and w4 are weighting coefficients.

[0185] Choose N with the highest quality score. candidate Each pose as a candidate set The expression is as follows:

[0186] ;

[0187] Where K is the total number of poses generated by the generator.

[0188] Example 8

[0189] This invention provides another embodiment, which is a method for optimizing the intelligent inspection path of UAVs based on multi-source perception, wherein: the pose quality assessment is modeled as a sequential decision problem by a reinforcement learning filter, and the candidate observation poses are optimized under preset constraints to obtain the optimal pose combination, as detailed below:

[0190] Based on the candidate observation pose set generated for each inspection target, the agent then makes selections step by step according to the target order or the path segment progression time t: state s tIt consists of the current inspection progress, the coverage of the selected pose set, the geometric relationship with the initial inspection path, and the local risks of the layered world model, including the Euclidean symbolic distance field (ESDF) minimum margin, semantic risk cost, dynamic feasible airspace occupancy probability, and remaining range characteristics; action a t To make decisions about selecting or skipping a candidate pose for the current target, including fine-tuning such as discrete yaw angle settings; after executing the action, update the selected set and remaining budget, and move on to the next target;

[0191] Under preset constraints, such as minimum safe distance ESDF(x) ≥ dmin, zero tolerance for no-fly zones, thresholds for shooting GSD / viewpoint / overlap rate, probability of conflict with dynamic body prediction not exceeding threshold δ, and total flight time / range / battery usage not exceeding budget, a reinforcement learning approach combining hard constraints and soft penalties is adopted: unviolated constraints are directly masked in the action feasible set, while trade-off constraints are entered into the reward function as penalty terms. The reward function r t The benefits include: observational gains (target visibility, occlusion rate, resolution / GSD satisfaction, and overlap coverage contribution with the selected pose); safety and compliance (maximum safety margin, minimum semantic risk, and minimum dynamic conflict risk); and efficiency costs (transition distance / time from the current pose to the next pose, pose change magnitude, and deviation from the initial inspection path), with cumulative returns through discounts. The agent learns to obtain the optimal strategy within the feasible domain that satisfies the constraints.

[0192] During the training phase, different scenarios are repeatedly sampled in historical tasks and simulation environments, including different target distributions, obstacle densities, dynamic volume levels, and weather / lighting, to learn policies using the PPO policy gradient method. Course learning is introduced during training, including first static scenarios without dynamic volumes, then semantic high-risk scenarios, and finally dynamic prediction and strict budgeting to improve convergence stability.

[0193] During the inference phase, given a set of candidate poses for a specific task and the current world model, the reinforcement learning filter outputs the selection results in sequence and updates the coverage and budget state immediately after each selection, ultimately obtaining the optimal pose combination that satisfies the constraints: according to the pose index and temporal sequence corresponding to the target.

[0194] Example 9

[0195] This invention provides another embodiment, which is a method for optimizing intelligent inspection paths for unmanned aerial vehicles (UAVs) based on multi-source perception. Specifically, based on the initial inspection path and the optimal pose combination, the path planning problem is mapped to a graph structure, and a graph attention network is used to learn the feature representations of nodes and edges, as detailed below:

[0196] A directed weighted graph G=(V,E) is constructed, where node V represents the observation poses that must be visited, including the start / end / return points and necessary waypoints. Each node carries pose geometric attributes, including position, orientation, and shooting time window, observation benefit features, including visibility, GSD satisfaction, and coverage contribution, and risk features from the hierarchical world model, including the Euclidean signed distance field (ESDF) margin at the node, semantic risk value, dynamic conflict probability, and positioning uncertainty. Edge E represents a feasible transfer segment from node i to j, and edge features are obtained from reachability assessment, including transfer distance / time / energy consumption, minimum Euclidean signed distance field (ESDF) safety margin, cumulative cost of traversing semantically high-risk areas, spatiotemporal conflict risk with dynamic volume prediction, and kinematic feasibility, including climb rate and turning radius. At the same time, hard constraints, including no-fly zones, minimum safe distance, time window, and budget limit, are embedded in the graph in the form of infeasible edge removal, so that subsequent learning and search naturally take place within the feasible region.

[0197] Based on the above graph structure, a Graph Attention Network (GAT) is used to learn the representations of nodes and edges: through a message passing mechanism, each node aggregates information about its adjacent edges and neighboring nodes, learning the implicit patterns of which next steps are better from the current pose; using the initial feature vector h of the nodes... i (0) and edge feature e ij As input, attention weights are computed for each edge at layer l, reflecting the relative priority from i to j, and the updated node representation h is obtained by weighted aggregation of neighbor information. i (l+1) attention simultaneously considers geometric cost, risk cost and task benefit, enabling the network to learn a data-driven trade-off between shorter routes and lower risk / higher benefit.

[0198] Example 10

[0199] This invention provides another embodiment, which is a method for optimizing the intelligent inspection path of unmanned aerial vehicles based on multi-source perception, wherein: the optimized path obtains the globally optimal flight path, as follows:

[0200] By leveraging the scoring results of nodes and edges using the Graph Attention Network (GAT), combinatorial optimization problems—such as the Traveling Salesman Problem (TSP) / Orienteering / Path Constraint problems with risk and time windows—are transformed into learning-guided graph search. First, the GAT network is used to reweight the edges, forming new edge costs. The process involves a trade-off between overall transfer costs, risks, and benefits, and pruning of obviously poor or high-risk edges. Then, a global solver, A*, is run on the pruned graph to minimize the total route cost and meet budget and time window constraints.

[0201] If the hierarchical world model is updated during online operation, including changes in dynamic obstacles, increased semantic risk, and changes in the Euclidean symbolic distance field (ESDF), the system can incrementally update the features of the affected edges and recompute the graph attention network (GAT) in the forward pass. This allows for rapid re-optimization only within local subgraphs, thus maintaining real-time performance. Ultimately, the system outputs the globally optimal route, which includes a complete sequence of waypoints, expected arrival times for each segment, and shooting trigger points. It achieves a globally optimal trade-off between total flight time / energy consumption, risk, and observation quality.

[0202] Example 11

[0203] This invention provides another embodiment of a UAV intelligent inspection path optimization device based on multi-source perception, used to implement the UAV intelligent inspection path optimization method based on multi-source perception described in embodiments 1-10, including:

[0204] The acquisition module is used to acquire airborne sensor data streams and external data, perform preprocessing, and acquire synchronous data streams in a unified coordinate system.

[0205] The layered world module is used to fuse laser point clouds with synchronous data streams in a unified coordinate system, identify obstacles using a semantic segmentation network, and track and predict dynamic objects to obtain a layered world model.

[0206] The initial module is used to obtain the initial inspection path based on the hierarchical world model and task requirements;

[0207] The pose module is used to generate candidate observation poses that meet the shooting requirements based on the initial inspection path and the conditional generative adversarial network cGAN, based on the inspection target and environmental constraints.

[0208] The optimization module is used to model pose quality assessment as a sequential decision problem through a reinforcement learning filter, optimize candidate observation poses under preset constraints, and obtain the optimal pose combination.

[0209] The path planning module is used to map the path planning problem to a graph structure based on the initial inspection path and the optimal pose combination, and to optimize the path to obtain the globally optimal route.

[0210] Example 12

[0211] This invention provides another embodiment, a UAV intelligent inspection path optimization device based on multi-source perception, used to implement the UAV intelligent inspection path optimization method based on multi-source perception described in embodiments 1-10. The acquisition module is used to acquire airborne sensor data streams and external data, perform preprocessing, and acquire a synchronous data stream in a unified coordinate system. The preprocessing includes:

[0212] Implement intrinsic and extrinsic parameter management and coordinate transformation link fixing for each sensor;

[0213] Establish transformation relationships from various sensor coordinate systems to a unified world coordinate system;

[0214] The required external data is also resampled to a unified world coordinate system and topologically consistent with the no-fly zone boundary, and finally a synchronized data stream is output.

[0215] Example 13

[0216] Based on the same inventive concept, this invention also provides a computer device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor. When the processor executes the computer program, it implements the steps of the UAV intelligent inspection path optimization method based on multi-source perception described in Embodiment 1.

[0217] Example 14

[0218] Based on the same inventive concept, this embodiment of the invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the UAV intelligent inspection path optimization method based on multi-source perception described in Embodiment 1.

[0219] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0220] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0221] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0222] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0223] In the description of this specification, the terms "one embodiment," "some embodiments," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0224] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for optimizing intelligent inspection paths for unmanned aerial vehicles (UAVs) based on multi-source perception, characterized in that, Includes the following steps: Acquire airborne sensor data streams and external data, perform preprocessing, and obtain synchronized data streams in a unified coordinate system; Based on synchronous data streams in a unified coordinate system, laser point clouds are fused, semantic segmentation networks are used to identify obstacles, and dynamic bodies are tracked and predicted to obtain a hierarchical world model. Based on the hierarchical world model and task requirements, obtain the initial inspection path; Based on the initial inspection path, a conditional generative adversarial network (cGAN) is used to generate candidate observation poses that meet the shooting requirements, based on the inspection target and environmental constraints. By using a reinforcement learning filter, pose quality assessment is modeled as a sequential decision problem, and candidate observation poses are optimized under preset constraints to obtain the optimal pose combination. Based on the initial inspection path and the optimal pose combination, the path planning problem is mapped to a graph structure, and the path is optimized to obtain the globally optimal route.

2. The method for optimizing intelligent inspection paths for unmanned aerial vehicles based on multi-source perception according to claim 1, characterized in that, The acquisition of airborne sensor data streams and external data includes the following steps: The system acquires data streams from airborne sensors, processes the data using a unified time reference, and corrects the timestamps of each sensor to compensate for fixed delays and jitter, using the time of the Global Navigation Satellite System as a reference. Pull external data that matches the task area from external systems, and slice and cache low-frequency external data according to validity period and spatial range to ensure fast access during planning and replanning. After performing the above steps, a raw multi-source data set that is time-aligned and range-matched is obtained; The preprocessing process to obtain the synchronized data stream in a unified coordinate system includes the following steps: Implement intrinsic and extrinsic parameter management and coordinate transformation link fixing for each sensor; Establish transformation relationships from various sensor coordinate systems to a unified world coordinate system; The required external data is also resampled to a unified world coordinate system and topologically consistent with the no-fly zone boundary, and finally a synchronized data stream is output.

3. The method for optimizing intelligent inspection paths for unmanned aerial vehicles based on multi-source perception according to claim 1, characterized in that, The process of obtaining a hierarchical world model based on synchronous data streams in a unified coordinate system, fusing laser point clouds, using a semantic segmentation network to identify obstacles, and tracking and predicting dynamic objects includes the following steps: The laser point cloud at each moment is distorted and its coordinates are transformed in the corresponding pose, then projected onto the world coordinate system and fused across multiple frames. Let the original coordinates of the i-th laser point in the sensor coordinate system be... The measurement time is t i The reference time is t. ref The expression is as follows: ； In the formula, The coordinates of the i-th laser point after motion compensation are denoted as . From time t i to reference time t ref The rotation matrix; From time t i to reference time t ref Translation vector; Where the rotation matrix Translation vector Obtained through linear interpolation: ； Among them, Slerp(R) ti ,R tref ,α) is a spherical linear interpolation function used for smooth interpolation between two rotation matrices; α is the interpolation parameter, ranging from [0,1], representing the relative position within the scanning time window at the current moment; t start t represents the start time of the current scan frame. end This represents the end time of the current scan frame; The distortion-corrected laser point cloud is transformed from the sensor coordinate system to the world coordinate system, as shown in the following expression: ； in, Let be the final coordinates of the i-th laser point in the world coordinate system; This is the rotation matrix from the sensor coordinate system to the world coordinate system; This is the translation vector from the sensor coordinate system to the world coordinate system; The fusion process employs a ray casting and probability update mechanism: the space between the sensor and the measurement point is updated to free, and the voxels hit by the measurement are updated to occupied. For each laser point... From sensor location All voxels on the ray leading to the target point are updated using the 3D Bresenham algorithm, as shown in the following expression: ； Among them, v j s represents the world coordinate position of the j-th voxel on the ray; world This indicates the sensor's position in the world coordinate system; j is the voxel index along the ray direction; voxel_size is the resolution of the voxel grid, i.e., the side length of each voxel; This yields a three-dimensional occupancy grid for collision detection. Based on the occupancy grid, the Euclidean signed distance field (ESDF) is further calculated, enabling any point in space to quickly query the signed distance and gradient information of the nearest obstacle. This provides a continuous and differentiable geometric basis for subsequent safety distance constraints, risk cost calculations, and gradient-based trajectory optimization. Run a semantic segmentation network on synchronized images to output pixel-level results for different categories of obstacles; By utilizing the external pose of the camera, the semantic results are back-projected into the three-dimensional space and aligned with the occupying grid in the same voxel grid or grid map to form a static geometric layer. To support differentiated risk avoidance in path planning, the system maps different categories to different risk weights and safety buffers, and combines the Euclidean distance field ESDF distance to continuously diffuse the risk, thus obtaining a semantic risk layer. For dynamic targets, multi-target association is performed based on the segmentation results and laser point cloud clustering at each time step. Kalman filtering is used to estimate their state, and future trajectory distribution is generated within a short-time prediction window. The prediction results are converted into time-related spatiotemporal occupancy to obtain the dynamic prediction layer. The dynamic prediction layer, static geometry layer, and semantic risk layer are superimposed to form a short-term feasible space for immediate avoidance and replanning. Finally, a hierarchical world model is output.

4. The method for optimizing intelligent inspection paths for unmanned aerial vehicles based on multi-source perception according to claim 1, characterized in that, The process of obtaining the initial inspection path based on the hierarchical world model and task requirements includes the following steps: Using a layered world model as the environmental basis, the task requirements are structured, clarifying the set of inspection objects, the mandatory content of each object, the timeliness and priority, and flight constraints; Extract the environmental elements required for planning from the hierarchical world model, including geometrically feasible space, semantic risk cost field, and short-term infeasible spatiotemporal region; We obtain a computationally usable definition of the planning problem, including the objective point, the set of hard constraints, the soft constraint cost terms and their weights; After defining the planning problem, an initial feasible route network is constructed on the static geometry layer. The free space is voxelized at a fixed resolution or a route map is constructed. A safety boundary is set using the Euclidean symbolic distance field (ESDF) to eliminate nodes and edges close to obstacles. Apply hard constraints to no-fly zones and semantically high-risk areas to keep paths away from sensitive objects; The skeleton path is obtained by using classical graph search. If the world model contains short-term infeasible areas in the dynamic prediction layer, a coarse avoidance is performed in the generation stage of the initial inspection path using a time window to ensure that the initial path is executable in a short time. After obtaining the skeleton path, it is converted into an executable initial inspection path.

5. The method for optimizing intelligent inspection paths for unmanned aerial vehicles based on multi-source perception according to claim 1, characterized in that, The process of generating candidate observation poses that meet the imaging requirements based on the initial inspection path and using a conditional generative adversarial network (cGAN) based on the inspection target and environmental constraints includes the following steps: A conditional generative adversarial network (cGAN) is employed, where the generator G learns to generate the observed pose from noise z and condition c, the discriminator D distinguishes the generated pose from the true pose, and the adversarial loss function is used. The expression is as follows: ； Where x is the actual observed pose and c is the condition vector; denoted as the expected value of the true data; D(x|c) is the score of the discriminator for the true sample under given conditions; Let z be the expected value of the generated data; let g be the random noise vector; let G(z|c) be the pose generated by the generator from the random noise vector z under condition c; and let D(G(z|c)|c) be the score of the discriminator on the generated sample. For the i-th inspection target T i Construct target feature vector The expression is as follows: ； in, The coordinates of the target center; The primary orientation of the target; Target size; Minimum resolution requirement; To meet the required field of view range; For overlap ratio requirements; Environmental features extracted based on a hierarchical world model The expression is as follows: ； Among them, F ESDF Euclidean symbolic distance field (ESDF) characteristics; F semantic For semantic features, a 3D convolutional neural network (3D CNN) is used to extract the semantic risk distribution around the target, as shown in the following expression: ; in, The coordinates of the target center are Let the i-th inspection target be the center and the radius be R. roi Local semantic voxel blocks; F dynamic For dynamic constraints, the dynamic risk encoding within the prediction time window is expressed as follows: ; Among them, F weather Let w(t) be a weather feature, and R be a time weighting function. dynamic (p,t) is the dynamic risk function, representing the risk value generated by the dynamic body motion prediction at time t and position p; A multi-scale generator is used to progressively refine the observation pose, with each candidate observation pose containing: pose i =[p i ,q i ,v i ,f i ,t i ]; Where, p i For camera position; q i Quaternion pose; v i f represents the direction of flight speed; i t is the focal length factor; i For the timing of the shoot; Generative constraints introduce hard constraints through a loss function, including: line-of-sight constraints The expression is as follows: ； Field of view constraint The expression is as follows: ； Safety distance constraints The expression is as follows: ； Occlusion constraints The expression is as follows: ； Where, p i This is the i-th generated observation location; Let d be the center position of the i-th inspection target; max The maximum allowable observation distance; d min The minimum allowable observation distance; θ min The minimum required field of view; n i Let d be the unit vector of camera orientation at the i-th observation position; safe The required minimum safe distance threshold ESDF(p) i ) represents position p i Euclidean sign distance field value at ; max(0,d) safe- ESDF(p i The occlusion indicator function returns 1 as a penalty for insufficient safe distance, starting from position p. i To the target Obstructed; return to 0, unobstructed view; The overall loss function is expressed as follows: ； Where, λ adv To counteract loss weights; λ dist λ represents the distance constraint weights. fov λ represents the field-of-view constraint weights. safe Weights for safety constraints; λocc Weights for occlusion constraints; The overall quality score is calculated for each generated pose, as shown in the following expression: ； Among them, Q visibility For target visibility and occlusion level; Q resolution For imaging resolution satisfaction; Q safety For safety distance and risk assessment; Q efficiency The connection cost is the cost of the path; w1, w2, w3, and w4 are weighting coefficients. Choose N with the highest quality score. candidate Each pose as a candidate set The expression is as follows: ； Where K is the total number of poses generated by the generator.

6. The method for optimizing intelligent inspection paths for unmanned aerial vehicles based on multi-source perception according to claim 1, characterized in that, The step of modeling pose quality assessment as a sequential decision problem using a reinforcement learning filter, optimizing candidate observation poses under preset constraints, and obtaining the optimal pose combination includes the following steps: Based on the candidate observation pose set generated for each inspection target, the agent then makes selections step by step according to the target order or the path segment progression time t: state s t It consists of the current inspection progress, the coverage of the selected pose set, the geometric relationship with the initial inspection path, the local risks of the layered world model, and the characteristics of the remaining range; Action a t To make a decision on whether to select one or skip the candidate poses for the current target; After executing the action, the environment updates the selected set and remaining budget, and then proceeds to the next objective; Under predefined constraints, reinforcement learning is embedded using a combination of hard constraints and soft penalties: unviolationable constraints are directly masked in the feasible action set, while trade-off constraints are incorporated into the reward function as penalty terms. The reward function r... t It consists of observational benefits, security and compliance costs, and efficiency costs, with cumulative returns through discounts. The agent learns to obtain the optimal strategy within the feasible domain that satisfies the constraints. During the training phase, different scenarios are repeatedly sampled in historical tasks and simulation environments to learn policies, using the PPO policy gradient method. Introducing course learning into the training process improves convergence stability; During the inference phase, given a set of candidate poses for a specific task and the current world model, the reinforcement learning filter outputs the selection results in sequence and updates the coverage and budget state immediately after each selection, ultimately obtaining the optimal pose combination that satisfies the constraints. The process of mapping the path planning problem to a graph structure based on the initial inspection path and the optimal pose combination, and optimizing the path to obtain the globally optimal route includes the following steps: Construct a directed weighted graph G=(V,E), where node V represents an observation pose that must be visited. Each node carries the geometric properties of the pose, the observation reward features, and the risk features from the hierarchical world model. Edge E represents a feasible transition segment from node i to j. Edge features are obtained from reachability evaluation, including transition distance / time / energy consumption, minimum Euclidean symbol distance field (ESDF) safety margin, cumulative cost of traversing semantically high-risk areas, spatiotemporal conflict risk with dynamic volume prediction, and kinematic feasibility. At the same time, hard constraints are embedded in the graph in the form of infeasible edge removal, so that subsequent learning and search can be carried out naturally within the feasible region. Based on the above graph structure, the Graph Attention Network (GAT) is used to learn the representation of nodes and edges: through the message passing mechanism, each node aggregates the information of its adjacent edges and neighboring nodes, and learns the implicit rules of which next steps are better from the current pose. Using the initial feature vector h of the node i (0) and edge feature e ij As input, attention weights are calculated for each edge at layer l, and the updated node representation h is obtained by weighted aggregation of neighbor information. i (l+1) Attention simultaneously considers geometric cost, risk cost, and task benefit, enabling the network to learn a data-driven trade-off between shorter path and lower risk / higher benefit. By leveraging the scoring results of nodes and edges using the Graph Attention Network (GAT), the combinatorial optimization problem is transformed into a learning-guided graph search. First, the GAT is used to reweight the edges, generating new edge costs. Then, prune obviously inferior or high-risk edges; then run a global solver on the pruned graph to minimize the total route cost and meet budget and time window constraints. If the hierarchical world model is updated during online operation, the system can incrementally update the features of the affected edges and recompute the graph attention network (GAT) in the forward direction. It can quickly re-optimize only within the local subgraph, thereby maintaining real-time performance and ultimately outputting the globally optimal route: containing the complete waypoint sequence, the expected arrival time of each segment and the shooting trigger point, and achieving the optimal trade-off in a global sense between total flight time / energy consumption, risk and observation quality.

7. A UAV intelligent inspection path optimization device based on multi-source perception, characterized in that, include: The acquisition module is used to acquire airborne sensor data streams and external data, perform preprocessing, and acquire synchronous data streams in a unified coordinate system. The layered world module is used to fuse laser point clouds with synchronous data streams in a unified coordinate system, identify obstacles using a semantic segmentation network, and track and predict dynamic objects to obtain a layered world model. The initial module is used to obtain the initial inspection path based on the hierarchical world model and task requirements; The pose module is used to generate candidate observation poses that meet the shooting requirements based on the initial inspection path and the conditional generative adversarial network cGAN, based on the inspection target and environmental constraints. The optimization module is used to model pose quality assessment as a sequential decision problem through a reinforcement learning filter, optimize candidate observation poses under preset constraints, and obtain the optimal pose combination. The path planning module is used to map the path planning problem to a graph structure based on the initial inspection path and the optimal pose combination, and to optimize the path to obtain the globally optimal route.

8. The UAV intelligent inspection path optimization device based on multi-source perception according to claim 7, characterized in that, The acquisition module is used to acquire airborne sensor data streams and external data, perform preprocessing, and acquire a synchronized data stream in a unified coordinate system. The preprocessing includes: Implement intrinsic and extrinsic parameter management and coordinate transformation link fixing for each sensor; Establish transformation relationships from various sensor coordinate systems to a unified world coordinate system; The required external data is also resampled to a unified world coordinate system and topologically consistent with the no-fly zone boundary, and finally a synchronized data stream is output.

9. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the UAV intelligent inspection path optimization method based on multi-source perception as described in any one of claims 1-6.

10. A computer storage medium, characterized in that, The computer storage medium contains a computer program, which, when executed by a processor, implements the steps of the UAV intelligent inspection path optimization method based on multi-source perception as described in any one of claims 1-6.

Images