Multi-sensor-based urban unmanned aerial vehicle autonomous inspection planning method and system

Abstract

The application discloses a multi-sensor-based urban unmanned aerial vehicle autonomous inspection planning method and system, and belongs to the technical field of unmanned aerial vehicle inspection planning; through a multi-modal sensor array, visual images and laser point clouds of the urban environment are synchronously collected, an optical-geometric contradiction feature space is constructed based on a Fresnel equation, and a reflection penetration contradiction factor and a semantic conflict vector of a heterogenous sensor in a glass curtain wall and a mirror surface water body scene are extracted; the extracted reflection penetration contradiction factor and the semantic conflict vector are input into a constructed non-Gaussian causal graph model, through counterfactual reasoning, a sensor illusion that violates physical intuition is stripped, and a non-illusion semantic occupancy grid containing a real transparent obstacle is reconstructed; according to the reconstructed non-illusion semantic occupancy grid, through a topological manifold planning algorithm, an unmanned aerial vehicle autonomous inspection trajectory that avoids virtual bodies and transparent obstacles is generated, and when a dynamic semantic misjudgment risk is detected, a trajectory re-planning based on causal intervention is triggered.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) inspection planning technology, specifically to a method and system for autonomous inspection planning of urban UAVs based on multiple sensors. Background Technology

[0002] Traditional obstacle detection relies on geometric information; that is, if there is a point cloud, it is considered to be an object. However, in complex urban artificial environments, sensors themselves can produce "illusions" that defy physical intuition, and the illusions from different sensors can contradict each other. A typical scenario is a glass curtain wall and a mirror-like body of water. Visual algorithms may identify the reflection of another building or the sky in the glass as a real space with depth and texture. Some signals from ultrasonic or lidar may penetrate the glass and measure objects inside, while others are reflected by the glass itself. This results in the fusion of a geometrically fragmented "virtual body," while missing the real but transparent obstacle, the glass. More insidiously, if the vision detects a "pedestrian" or "vehicle" in the glass reflection, the system may mistakenly believe that there are dynamic traffic participants around the object, making unnecessary emergency avoidance and pushing itself into other dangerous areas. This error has risen from "geometric false detection" to "semantic misjudgment." Traditional fusion methods based on Bayesian probability are powerless against such conflicting structures because they assume that "sensor noise follows a Gaussian distribution," and suffer from the defect of semantic-level illusion conflicts between different sensors. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for autonomous inspection planning of urban drones based on multiple sensors, in order to solve the technical problems mentioned in the background art.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] A multi-sensor-based autonomous inspection planning method for urban drones includes:

[0006] By using a multimodal sensor array, visual images and laser point clouds of the urban environment are collected simultaneously. Based on Fresnel equations, an optical-geometric contradiction feature space is constructed, and the reflection and penetration contradiction factors and semantic conflict vectors of heterogeneous sensors in glass curtain wall and mirror water body scenarios are extracted.

[0007] The extracted reflection penetration contradiction factor and semantic conflict vector are input into the constructed non-Gaussian causal graph model. Through counterfactual reasoning, sensor illusions that violate physical intuition are stripped away, and a non-illusion semantic occupying grid containing real transparent obstacles is reconstructed.

[0008] Based on the reconstructed non-illusion semantic occupancy grid, a topological manifold planning algorithm is used to generate an autonomous inspection trajectory for the UAV that avoids virtual objects and transparent obstacles. When a risk of dynamic semantic misjudgment is detected, trajectory replanning based on causal intervention is triggered.

[0009] Furthermore, when constructing the optical-geometric contradictory feature space based on the Fresnel equation, the polarization degree is calculated by acquiring images with different polarization angles, and the echo pulse width and intensity are obtained. The visual polarization degree, laser echo pulse width, and the difference between visual depth and laser depth are mapped to a high-dimensional space to construct the optical-geometric contradictory feature space.

[0010] Furthermore, based on the constructed optical-geometric contradictory feature space, the theoretical reflectivity under the current observation perspective is calculated by combining the Fresnel equation, and the actual reflectivity is calculated by combining the peak intensity of the echo signal received by the lidar with the maximum reflection intensity calibrated by the system.

[0011] The reflection-penetration discrepancy factor is calculated by combining theoretical and actual reflectivity with pulse broadening.

[0012] Furthermore, when extracting semantic conflict vectors, the acquired visual polarization images are used to extract visual dynamic semantic labels through a semantic segmentation network; at the same time, the acquired full-waveform point cloud data of the LiDAR is used to calculate the geometric occupancy probability of the laser point cloud in the corresponding line of sight direction; combined with the calculated reflection-penetration contradiction factor as weight, the semantic conflict vector is calculated.

[0013] Furthermore, when constructing the non-Gaussian causal graph model, nodes are defined to include sensor observations, physical medium properties, real environment states, and semantic labels, while edges represent the causal relationship between physical optics and acoustic propagation. Heavy-tailed distributions are used to model non-Gaussian noise.

[0014] Furthermore, based on the constructed non-Gaussian causal graph model, the extracted reflection-penetration contradiction factor is used as the hallucination prior weight to perform three-stage counterfactual reasoning on each observed pixel to determine whether the observation is a sensor hallucination.

[0015] Furthermore, when reconstructing the non-illusion semantic occupancy grid, the visual semantics after removing illusions are fused with the laser point cloud, and at the location indicated by the reflection-penetration contradiction factor where there is a transparent medium, a transparent obstacle occupancy layer is forcibly generated, and the non-illusion semantic occupancy grid is output.

[0016] Furthermore, based on the non-illusion semantic occupancy grid, a topological manifold space is constructed, and when generating the initial trajectory, the non-illusion semantic occupancy grid is received and mapped from the discrete three-dimensional Euclidean space to a continuous topological manifold space to generate an initial inspection trajectory that avoids virtual bodies and transparent obstacles.

[0017] Furthermore, when detecting the risk of dynamic semantic misjudgment, the reflection and penetration contradiction factor and semantic conflict vector of the local environment area ahead are calculated in real time, and a dynamic semantic misjudgment risk index is constructed. Data analysis is performed on the dynamic semantic misjudgment risk index to determine whether there is a high probability of dynamic semantic misjudgment risk ahead of the current trajectory.

[0018] When triggering trajectory replanning based on causal intervention, the do operator is executed on the false dynamic target node in the causal graph model to cut off its causal influence on the UAV obstacle avoidance decision, and the inspection trajectory is regenerated based on the non-illusion semantic occupying grid after intervention.

[0019] A multi-sensor-based urban drone autonomous inspection and planning system includes:

[0020] Feature space construction and processing module: Through a multimodal sensor array, visual images and laser point clouds of the urban environment are collected simultaneously. Based on Fresnel equations, an optical-geometric contradiction feature space is constructed, and the reflection and penetration contradiction factors and semantic conflict vectors of heterogeneous sensors in glass curtain wall and mirror water scene are extracted.

[0021] The non-hallucinatory semantic occupancy grid reconstruction module inputs the extracted reflection penetration contradiction factor and semantic conflict vector into the constructed non-Gaussian causal graph model. Through counterfactual reasoning, it strips away sensor hallucinations that violate physical intuition and reconstructs a non-hallucinatory semantic occupancy grid containing real transparent obstacles.

[0022] Autonomous Inspection Trajectory Optimization Module: Based on the reconstructed non-illusion semantic occupancy grid, the module generates an autonomous inspection trajectory for the UAV that avoids virtual objects and transparent obstacles through a topological manifold planning algorithm. When a risk of dynamic semantic misjudgment is detected, the module triggers trajectory replanning based on causal intervention.

[0023] Compared to existing solutions, the beneficial effects achieved by this invention are:

[0024] This invention calculates the contradiction factor using Fresnel equations and pulse broadening, modeling from a physical mechanism perspective and effectively solving the problem that traditional probabilistic models are powerless to handle structural conflicts. By introducing a reflection-penetration contradiction factor as a weight for the semantic conflict vector, the system can clearly distinguish between real dynamic obstacles and false dynamic reflections on glass, preventing drones from misidentifying glass reflections as pedestrians or vehicles and triggering unnecessary emergency avoidance maneuvers. This also prevents drones from colliding with other real obstacles due to incorrect avoidance, significantly improving flight safety in complex urban environments. Furthermore, by using a degree-of-freedom adaptive Student-t distribution instead of a Gaussian distribution, its heavy-tailed characteristics naturally accommodate sensor behavior in complex environments. The significant deviations in observations generated by the medium avoid the misclassification of structural illusions as low-probability but reasonable noise and their retention in the fusion results under the Gaussian assumption. By intervening with the do operator and counterfactual reasoning, illusory components are accurately identified and removed at the causal level. Global path search using topological manifold space avoids the dimensionality curse and local minima problems caused by direct global optimization in high-resolution 3D grids. Local replanning is performed in conjunction with a conservative safe grid after causal intervention, which not only ensures the response speed of replanning but also ensures that the UAV always has a physically safe escape or detour trajectory in extreme sensor illusion scenarios, significantly improving the engineering practicality and flight safety of the system. Attached Figure Description

[0025] The invention will now be further described with reference to the accompanying drawings.

[0026] Figure 1 This is a flowchart of the autonomous inspection planning method for urban drones based on multiple sensors according to the present invention.

[0027] Figure 2 This is a block diagram of the multi-sensor-based urban drone autonomous inspection and planning system of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Example 1, such as Figure 1 As shown, this invention is a multi-sensor-based autonomous inspection and planning method for urban unmanned aerial vehicles (UAVs), comprising:

[0030] By using a multimodal sensor array, visual images and laser point clouds of the urban environment are simultaneously acquired. Based on the Fresnel equation, an optical-geometric contradiction feature space is constructed, and reflection-penetration contradiction factors and semantic conflict vectors of heterogeneous sensors in glass curtain wall and mirror water body scenarios are extracted; specifically:

[0031] By using a multimodal sensor array mounted on a drone and employing a timestamp synchronization mechanism, visual polarization images of the urban environment and full-waveform point cloud data from a lidar system are acquired. Light intensity data at different polarization angles are extracted from the visual polarization images, and the waveform features and spatial coordinates of the echo pulses are extracted from the full-waveform point cloud data from the lidar system to obtain the maximum polarization light intensity. Minimum polarized light intensity echo pulse full width at half maximum (FWHM) Transmit pulse full width at half maximum (FWHM) Visual depth estimation and lidar ranging values ;

[0032] When constructing an optical-geometric contradictory feature space based on Fresnel equations, the degree of polarization representing the surface optical properties is calculated based on the acquired light intensity and waveform data. Laser echo pulse broadening, representing the medium's penetration characteristics. and depth differences representing spatial geometric inconsistencies Among them, degree of polarization The calculation formula is: In the formula, This represents the light intensity value in the direction of maximum polarization in the visual polarization image; This represents the light intensity value in the direction of minimum polarization in the visual polarization image.

[0033] Pulse broadening The calculation formula is: In the formula, The full width at half maximum (FWHM) of the echo pulse received by the lidar in the time dimension; The full width at half maximum (FWHM) of the original laser pulse emitted by the time-dimension lidar;

[0034] Depth difference The calculation formula is: In the formula, The spatial dimension is the scene depth value estimated by a visual algorithm; The spatial dimension represents the scene distance value measured by lidar time-of-flight.

[0035] By concatenating the features from the three dimensions mentioned above, an optical-geometric contradictory feature space vector is constructed. ;

[0036] In the constructed optical-geometric contradictory feature space Based on this, and using the Fresnel equations, the theoretical reflectivity from the current observation perspective is calculated. And combined with the peak intensity of the echo signal received by the lidar Compared with the system's calibrated maximum reflection intensity Calculate the actual reflectivity Among them, theoretical reflectivity The calculation formula is: In the formula, The refractive index of air, The preset refractive index of the glass or water medium. The laser incident angle, The angle of refraction is calculated according to Snell's law;

[0037] Actual reflectivity The calculation formula is: ;

[0038] The reflection-penetration discrepancy factor is calculated by combining theoretical and actual reflectivity with pulse broadening. ; Reflection and penetration contradiction factors The calculation formula is: In the formula, is the standard deviation of the systematic error in reflectivity measurement; exp() is the exponential function; It is the hyperbolic tangent function;

[0039] When extracting semantic conflict vectors, the acquired visual polarization images are used to extract visual dynamic semantic labels through a semantic segmentation network. For example, dynamic traffic participants such as pedestrians and vehicles; at the same time, using the collected full-waveform point cloud data from lidar, the geometric occupancy probability of the lidar point cloud in the corresponding line-of-sight direction is calculated. Geometric occupancy probability of laser point cloud The calculation expression is:

[0040] In the formula, k is the traversal index of the echo pulse, which ranges from 1 to N, and N is the total number of echo pulses detected in the current line of sight, i.e., a single laser scanning line. The peak reflection intensity of the k-th echo pulse in the current line of sight; The maximum effective reflection intensity reference value calibrated for the lidar system; This represents the arrival time of the k-th echo pulse in the current line-of-sight direction; The arrival time of the main echo pulse with the highest reflection intensity among all echo pulses in the current line of sight; The time width parameter of the lidar echo pulse, i.e., the standard deviation, is used to characterize the pulse's broadening and attenuation characteristics on the time axis.

[0041] Combined with the calculated reflection-penetration contradiction factor As weights, the semantic conflict vector is calculated. ;

[0042] Among them, semantic conflict vector The calculation formula is as follows: In the formula, As an indicator function, when visual dynamic semantic labels Belongs to the dynamic target set When the time condition is met, the value is 1; otherwise, it is 0. A predefined set of semantic tags for dynamic traffic participants; This is a three-dimensional unit vector representing the sensor's observation direction.

[0043] It should be noted that this step, by introducing polarization optical characteristics and the physical characteristics of the full waveform of laser, breaks the traditional method's pure geometric fusion paradigm that relies solely on depth maps and point cloud coordinates. By constructing an optical-geometric contradictory feature space, the contradictory phenomenon of partial reflection and partial penetration of the sensor when encountering glass / water is quantified at the physical level. By calculating the reflection-penetration contradiction factor and semantic conflict vector, the contradiction at the physical layer is mapped to the semantic layer, accurately locating false dynamic targets generated by visual algorithms due to glass reflection, and geometric virtual objects generated by lidar due to penetration.

[0044] Traditional Bayesian fusion methods assume that sensor errors are random Gaussian noise, while reflections / penetration caused by glass curtain walls are systemic physical structural conflicts. This step calculates the conflict factor using Fresnel equations and pulse broadening, modeling from a physical mechanism perspective, effectively solving the problem that traditional probabilistic models are powerless to handle structural conflicts. By introducing the reflection-penetration conflict factor as a weight for the semantic conflict vector, the system can clearly distinguish between real dynamic obstacles and false dynamic reflections on the glass, preventing drones from misidentifying glass reflections as pedestrians or vehicles and triggering unnecessary emergency avoidance, thus preventing drones from colliding with other real obstacles due to incorrect avoidance and significantly improving flight safety in complex urban environments.

[0045] It should be noted that in this embodiment, conventional point cloud filtering or visual confidence weighting schemes in the field are abandoned. Instead, the difference in polarization intensity and the broadening of laser echo waveforms (the pulse will produce multi-peak or broadening effects when penetrating and reflecting glass) are creatively used to construct contradictory feature spaces, providing a novel heterogeneous sensor conflict quantification method based on physical optics and acoustic propagation mechanisms.

[0046] The extracted reflection penetration contradiction factor and semantic conflict vector are input into the constructed non-Gaussian causal graph model. Through counterfactual reasoning, sensor illusions that violate physical intuition are stripped away, and a non-illusion semantic occupying grid containing real transparent obstacles is reconstructed.

[0047] When constructing a non-Gaussian causal graph model for sensor hallucinations, the reflection-penetration inconsistency factor based on the output is considered. semantic conflict vector Construct a non-Gaussian causal graph model ,in, For a set of causal nodes, For a set of directed causal edges, It is the set of structural equations between nodes;

[0048] Define the set of causal nodes The meanings of each node are as follows:

[0049] For physical medium attribute nodes, the value is a set of discrete states. , , , These correspond to transparent media, highly reflective media, and opaque media, respectively. For real-world geometric state nodes, represent the three-dimensional spatial position of real obstacles in the direction observed by the sensor; As a visual sensor observation node, it carries the acquired visual depth estimate. Visual semantic tags ; As a lidar observation node, it carries the acquired laser ranging values. With echo pulse broadening ; These are semantic label nodes, representing the semantic categories that actually exist in the direction of observation;

[0050] Define a set of directed causal edges It expresses the causal relationship of the physical propagation mechanism, specifically:

[0051] This indicates that the properties of the physical medium determine the optical reflection and refraction behavior of visual observation;

[0052] This indicates that the physical properties of the medium determine the ratio of laser signal penetration to reflection;

[0053] This indicates that the geometric projection of visual observation is determined by the actual environmental conditions;

[0054] This indicates that the actual environmental conditions determine the geometric ranging of the lidar;

[0055] This indicates that the actual environmental state determines the actual semantic label;

[0056] This indicates that the physical medium properties affect the correctness of the semantic tag, such as reflection leading to semantic misjudgment;

[0057] Define the set of structural equations For visual observation nodes and lidar observation nodes The Student-t distribution with adaptive degrees of freedom is used instead of the traditional Gaussian distribution for non-Gaussian noise modeling, specifically including:

[0058] The visual observation likelihood function is:

[0059] In the formula, The likelihood value is the visual observation value. The parameter represents the degrees of freedom of the Student-t distribution for visual observation. It controls the thickness of the distribution tails. The smaller the value, the thicker the tails and the higher the tolerance for outlier observations. The scale parameter for visual observation represents the degree of dispersion of the fundamental measurements of the visual sensor; It is a conditional mean function of visual observation, determined by both the real environmental state and the properties of the medium; It is a gamma function;

[0060] The likelihood function for lidar observation is:

[0061] In the formula, The likelihood value is the result of lidar observation. The degrees of freedom parameters for the Student-t distribution observed by lidar have the same meaning as... ; The scale parameter for lidar observations represents the degree of dispersion in the fundamental lidar measurements. The conditional mean function for lidar observation is determined by both the actual environmental conditions and the properties of the medium.

[0062] Among them, the conditional mean function of visual observation Conditional mean function of lidar observations According to the media properties Define the state in segments:

[0063] In the formula, This represents the depth value of the virtual image generated on the surface of the reflecting medium. This represents the true surface depth of the opaque obstacle.

[0064] In the formula, This represents the depth of the object behind the transparent medium. This represents the true depth value of the physical surface of the medium. This is the proportionality coefficient for laser signals to penetrate transparent media, with a value range of [0,1].

[0065] When identifying and removing sensor hallucinations based on counterfactual reasoning, the extracted reflection-penetration contradiction factor is used as a priori weight for hallucination based on the constructed non-Gaussian causal graph model. A three-stage counterfactual reasoning process is then performed on each observed pixel to determine whether the observation is a sensor hallucination. Specifically, this includes:

[0066] When performing the first phase of cause ablation, based on the current observations... , And the extracted reflection and penetration contradiction factors Nodes that infer physical medium properties through Bayesian posterior reasoning and real environment state nodes The posterior distribution:

[0067] In the formula, For the joint posterior probability distribution, it represents the probability that the physical medium property M and the real environment state S are both true, given the current visual observation value, the lidar observation value, and the reflection and penetration contradiction factor. Let S be the visual observation likelihood function, representing the likelihood of the visual sensor observing the current data under the assumption that the real environment state is S and the physical medium property is M. The conditional probability reflects the compatibility between the physical mechanism and visual observation; Let S be the likelihood function for lidar observation, representing the likelihood of the lidar observing the current data under the assumption that the real environment state is S and the physical medium property is M. The conditional probability reflects the compatibility between the laser signal propagation mechanism and radar observation; Let be the prior probability of the medium property, representing the reflection-penetration inconsistency factor given the prior probability. In the case of a certain state, the probability distribution of the physical medium property M is in a certain state, such as transparent, highly reflective or opaque. This prior is dynamically modulated by conflicting factors, which guide the model to be more inclined to the specific medium assumption when the conflict is severe. The edge prior probability of the environment state represents the inherent probability distribution of the real environment state S in the absence of current sensor observation data. It is usually set based on the geometric continuity of the scene or historical map information. This is the actual visual observation value of the current pixel; This represents the actual lidar observation value for the current pixel; is the normalization constant for the Bayesian posterior;

[0068] Among them, the prior probability of medium properties The contradiction factor is reflected and penetrated. modulation:

[0069] In the formula, It is the probability that the medium is "transparent" given a contradictory factor; It serves as a smoothing coefficient for the prior distribution of medium properties, preventing the zero-probability problem;

[0070] In the formula, It is the probability that the medium is "highly reactive" given a given contradictory factor; is the a priori scaling factor of the reflective medium relative to the opaque medium under non-transparent conditions, and its value ranges from [0,1].

[0071] In the formula, It is the probability that the medium is "opaque" given a contradictory factor;

[0072] The most probable state of the medium properties is obtained through maximum a posteriori estimation. The most likely estimate of the actual environmental state And calculate the residual noise term:

[0073] In the formula, This is the noise term of the causal residuals from visual observations.

[0074] In the formula, This is the causal residual noise term from lidar observations;

[0075] During the second phase of intervention, the do operator is applied to the causal graph model to force the physical medium attribute nodes to be set to a transparent medium state. Cut off the original causal path from the medium property node to the observation node, and calculate the counterfactual observation expectation after intervention:

[0076] In the formula, The expected value of visual counterfactual observation after intervention, that is, the inferred true state. Under the assumption that the medium is transparent, the theoretically visible visual information should be as follows:

[0077] In the formula, The expected value of the counterfactual observation of the lidar after intervention, that is, in the inferred true state. Under the assumption that the medium is transparent, the theoretical value that the lidar should measure is:

[0078] When determining hallucinations in Phase 3, calculate the semantic consistency shift of visual observations before and after the intervention. With geometric offset Combined with the extracted semantic conflict vector Length of the module Calculate the probability of hallucination :

[0079] In the formula, This refers to the semantic consistency offset before and after the intervention; Semantic labels obtained from counterfactual reasoning after intervention;

[0080] In the formula, The geometric offset before and after the intervention;

[0081] In the formula, The probability that the currently observed pixel is identified as a sensor illusion; The weighting coefficient for semantic offset; The weighting coefficient for the geometric offset; Let be the weight coefficients of the semantic conflict vector magnitude, and satisfy . + + =1;

[0082] It should be explained that the specific values ​​of the weighting coefficients involved in this embodiment are not limited. They can all be calibrated using statistical or machine learning methods based on a large amount of sensor data collected in the early stages of actual flight or simulation environments, or they can be determined based on grid search and ablation experiments in the simulation environment.

[0083] When the probability of hallucination Greater than the preset hallucination detection threshold At that time, the observed pixel is labeled as a hallucination observation, its visual semantic label and geometric depth value are stripped away, and the true environment state estimate output by counterfactual reasoning is retained. With medium property estimation Hallucination detection threshold It can be determined based on big data from previous simulation tests, such as the median of all hallucination probabilities obtained from previous simulation tests;

[0084] When reconstructing a non-illusion semantically occupied grid containing real transparent obstacles, the results are based on the output illusion labeling and the estimation of the real environment state. With medium property estimation Reconstructing 3D non-illusion semantics occupying the grid Specifically:

[0085] Initialize 3D semantic occupancy grid For observed pixels not marked as hallucinations, estimate their corresponding real-world conditions. Project onto the raster space and write the geometric occupancy value and semantic label:

[0086] In the formula, These are three-dimensional coordinates in the raster space. This is the geometric occupancy value of the grid, where 1 indicates that it is occupied and 0 indicates that it is free. The true semantic labels output by counterfactual reasoning;

[0087] The medium properties are estimated as follows A set of pixels, in the corresponding viewing direction, forcibly generates a transparent obstacle occupancy layer, the occupancy probability of the transparent obstacle... The contradiction factor is reflected and penetrated. Probability of hallucination The decision is made jointly, and the corresponding expression is:

[0088] In the formula, The three-dimensional coordinates of the transparent medium surface in the grid space; This is a scaling factor for the probability of a transparent obstacle being occupied, with a value range of (0,1).

[0089] Overlay the transparent obstacle occupancy layer onto the initial grid, and output a non-illusion semantic occupancy grid. :

[0090] In the formula, transparent_obstacle is the label for a transparent obstacle.

[0091] It should be noted that this step integrates the extracted physical layer contradiction features (reflection and penetration contradiction factors) and semantic layer conflict features (semantic conflict vectors) into a non-Gaussian causal graph model for joint reasoning. Through three-stage counterfactual reasoning of attribution, intervention, and judgment, it fundamentally distinguishes between random errors caused by sensor noise and structural illusions caused by physical media. It accurately removes false observations that violate physical intuition, such as false pedestrians generated by glass reflection and indoor virtual images measured through glass, from the environmental perception results and reconstructs occupancy information at the real position of transparent obstacles, outputting an illusion-free semantic occupancy grid that can be directly used by downstream path planning.

[0092] Traditional methods assume that sensor noise follows a Gaussian distribution, which fails to effectively model systemic and structural conflicts arising in scenarios such as glass curtain walls. This step replaces the Gaussian distribution with a degree-of-freedom adaptive Student-t distribution. Its heavy-tailed characteristic naturally accommodates significant deviations in observations caused by sensors on complex media, avoiding the misclassification of structural illusions as low-probability but reasonable noise and their inclusion in the fusion result under the Gaussian assumption. Furthermore, traditional multi-sensor fusion methods, such as Kalman filtering, particle filtering, and confidence weighting, essentially compromise the probability of contradictory observations, resulting in a geometrically fragmented virtual object that is neither a reflection nor a real object. This step, through the intervention of the do operator and counterfactual reasoning, answers the counterfactual question: "If this surface is transparent glass, what should the observation look like?" This allows for the precise identification and removal of illusory components at the causal level, rather than simply making probability compromises.

[0093] Finally, traditional methods often lose information about the existence of transparent obstacles while removing illusions. This step uses posterior estimation of medium properties to remove false observations and uses the reflection-penetration contradiction factor and illusion probability to jointly calculate the occupancy probability of transparent obstacles. This forces the generation of an occupancy layer on the real physical surface of the transparent medium, ensuring that the UAV path planner can perceive the glass curtain wall as a real but transparent obstacle, thus fundamentally avoiding flight accidents that collide with the glass curtain wall.

[0094] It should be explained that conventional techniques in this field typically address sensor conflicts by increasing sensor redundancy (e.g., adding millimeter-wave radar) or post-processing filtering (e.g., morphological opening and closing operations to remove isolated noise). These methods cannot solve the problem of false pedestrian reflections on glass leading to emergency avoidance at the semantic level. This step creatively introduces the counterfactual framework from causal reasoning into the field of UAV multi-sensor fusion. Through physical causal graph modeling and do operator intervention, a novel sensor illusion stripping paradigm based on causal mechanisms rather than statistical correlation is achieved, which has outstanding substantive features and significant progress.

[0095] Based on the reconstructed non-illusion semantic occupancy grid, a topological manifold planning algorithm is used to generate an autonomous inspection trajectory for the UAV that avoids virtual objects and transparent obstacles. When a risk of dynamic semantic misjudgment is detected, trajectory replanning based on causal intervention is triggered. Specifically:

[0096] Based on a non-illusion semantic occupancy grid, a topological manifold space is constructed. When generating the initial trajectory, the non-illusion semantic occupancy grid is received and mapped from a discrete 3D Euclidean space to a continuous topological manifold space to generate an initial inspection trajectory that avoids virtual bodies and transparent obstacles; specifically:

[0097] First, in the non-hallucination semantics occupy the grid Extract all geometric occupancy values. The set of free grid cells is used to construct a safe free space by extending the safety boundaries of transparent obstacles (labeled transparent_obstacle) and regular opaque obstacles outward through a morphological dilation operation. In safe and free space Within, extract the three-dimensional connected components as a set of topological nodes. Calculate the shortest collision-free path between adjacent connected components and use it as the set of topological edges. Thus, a topology graph is constructed. ;

[0098] Secondly, in the topology graph Above, the A* algorithm is used to search for the drone's current starting point. To the inspection target point Global optimal topology path sequence ;

[0099] It should be explained that the A* (A-star) algorithm is a heuristic shortest path search algorithm used to find the path from the starting point to the destination with the minimum cost in maps, mazes, road networks, and game maps. It combines the advantages of two classic algorithms: Dijkstra's algorithm and greedy search.

[0100] Dijkstra's algorithm: It only calculates the distance already traveled, and the result is always the shortest, but it traverses a large number of nodes, so it is slow.

[0101] Greedy search: It only estimates the remaining distance, which is fast, but it is easy to take detours and does not guarantee the optimal solution.

[0102] The A* algorithm combines the distance already traveled with the estimated remaining distance, guaranteeing the shortest path while significantly improving search efficiency. It is currently the most mainstream pathfinding algorithm in industry. The A* algorithm is used to search for a path from the drone's current starting point. To the inspection target point Global optimal topology path sequence This is an existing conventional technical solution, and the specific implementation steps will not be elaborated here;

[0103] Finally, the globally optimal topology path sequence As an initial guess, a trajectory optimization objective function is constructed based on minimizing control energy and obstacle repulsion. The corresponding expression is:

[0104] In the formula, The total flight time for the planned trajectory; For drones in time 3D position vector; The third derivative of the UAV's position vector with respect to time is used to measure control smoothness. For the repulsive potential field of conventional obstacles; Let be the weighting coefficients of the repulsive potential field of the transparent obstacle, and satisfy . + =1; Let be the repulsive potential field function of a conventional obstacle. Let the repulsive potential function of the transparent obstacle be given by the following expression:

[0105] In the formula, A set of grid indices occupied by regular opaque obstacles; For the first The three-dimensional center coordinates of a typical opaque obstacle; The safe expansion distance parameter for conventional obstacles;

[0106] In the formula, A set of grid indices occupied by transparent obstacles; For the first The three-dimensional center coordinates of a transparent obstacle; The safe expansion distance parameter for transparent obstacles; For the calculation of the first The probability of occupancy of a transparent obstacle grid;

[0107] A smooth initial inspection trajectory is generated using a quadratic programming solver. The corresponding expression is: In the formula, The smooth initial inspection trajectory is generated by a quadratic programming solver and represents the optimal three-dimensional position vector of the UAV at time t. For the optimization operator, let f(x) represent the search for the trajectory function that minimizes the integral value of the objective function. ;

[0108] When real-time detection and assessment of the risk of dynamic semantic misjudgment, the drone follows the initial inspection trajectory. During flight, an onboard multimodal sensor array is used to continuously acquire real-time visual polarization images and laser point cloud data within the forward field of view (FOV); combined with the methods of the previous steps, the reflection-penetration discrepancy factor of the local environmental region in front is calculated in real time. semantic conflict vector ;

[0109] To quantify the risk of dynamic semantic misjudgment by drones due to the surrounding environment, such as mistaking a fake pedestrian reflected in glass for a real pedestrian and triggering incorrect avoidance, a dynamic semantic misjudgment risk index is constructed. The corresponding expression is:

[0110] In the formula, Risk weight coefficients for the semantic conflict vector magnitude; The risk weight coefficient for the reflection and penetration contradiction factor; Spatial coordinates Real-time visual semantic labels at the location; A predefined set of semantic tags for dynamic traffic participants; This is an indicator function that takes the value 1 if the condition is true, and 0 otherwise. This represents the current three-dimensional forward field of view area of ​​the UAV's onboard sensors.

[0111] Set dynamic misjudgment risk threshold When calculated in real time If a high probability of dynamic semantic misjudgment risk is detected ahead of the current trajectory, a trajectory replanning mechanism based on causal intervention is immediately triggered.

[0112] When the reprogramming mechanism is triggered, the system no longer relies on the original observation data, which may contain semantic illusions. Instead, it directly performs causal intervention operations on the non-Gaussian causal graphical model constructed in the previous steps. Specifically:

[0113] For high-risk areas The physical medium property node within the object is intervened by the do operator, forcing it to be set to a completely opaque and highly reflective entity state. This involves cutting off potential spurious dynamic semantic interference from the original observations and calculating the update occupancy probability after intervention. The corresponding expression is:

[0114] In the formula, Spatial coordinates The lidar ranging value at the location; Spatial coordinates Visual depth estimate at the location; The occupancy probability of the original grid before intervention;

[0115] Using the updated occupancy probability Replace the original non-hallucination semantics occupying the grid. The occupancy value of the corresponding region is used to generate a conservative safety grid after intervention. ;

[0116] Based on the current real-time location of the drone Starting with the conservative safety grid In the constructed updated topological manifold space, an emergency avoidance penalty term is introduced. The trajectory optimization objective function is re-solved to generate a safe replanning trajectory that avoids potential virtual bodies and transparent obstacles. The corresponding expression is:

[0117] In the formula, The objective function for trajectory optimization during the replanning phase; Optimize the objective function for the defined initial trajectory;

[0118] This is the weighting coefficient for the emergency avoidance penalty, with a value range of (0,1). The remaining flight time for replanning the trajectory; High-risk area The coordinates of the three-dimensional geometric center; This is the distance decay parameter for the repulsive potential field during emergency avoidance.

[0119] It should be noted that this step transforms the "clean" environment model (without hallucination semantics occupying the grid) output from the previous step into physical motion commands that the UAV can execute. Through topological manifold planning algorithms, the global connectivity and local smoothness of the trajectory are guaranteed at the mathematical level. Differentiated repulsive potential fields are designed for transparent obstacles and conventional obstacles. At the same time, during the flight execution phase, dynamic semantic misjudgment risk indicators are calculated in real time. When the risk exceeds the standard, trajectory replanning based on causal intervention is triggered, forcibly treating suspicious high-reflectivity areas as absolutely impenetrable entities. Thus, at the control execution level, the safety link of perception, planning and risk avoidance is completely closed.

[0120] Traditional path planning algorithms typically treat all obstacles the same and perform simple geometric expansion. This step, however, distinguishes between the repulsive potential fields of regular obstacles and transparent obstacles, and uses the calculated probability of transparent obstacles being occupied as a modulation factor for the potential field strength. This allows the UAV to maintain a greater safety redundancy distance when facing glass curtain walls, effectively preventing collision accidents caused by reflections from the glass surface or range drift.

[0121] In urban patrols, the reflection of fake pedestrians or vehicles in glass curtain walls can easily cause drones to trigger emergency avoidance, i.e., ghost braking, which may lead to collisions with other real obstacles. This step constructs a dynamic semantic misjudgment risk indicator and uses the do operator to forcibly intervene in the causal graph when triggered, forcibly setting the suspicious area as an entity. This directly cuts off the interference of false dynamic semantics on the planner and ensures that the drone will not make dangerous maneuvers because of the fake people in the glass.

[0122] This step employs a topological manifold space for global path search, avoiding the dimensionality curse and local minima problems caused by direct global optimization in a high-resolution 3D grid. Combined with a conservative safety grid after causal intervention for local replanning, it ensures both the response speed of replanning and that the UAV always has a physically safe escape or detour trajectory even in extreme sensor illusion scenarios, significantly improving the system's engineering practicality and flight safety.

[0123] Example 2, as Figure 2 As shown, the urban drone autonomous inspection and planning system based on multiple sensors includes:

[0124] Feature space construction and processing module: Through a multimodal sensor array, visual images and laser point clouds of the urban environment are collected simultaneously. Based on Fresnel equations, an optical-geometric contradiction feature space is constructed, and the reflection and penetration contradiction factors and semantic conflict vectors of heterogeneous sensors in glass curtain wall and mirror water scene are extracted.

[0125] The non-hallucinatory semantic occupancy grid reconstruction module inputs the extracted reflection penetration contradiction factor and semantic conflict vector into the constructed non-Gaussian causal graph model. Through counterfactual reasoning, it strips away sensor hallucinations that violate physical intuition and reconstructs a non-hallucinatory semantic occupancy grid containing real transparent obstacles.

[0126] Autonomous Inspection Trajectory Optimization Module: Based on the reconstructed non-illusion semantic occupancy grid, the module generates an autonomous inspection trajectory for the UAV that avoids virtual objects and transparent obstacles through a topological manifold planning algorithm. When a risk of dynamic semantic misjudgment is detected, the module triggers trajectory replanning based on causal intervention.

[0127] In the several embodiments provided by this invention, it should be understood that the disclosed system can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative; for example, the division of modules is only a logical functional division, and there may be other division methods in actual implementation.

[0128] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module described above can be implemented in hardware or in the form of hardware plus software functional modules.

[0130] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the essential characteristics of the present invention.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-sensor-based autonomous inspection planning method for urban unmanned aerial vehicles (UAVs), characterized in that: include: By using a multimodal sensor array, visual images and laser point clouds of the urban environment are collected simultaneously. Based on Fresnel equations, an optical-geometric contradiction feature space is constructed, and the reflection and penetration contradiction factors and semantic conflict vectors of heterogeneous sensors in glass curtain wall and mirror water body scenarios are extracted. The extracted reflection penetration contradiction factor and semantic conflict vector are input into the constructed non-Gaussian causal graph model. Through counterfactual reasoning, sensor illusions that violate physical intuition are stripped away, and a non-illusion semantic occupying grid containing real transparent obstacles is reconstructed. Based on the reconstructed non-illusion semantic occupancy grid, a topological manifold planning algorithm is used to generate an autonomous inspection trajectory for the UAV that avoids virtual objects and transparent obstacles. When a risk of dynamic semantic misjudgment is detected, trajectory replanning based on causal intervention is triggered.

2. The urban unmanned aerial vehicle (UAV) autonomous inspection and planning method based on multiple sensors according to claim 1, characterized in that, When constructing an optical-geometric contradictory feature space based on Fresnel equations, the degree of polarization is calculated by acquiring images at different polarization angles, and the echo pulse width and intensity are obtained. The visual polarization degree, laser echo pulse width, and the difference between visual depth and laser depth are mapped to a high-dimensional space to construct the optical-geometric contradictory feature space.

3. The urban unmanned aerial vehicle (UAV) autonomous inspection planning method based on multiple sensors according to claim 2, characterized in that, Based on the constructed optical-geometric contradictory feature space, the theoretical reflectivity under the current observation perspective is calculated by combining the Fresnel equation, and the actual reflectivity is calculated by combining the peak intensity of the echo signal received by the lidar with the maximum reflection intensity calibrated by the system. The reflection-penetration discrepancy factor is calculated by combining theoretical and actual reflectivity with pulse broadening.

4. The urban unmanned aerial vehicle (UAV) autonomous inspection planning method based on multiple sensors according to claim 3, characterized in that, When extracting semantic conflict vectors, the acquired visual polarization images are used to extract visual dynamic semantic labels through a semantic segmentation network. At the same time, the acquired full-waveform point cloud data of the LiDAR is used to calculate the geometric occupancy probability of the laser point cloud in the corresponding line of sight. Combined with the calculated reflection and penetration contradiction factor as weights, the semantic conflict vector is calculated.

5. The urban unmanned aerial vehicle (UAV) autonomous inspection planning method based on multiple sensors according to claim 4, characterized in that, When constructing a non-Gaussian causal graph model, nodes are defined to include sensor observations, physical medium properties, real environment states, and semantic labels. Edges represent the causal relationship between physical optics and acoustic propagation. Heavy-tailed distributions are used to model non-Gaussian noise.

6. The urban unmanned aerial vehicle (UAV) autonomous inspection planning method based on multiple sensors according to claim 5, characterized in that, Based on the constructed non-Gaussian causal graph model, the extracted reflection-penetration contradiction factor is used as the hallucination prior weight to perform three-stage counterfactual reasoning on each observed pixel to determine whether the observation is a sensor hallucination.

7. The urban unmanned aerial vehicle (UAV) autonomous inspection planning method based on multiple sensors according to claim 6, characterized in that, When reconstructing the non-illusion semantic occupancy grid, the visual semantics after removing illusions are fused with the laser point cloud, and a transparent obstacle occupancy layer is forcibly generated at the location indicated by the reflection-penetration contradiction factor where a transparent medium exists, and the non-illusion semantic occupancy grid is output.

8. The urban unmanned aerial vehicle (UAV) autonomous inspection planning method based on multiple sensors according to claim 7, characterized in that, Based on the non-illusion semantic occupancy grid, a topological manifold space is constructed. When generating the initial trajectory, the non-illusion semantic occupancy grid is received and mapped from the discrete three-dimensional Euclidean space to the continuous topological manifold space to generate an initial inspection trajectory that avoids virtual bodies and transparent obstacles.

9. The urban unmanned aerial vehicle (UAV) autonomous inspection planning method based on multiple sensors according to claim 8, characterized in that, When detecting the risk of dynamic semantic misjudgment, the reflection and penetration contradiction factor and semantic conflict vector of the local environment area ahead are calculated in real time, and a dynamic semantic misjudgment risk index is constructed. Data analysis is performed on the dynamic semantic misjudgment risk index to determine whether there is a high probability of dynamic semantic misjudgment risk ahead of the current trajectory. When triggering trajectory replanning based on causal intervention, the do operator is executed on the false dynamic target node in the causal graph model to cut off its causal influence on the UAV obstacle avoidance decision, and the inspection trajectory is regenerated based on the non-illusion semantic occupying grid after intervention.

10. A multi-sensor-based autonomous urban drone inspection and planning system, employing the multi-sensor-based autonomous urban drone inspection and planning method as described in any one of claims 1-9, characterized in that, include: Feature space construction and processing module: Through a multimodal sensor array, visual images and laser point clouds of the urban environment are collected simultaneously. Based on Fresnel equations, an optical-geometric contradiction feature space is constructed, and the reflection and penetration contradiction factors and semantic conflict vectors of heterogeneous sensors in glass curtain wall and mirror water scene are extracted. The non-hallucinatory semantic occupancy grid reconstruction module inputs the extracted reflection penetration contradiction factor and semantic conflict vector into the constructed non-Gaussian causal graph model. Through counterfactual reasoning, it strips away sensor hallucinations that violate physical intuition and reconstructs a non-hallucinatory semantic occupancy grid containing real transparent obstacles. Autonomous Inspection Trajectory Optimization Module: Based on the reconstructed non-illusion semantic occupancy grid, the module generates an autonomous inspection trajectory for the UAV that avoids virtual objects and transparent obstacles through a topological manifold planning algorithm. When a risk of dynamic semantic misjudgment is detected, the module triggers trajectory replanning based on causal intervention.

Images