Railway scene unmanned aerial vehicle inspection method and device

Abstract

The present application relates to the technical field of railway inspection, and provides a kind of unmanned aerial vehicle inspection method and device for railway scene, wherein the method comprises: by obtaining the multisource perception data of three physical principles and different applicable scenes, such as vision image sequence, laser point cloud sequence and inertial measurement unit data, and extracting mutually independent vision re-projection constraint, laser odometry constraint and inertial pre-integral constraint from multisource perception data, then by constructing target factor graph, vision re-projection constraint, laser odometry constraint and inertial pre-integral constraint are tightly coupled and fused and globally nonlinearly optimized, thus, even in the railway scene where GPS signal is attenuated or even interrupted due to strong electromagnetic interference or physical obstruction, continuous and accurate pose solution can be achieved by fusing the constraints of multisource perception information, ensuring the navigation stability and positioning accuracy of unmanned aerial vehicle in complex railway environment.

Description

Technical Field

[0001] This invention relates to the field of railway inspection technology, and in particular to a method and apparatus for unmanned aerial vehicle (UAV) inspection in railway scenarios. Background Technology

[0002] To ensure the safety of the railway line, it is necessary to conduct regular and meticulous inspections of the railway line, its ancillary facilities, and the surrounding environment.

[0003] Currently, to meet this demand, the industry mainly adopts technical solutions such as manual inspection, track inspection vehicle inspection, and traditional drone inspection. Among them, manual inspection and track inspection vehicle inspection have inherent limitations due to factors such as efficiency, coverage, and cost; while drone inspection solutions acquire data through an aerial platform, showing great potential in terms of flexibility and coverage. They typically use drones and rely on the Global Positioning System (GPS) or BeiDou satellite navigation system for positioning and navigation to perform inspection tasks.

[0004] However, existing drone inspection solutions heavily rely on a single Global Positioning System (GPS) signal for navigation. Along railway lines, especially in areas with strong electromagnetic interference such as high-voltage overhead contact lines and substations, as well as in environments with physical obstructions like tunnels and under bridges, GPS signals are prone to attenuation or interruption. This unstable signal quality directly leads to a significant decrease in drone positioning accuracy, and can even trigger flight control system malfunctions when the signal is completely lost, causing drones to go out of control or crash. This prevents drones from conducting safe and reliable inspection operations in these critical areas, severely limiting the breadth and depth of their application. Summary of the Invention

[0005] This invention provides a method and apparatus for unmanned aerial vehicle (UAV) inspection in railway scenarios, which solves the problem that existing UAV inspection schemes rely too heavily on a single global positioning system. In environments with strong electromagnetic interference and physical obstruction, the UAVs are prone to positioning failure due to signal attenuation or loss of lock, which can lead to control anomalies and flight safety risks, severely restricting their operational reliability and coverage in key railway areas.

[0006] This invention provides a method for unmanned aerial vehicle (UAV) inspection in railway scenarios, comprising the following steps.

[0007] Acquire multi-source sensing data synchronously collected by the UAV in a railway scenario; the multi-source sensing data includes at least visual image sequences, laser point cloud sequences, and inertial measurement unit data; Visual features are extracted from the visual image sequence, and visual reprojection constraints are determined based on the reprojection residuals of visual features between different image frames in the visual image sequence. Geometric features are extracted from the laser point cloud sequence, and laser odometry constraints are determined based on the registration residuals of geometric features between consecutive point cloud frames in the laser point cloud sequence. The inertial measurement unit data between the acquisition times of two key frames in the visual image sequence is pre-integrated to obtain the pre-integration processing result, and the inertial pre-integration constraint is determined based on the pre-integration processing result; Using the six-degree-of-freedom pose of the UAV at multiple keyframes as the pose nodes of the factor graph, and using the visual reprojection constraint, the laser odometry constraint, and the inertial pre-integration constraint as the edges of the factor graph, a target factor graph is constructed. The pose information of the UAV is obtained by performing nonlinear optimization on the target factor map.

[0008] According to the present invention, a method for unmanned aerial vehicle (UAV) inspection in a railway scenario includes extracting visual features from the visual image sequence, comprising: Descriptor features are extracted from the visual image sequence as the visual features.

[0009] According to the present invention, a method for unmanned aerial vehicle (UAV) inspection in a railway scenario includes extracting geometric features from the laser point cloud sequence, comprising: Line and surface features are extracted from the laser point cloud sequence as the geometric features.

[0010] According to the present invention, a method for unmanned aerial vehicle (UAV) inspection in a railway scenario includes performing nonlinear optimization on the target factor map to obtain the pose information of the UAV, comprising: The target cost function of the target factor map is determined; the target cost function is determined based on the first weight of the visual reprojection constraint residual, the second weight of the laser odometry constraint residual, and the third weight of the inertial pre-integration constraint residual, as well as the visual reprojection constraint residual, the laser odometry constraint residual, and the inertial pre-integration constraint residual; the first weight is determined based on the covariance matrix of the visual reprojection constraint; the second weight is determined based on the covariance matrix of the laser odometry constraint; the third weight is determined based on the covariance matrix of the inertial pre-integration constraint. The visual reprojection constraint residual is used to characterize the error of the visual reprojection constraint. The visual reprojection constraint residual is determined based on the difference between the observed value and the predicted value of the feature points extracted from the visual image sequence. The predicted value of the feature is determined based on the pose node and the camera projection model. The laser odometry constraint residual is used to characterize the error of the laser odometry constraint. The laser odometry constraint residual is the registration residual between the geometric features of the current frame and the geometric features of the previous frame in the laser point cloud sequence. The inertial pre-integration constraint residual is used to characterize the error of the inertial pre-integration constraint. The inertial pre-integration constraint residual is determined based on the relative motion and the inertial pre-integration quantity. The relative motion is determined based on two adjacent pose nodes in the target factor graph, and the inertial pre-integration quantity is determined based on the inertial pre-integration constraint. The pose information of the UAV is obtained by minimizing the objective cost function.

[0011] According to the present invention, a method for unmanned aerial vehicle (UAV) inspection in a railway scenario includes, prior to the nonlinear optimization of the target factor map to obtain the pose information of the UAV: Obtain the signal strength of signals in the Global Positioning System; When the signal strength is greater than or equal to a preset strength threshold, the position observation value in the global positioning system is obtained; Determine the pose nodes to be associated corresponding to the keyframes in the target factor graph that match the timestamps of the location observations; Extract the position components from the pose nodes to be associated, and determine new edges for the pose nodes to be associated based on the observation residuals between the position observations and the position components, so as to constrain the pose nodes to be associated.

[0012] According to the present invention, a method for unmanned aerial vehicle (UAV) inspection in a railway scenario includes, after performing nonlinear optimization on the target factor map to obtain the pose information of the UAV, the method further includes: Generate a baseline route based on railway line design data; Based on the pose information of the UAV, the UAV is controlled to fly along the reference route, and during the flight, dynamic obstacle perception is performed based on the multi-source perception data to obtain obstacle perception results. When the obstacle perception result indicates the presence of a dynamic obstacle, adaptive obstacle avoidance control is performed based on the UAV's pose information and the obstacle perception result to obtain a detour route, and the UAV is controlled to fly along the detour route.

[0013] According to the present invention, a method for unmanned aerial vehicle (UAV) inspection in a railway scenario includes, during flight, performing dynamic obstacle perception based on the multi-source sensing data to obtain obstacle perception results, comprising: During flight, the multi-source sensing data is input into a multi-scale target detection model to obtain the obstacle sensing results output by the multi-scale target detection model. The multi-scale target detection model includes a feature extraction backbone network, a feature pyramid, an attention module, and a classification layer; The feature extraction backbone network is used to extract texture semantic features from the visual image sequence and geometric features from the laser point cloud sequence; The feature pyramid is used to extract the texture semantic features at multiple scales to obtain multi-scale texture semantic features, and to fuse the multi-scale texture semantic features across scales to obtain multi-scale visual features. The attention module is used to fuse the multi-scale visual features and the geometric features to obtain fused features; The classification layer is used to generate the obstacle perception result based on the fusion features.

[0014] According to the present invention, a method for unmanned aerial vehicle (UAV) inspection in a railway scenario further includes: Risk targets are identified based on the multi-source sensing data; the risk targets include indicators of impact range, probability of occurrence, degree of harm, and difficulty of handling. The risk objectives are matched with a pre-constructed railway operation and maintenance knowledge graph to obtain the safety rules corresponding to the risk objectives; Based on the aforementioned safety rules, a first quantitative score for the impact range indicator, a second quantitative score for the occurrence probability indicator, a third quantitative score for the severity of harm indicator, and a fourth quantitative score for the handling difficulty indicator are determined respectively. Based on the target weight, the first quantitative score, the second quantitative score, the third quantitative score, and the fourth quantitative score are weighted and summed to obtain the risk quantitative value; The risk level is determined based on the risk quantification value, and the treatment recommendations are determined based on the risk level.

[0015] According to the present invention, a method for unmanned aerial vehicle (UAV) inspection in a railway scenario is provided, wherein the target weight includes the weight of the influence range index, the weight of the occurrence probability index, the weight of the hazard degree index, and the weight of the handling difficulty index. The steps for determining the target weight include: Determine the static basic weights; the static basic weights include the first initial weight of the influence range indicator, the second initial weight of the occurrence probability indicator, the third initial weight of the hazard degree indicator, and the fourth initial weight of the handling difficulty indicator; When the risk target is detected to be within a preset distance range of the railway mainline, the third initial weight in the static basic weight is increased to obtain the target weight; When the risk target is detected to be in motion, the second initial weight in the static basic weight is increased to obtain the target weight.

[0016] The present invention also provides a drone inspection device for railway scenarios, comprising the following units: The acquisition unit is used to acquire multi-source perception data synchronously collected by the UAV in the railway scene; the multi-source perception data includes at least visual image sequences, laser point cloud sequences and inertial measurement unit data; The first extraction unit is used to extract visual features from the visual image sequence and determine visual reprojection constraints based on the reprojection residuals of visual features between different image frames in the visual image sequence. The second extraction unit is used to extract geometric features from the laser point cloud sequence and determine the laser odometry constraint based on the registration residual of the geometric features between consecutive point cloud frames in the laser point cloud sequence. The pre-integration unit is used to perform pre-integration processing on the inertial measurement unit data between the acquisition times of two key frames in the visual image sequence, to obtain the pre-integration processing result, and to determine the inertial pre-integration constraint based on the pre-integration processing result. The construction unit is used to construct a target factor graph by using the six-degree-of-freedom pose of the UAV at multiple key frames as the pose nodes of the factor graph, and using the visual reprojection constraint, the laser odometry constraint and the inertial pre-integration constraint as the edges of the factor graph. An optimization unit is used to perform nonlinear optimization on the target factor map to obtain the pose information of the UAV.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the unmanned aerial vehicle inspection method for any of the railway scenarios described above.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the unmanned aerial vehicle (UAV) inspection method for railway scenarios as described above.

[0019] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the unmanned aerial vehicle (UAV) inspection method for any of the railway scenarios described above.

[0020] The UAV inspection method and apparatus for railway scenarios provided by this invention acquires three types of multi-source perception data with different physical principles and applicable scenarios: visual image sequences, laser point cloud sequences, and inertial measurement unit data. It extracts independent visual reprojection constraints, laser odometer constraints, and inertial pre-integration constraints from these multi-source perception data. Then, by constructing a target factor graph, it tightly couples and fuses these constraints with global nonlinear optimization. Thus, even in railway scenarios where GPS signals are attenuated or interrupted due to strong electromagnetic interference or physical obstruction, it can continuously and accurately calculate pose by fusing constraints from multi-source perception information. This solves the problems of poor positioning reliability and easy loss of control caused by relying on a single global positioning system signal in traditional UAV inspection schemes, ensuring the navigation stability and positioning accuracy of UAVs in complex railway environments. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the unmanned aerial vehicle (UAV) inspection method for railway scenarios provided by the present invention.

[0023] Figure 2 This is a schematic diagram of the process for controlling a drone to fly along a bypass route, provided by the present invention.

[0024] Figure 3 This is a block diagram of the overall architecture of the railway scene drone autonomous inspection system provided by the present invention.

[0025] Figure 4 This is a schematic diagram of the structure of the unmanned aerial vehicle (UAV) inspection device for railway scenarios provided by the present invention.

[0026] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] This invention provides a method for unmanned aerial vehicle (UAV) inspection in railway scenarios. Figure 1 This is a flowchart illustrating the unmanned aerial vehicle (UAV) inspection method for railway scenarios provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 110: Acquire multi-source perception data synchronously collected by the UAV in the railway scenario; the multi-source perception data includes at least visual image sequences, laser point cloud sequences, and inertial measurement unit data.

[0029] Specifically, firstly, multi-source sensing data can be acquired synchronously by the drone in the railway scenario. The drone can be any aerial vehicle performing the inspection task, such as a multi-rotor drone, a fixed-wing drone, or a vertical take-off and landing fixed-wing drone, etc., and this embodiment of the invention does not specifically limit it.

[0030] In a preferred embodiment, to adapt to the complex airflow environment of railway scenarios and ensure the stability of hovering observation, an industrial-grade multi-rotor UAV is selected as the carrier. These UAVs typically possess high wind resistance and long endurance to meet the needs of long-distance line inspection. The industrial-grade multi-rotor UAV integrates various types of sensors, including visible light cameras, LiDAR (Light Detection and Ranging), inertial measurement units (IMUs), and radiometric calibrated infrared thermal imagers.

[0031] Here, "railway scenario" refers to the sum of all environments and facilities involved in railway operations. The railway scenario includes not only the railway line itself, such as tracks and turnouts, but also a wide range of ancillary facilities and environments along both sides of the line. Specifically, the railway scenario can include roadbeds, bridges, tunnels, overhead contact lines and their supports, power supply equipment such as substations, as well as slopes, protective forests, and safety protection zones around the line that may pose risks of illegal construction or foreign object encroachment. A common characteristic of railway scenarios is their complex structure, often accompanied by strong electromagnetic interference or GPS signal obstruction.

[0032] Among them, multi-source perception data is a collection of data that reflects multiple dimensions of information about the surrounding environment, collected synchronously by different types of sensors carried by the UAV during the inspection flight.

[0033] The multi-source sensing data includes at least visual image sequences, laser point cloud sequences, and inertial measurement unit (IMU) data. The visual image sequences are a series of image frames captured continuously at a preset frame rate by a visible light camera mounted on a drone, such as a high-resolution full-frame camera. These visual image sequences are primarily used to reflect the texture, color, and semantic information of the railway scene.

[0034] A laser point cloud sequence is a series of 3D point cloud data frames obtained by continuous scanning of the environment using a lidar system mounted on a drone. Each frame contains the 3D coordinate information of a large number of points. The laser point cloud sequence is mainly used to accurately reflect the geometric structure and spatial contour information of a railway scene.

[0035] Here, the inertial measurement unit (IMU) data refers to the carrier's angular velocity and linear acceleration data collected at high frequencies, such as 200Hz, by the IMU mounted on the UAV. IMU data is primarily used to reflect the UAV's own motion state changes over a short period of time.

[0036] In one specific implementation, a hardware-time-synchronized trigger control system can be used to send a unified trigger signal to the visible light camera, lidar, and inertial measurement unit, ensuring that all sensors begin acquiring data at the same time and that the timestamps of each data stream are aligned at the microsecond level. Simultaneously, during the sensor integration phase, hand-eye calibration and extrinsic parameter calibration processes are required to accurately determine the relative positions and attitude relationships between the coordinate systems of each sensor, thereby ensuring that the acquired multi-source sensing data are also aligned in spatial attitude. Furthermore, after acquiring the raw multi-source sensing data, data standardization and correction processing can be performed, such as lens distortion correction for visual images and noise removal for lidar point clouds, to improve data quality. This embodiment of the invention does not specifically limit the specific implementation of these methods.

[0037] Step 120: Extract visual features from the visual image sequence, and determine visual reprojection constraints based on the reprojection residuals of visual features between different image frames in the visual image sequence.

[0038] Specifically, visual features are extracted from the visual image sequence, and visual reprojection constraints are determined based on the reprojection residuals of visual features between different image frames in the visual image sequence.

[0039] Here, visual features are used to reflect the saliency information of local regions in a visual image. Visual features can be points, lines, or regions in an image that possess uniqueness and stability. For example, visual features can be corner points, spots, etc., and they can still be stably detected and matched under different lighting, scale, and rotation changes. Various algorithms can be used to extract visual features, such as the Features from Accelerated Segment Test (FAST) algorithm for fast corner point detection, or Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and ORB (Oriented FAST and Rotated BRIEF) algorithms for extracting feature points with descriptors.

[0040] Among them, the reprojection residual is used to measure the consistency of the projection relationship of a 3D spatial point between different image frames. Specifically, by calculating the reprojection residual between the predicted image plane coordinates and the actual observed image plane coordinates of the same spatial point on the image plane of adjacent frames, nonlinear constraints can be formed on the 3D structure of the spatial point and the pose of the UAV at adjacent time moments, namely, visual reprojection constraints.

[0041] Here, the goal of the visual reprojection constraint is to adjust the pose estimation of the UAV so as to minimize the sum of the reprojection residuals of all commonly observed visual features.

[0042] Step 130: Extract geometric features from the laser point cloud sequence, and determine the laser odometry constraint based on the registration residual of the geometric features between consecutive point cloud frames in the laser point cloud sequence.

[0043] Specifically, geometric features are extracted from the laser point cloud sequence, and laser odometry constraints are determined based on the registration residuals of geometric features between consecutive point cloud frames in the laser point cloud sequence. The purpose of this step is to extract the geometric structure information of the environment from the lidar point cloud data and use the geometric structure information to estimate the relative motion of the UAV over a short period of time.

[0044] Here, geometric features are used to reflect the local geometric shape characteristics in the three-dimensional space described by the laser point cloud. In structured scenarios such as railways, geometric features can be obvious linear or planar structures presented in the point cloud. Obvious linear structures can be the edges of railway tracks or overhead contact lines, while planar structures can be the ground, tunnel walls, bridge pier sides, etc. Geometric features in the laser point cloud sequence can be extracted using methods such as point cloud segmentation or curvature analysis.

[0045] The registration residual is used to measure the alignment between two consecutive point cloud frames. When the UAV moves, the geometric features contained in the current point cloud frame can be mapped to the coordinate system of the reference point cloud frame through rigid body transformation. The registration residual is defined as the sum of spatial deviations between similar geometric features after rigid body transformation alignment. Typical measures include point-to-line distance and point-to-surface distance.

[0046] The core of establishing laser odometry constraints lies in minimizing the registration residual to calculate the relative pose transformation of the UAV at adjacent time points. Based on iterative optimization strategies, especially iterative nearest point and its derivative algorithms, the optimal rotation matrix and translation vector that minimize the registration residual between two consecutive frame point clouds can be solved. The relative pose transformation obtained from this solution constitutes the laser odometry constraint, applying a nonlinear optimization boundary to the relative motion of the UAV within the two-frame data acquisition interval.

[0047] Step 140: Perform pre-integration processing on the inertial measurement unit data between the acquisition times of two key frames in the visual image sequence to obtain the pre-integration processing result, and determine the inertial pre-integration constraint based on the pre-integration processing result.

[0048] Specifically, the inertial measurement unit (IMU) data between the acquisition times of two keyframes in the visual image sequence is pre-integrated to obtain the pre-integration result, and inertial pre-integration constraints are determined based on the pre-integration result. The purpose of this step is to provide high-precision relative motion estimation using high-frequency IMU data and to serve as a direct constraint connecting the poses of the two keyframes.

[0049] Keyframes are representative image frames selected from a sequence of visual images. Not all image frames need to participate in the subsequent global optimization to reduce computational load. There are various keyframe selection strategies. For example, when the translation or rotation of the UAV exceeds a preset threshold, or when the number of visual features tracked jointly between the current frame and the previous keyframe is less than a preset threshold, the current frame is selected as the new keyframe.

[0050] Traditional IMU integration methods integrate IMU measurements in the world coordinate system, which requires re-integrating the IMU data every time the keyframe pose estimation is updated, resulting in extremely high computational costs. In this embodiment, the pre-integration process integrates the IMU measurements between the two keyframes in their respective local coordinate systems. The resulting relative motion depends only on the states of the two keyframes themselves, and is independent of their poses in the world coordinate system. When the pose estimation of a keyframe changes in subsequent optimization, there is no need to re-integrate the IMU data; only a first-order Taylor expansion correction is needed based on the minute changes in pose, significantly improving computational efficiency.

[0051] Here, the pre-integration result refers to the measurement quantity obtained through pre-integration processing that describes the change in relative motion state between two keyframes. Specifically, the pre-integration result typically includes estimates of relative rotation, relative velocity, and relative displacement from the previous keyframe to the next keyframe, as well as the uncertainty of the estimates of relative rotation, relative velocity, and relative displacement, i.e., the covariance matrix.

[0052] Among them, the inertial pre-integration constraint is a nonlinear optimization constraint term constructed from the pre-integration processing results, which is used to associate the state variables of adjacent keyframes, including pose, velocity and inertial measurement unit bias.

[0053] Step 150: Using the six-degree-of-freedom pose of the UAV at multiple keyframes as the pose nodes of the factor graph, and using the visual reprojection constraint, the laser odometry constraint, and the inertial pre-integration constraint as the edges of the factor graph, construct the target factor graph.

[0054] Specifically, the six-degree-of-freedom poses of the UAV at multiple keyframes are used as pose nodes of the factor graph, and the visual reprojection constraints, laser odometry constraints, and inertial pre-integration constraints are used as edges of the factor graph to construct the target factor graph.

[0055] The six-degree-of-freedom (DOF) pose is used to fully describe the position and attitude of the UAV in three-dimensional space. The six-DOF pose can include three dimensions of translation information and three dimensions of rotation information. The three dimensions of translation information can be the x, y, and z coordinates in the world coordinate system, and the three dimensions of rotation information can be the rotation angles about the x, y, and z axes, namely roll angle, pitch angle, and yaw angle.

[0056] Here, the factor graph is a bipartite graph used to represent the factor decomposition of the probabilistic model. The factor graph contains variable nodes and factor nodes (constraint nodes). Variable nodes represent the unknown state quantities to be estimated, including at least the six-DOF pose of the UAV at each keyframe, and can be extended to velocity and inertial measurement unit bias. Factor nodes represent the constraint relationships introduced by the measurement data, namely visual reprojection constraints, laser odometry constraints, and inertial pre-integration constraints. Each factor node connects to the variable nodes it involves to apply the corresponding nonlinear optimization constraints.

[0057] In this context, the pose node is the variable node in the factor graph. Specifically, the six-DOF pose of the UAV at each keyframe is modeled as a pose node. In addition to pose, the variable node can also contain other state variables to be optimized, such as velocity and IMU bias.

[0058] The pose nodes model the six-DOF pose of the UAV at each keyframe as variable nodes in the factor graph. Visual reprojection constraints, also as factor nodes, associate multiple pose nodes observing the same visual feature with the corresponding 3D spatial point variable nodes, thus establishing multi-view geometric constraints. Laser odometry constraints are also modeled as factor nodes, connecting pose nodes in adjacent time sequences and imposing rigid constraints on the relative pose transformation between them. Inertial pre-integration constraints also serve as factor nodes, associating pose nodes and state variables such as velocity and inertial measurement unit bias in consecutive keyframes to characterize the relative motion relationship between adjacent time steps. The target factor graph is composed of all the above pose nodes and constraint factor nodes of various types, comprehensively describing the nonlinear least-squares optimization framework and probabilistic graphical model structure of the UAV localization problem.

[0059] Step 160: Perform nonlinear optimization on the target factor map to obtain the pose information of the UAV.

[0060] Specifically, after constructing the target factor map, nonlinear optimization can be performed on the target factor map to obtain the pose information of the UAV.

[0061] Given that both the visual reprojection model and the inertial measurement unit (IMU) kinematic model exhibit nonlinear characteristics, the optimization problem corresponding to the target factor map is essentially a nonlinear least squares estimation problem. The objective of solving this nonlinear least squares estimation problem is to determine an optimal set of UAV state variables, including the six-DOF pose, velocity, and IMU bias for each keyframe, such that the weighted sum of squared residuals of all constraint factors reaches a minimum. The weighting coefficients are determined by the inverse of the covariance matrix corresponding to each constraint, and are used to characterize the statistical properties of the measurement noise.

[0062] The pose node estimates output after nonlinear optimization convergence are the pose information of the UAV. This pose information sequence maintains geometric consistency at a global scale, constituting a high-precision motion trajectory of the UAV during the inspection mission.

[0063] The method provided in this invention acquires three types of multi-source sensing data, each with different physical principles and applicable scenarios: visual image sequences, laser point cloud sequences, and inertial measurement unit (IMU) data. It then extracts independent visual reprojection constraints, laser odometer constraints, and inertial pre-integration constraints from these multi-source sensing data. Furthermore, by constructing a target factor graph, it tightly couples and fuses these constraints with global nonlinear optimization. Thus, even in railway scenarios where GPS signals are attenuated or interrupted due to strong electromagnetic interference or physical obstruction, continuous and accurate pose calculation can be performed through the constraints of fused multi-source sensing information. This solves the problem of poor positioning reliability and easy loss of control in traditional UAV inspection schemes that rely on a single GPS signal, ensuring the navigation stability and positioning accuracy of UAVs in complex railway environments.

[0064] Before execution, the UAV inspection method for railway scenarios of the present invention can preprocess the acquired raw multi-source sensing data to eliminate non-content interference introduced during flight and data collection, providing high-quality data support for subsequent navigation, positioning, and feature extraction. In a specific implementation, this preprocessing may include lens distortion correction, noise removal and normalization, and multi-source data synchronization.

[0065] Lens distortion correction can be achieved by using Zhang Zhengyou's calibration method to calibrate the internal and external parameters of the visible light camera on the UAV. Furthermore, a distortion correction formula can be used to eliminate radial and tangential distortions in the visual image sequence, ensuring the straightness and perspective accuracy of the acquired images. The correction formula is as follows: ; in, These are the pixel coordinates of the distorted image. These are the corrected pixel coordinates. The radial distortion coefficient is... The tangential distortion coefficient is... This represents the distance from a pixel to the center of the image.

[0066] Noise removal and normalization can be performed using methods such as 3×3 median filtering to eliminate salt-and-pepper noise in visual image sequences, and methods such as pass-through filtering to remove ground noise and speckles in laser point cloud sequences. Subsequently, the processed data values ​​can be normalized, for example, to a floating-point range of 0-1, to unify the input value distribution of subsequent processing models. The normalization formula is as follows: ; in, These are the normalized data values. Represents the original data value. , These are the minimum and maximum values ​​of the original data, respectively.

[0067] To ensure the correspondence between different types of data at the same timestamp and spatial location, a combination of hardware-triggered synchronization and software compensation can be used. For example, at the hardware level, a synchronization trigger signal controls all sensors to collect data simultaneously, while at the software level, a timestamp alignment algorithm compensates for minor time deviations that occur during the acquisition process. Combined with pre-calibrated spatial parameters, this achieves precise spatiotemporal alignment of multi-source data.

[0068] Based on the above embodiments, step 120, which involves extracting visual features from the visual image sequence, includes: Step 121: Extract descriptor features from the visual image sequence as the visual features.

[0069] Specifically, descriptor features can be extracted from visual image sequences as visual features. These descriptor features reflect image information within the neighborhood of a visual feature point. Descriptor features possess excellent properties such as rotation invariance, scale invariance, and illumination invariance, enabling them to uniquely identify visual feature points in a compact and discriminative manner. This significantly improves the accuracy and robustness of feature point matching between different image frames.

[0070] In one specific implementation, ORB (Oriented Fast and Rotated BRIEF) features can be extracted as descriptor features, which are fast to calculate and have stable performance.

[0071] Based on the above embodiments, the step 130 of extracting geometric features from the laser point cloud sequence includes: Step 131: Extract line and surface features from the laser point cloud sequence as the geometric features.

[0072] Specifically, line and surface features can be extracted as geometric features from laser point cloud sequences. These features reflect the characteristics of local geometric structures within the point cloud data. Compared to the original 3D points, line and surface features are geometric primitives obtained after structured analysis of the point cloud. Line and surface features include line features and surface features. Line features reflect sets of points in the point cloud that exhibit a clear linear distribution, such as the edges of railway tracks or overhead contact lines in a railway scene. Surface features reflect sets of points in the point cloud that exhibit a clear planar distribution, such as the ground, tunnel walls, and building facades.

[0073] In one specific implementation, the local curvature of each point in the point cloud can be calculated. Points with larger curvature and distributed on the edges or corners are classified as line feature points, while points with smaller curvature and distributed in flat areas are classified as surface feature points, thereby extracting line and surface features.

[0074] The method provided in this invention enables line and surface features to more accurately reflect the geometric contours of the environment in a scenario rich in structured elements, such as railways, thereby making laser odometer constraints more precise and effectively suppressing the cumulative drift of UAV pose estimation.

[0075] Based on the above embodiments, step 160 includes: Step 160-1: Determine the target cost function of the target factor map; the target cost function is determined based on the first weight of the visual reprojection constraint residual, the second weight of the laser odometry constraint residual, and the third weight of the inertial pre-integration constraint residual, as well as the visual reprojection constraint residual, the laser odometry constraint residual, and the inertial pre-integration constraint residual; the first weight is determined based on the covariance matrix of the visual reprojection constraint; the second weight is determined based on the covariance matrix of the laser odometry constraint; the third weight is determined based on the covariance matrix of the inertial pre-integration constraint. The visual reprojection constraint residual is used to characterize the error of the visual reprojection constraint. The visual reprojection constraint residual is determined based on the difference between the observed value and the predicted value of the feature points extracted from the visual image sequence. The predicted value of the feature is determined based on the pose node and the camera projection model. The laser odometry constraint residual is used to characterize the error of the laser odometry constraint. The laser odometry constraint residual is the registration residual between the geometric features of the current frame and the geometric features of the previous frame in the laser point cloud sequence. The inertial pre-integration constraint residual is used to characterize the error of the inertial pre-integration constraint. The inertial pre-integration constraint residual is determined based on the relative motion and the inertial pre-integration quantity. The relative motion is determined based on two adjacent pose nodes in the target factor graph, and the inertial pre-integration quantity is determined based on the inertial pre-integration constraint. Step 160-2: Minimize the objective cost function to obtain the pose information of the UAV.

[0076] Specifically, in a preferred embodiment, a sliding window factor graph can be constructed to jointly optimize the inertial pre-integration constraint, the visual reprojection constraint, and the laser odometry constraint.

[0077] First, the target cost function of the target factor map can be determined. This target cost function is based on a first weight of the visual reprojection constraint residual, a second weight of the laser odometry constraint residual, and a third weight of the inertial pre-integration constraint residual, as well as the visual reprojection constraint residual, laser odometry constraint residual, and inertial pre-integration constraint residual. Here, the first weight is determined based on the covariance matrix of the visual reprojection constraint; the second weight is determined based on the covariance matrix of the laser odometry constraint; and the third weight is determined based on the covariance matrix of the inertial pre-integration constraint.

[0078] Here, the objective cost function is used to globally measure the degree of agreement between the current UAV pose estimate and the measurement data from all sensors. The mathematical expression for the objective cost function can be: ; in, This represents the set of state variables to be optimized, the core of which is the six-degree-of-freedom pose parameters of the UAV in each keyframe. Represent the objective cost function; , and They represent the first The inertial pre-integral constraint residual, the first The visual reprojection constraint residual and the first A vector of laser odometry constraint residuals; Let represent the covariance matrices corresponding to the inertial pre-integration constraint residuals, visual reprojection constraint residuals, and laser odometry constraint residuals, respectively. The inverse matrix is... Indicates the third weight. Indicates the first weight. denoted as the second weight; n, m, and l represent the number of inertial pre-integration constraints, visual reprojection constraints, and laser odometry constraints within an optimization window, respectively.

[0079] It should be noted that the covariance matrix of the visual reprojection constraint is used to reflect the uncertainty information in the visual feature point extraction and matching process. The covariance matrix of the laser odometry constraint is used to reflect the uncertainty information in the laser point cloud registration process. The covariance matrix of the inertial pre-integration constraint is used to reflect the uncertainty information caused by the accumulation of IMU sensor noise and bias during integration.

[0080] The visual reprojection constraint residual is used to characterize the error of the visual reprojection constraint. This residual is determined based on the difference between the observed value of the feature point extracted from the visual image sequence (the actual pixel coordinates of the feature point in the visual image sequence) and the predicted feature value. Here, the predicted feature value is determined based on the pose node and the camera projection model. The process of obtaining the predicted feature value is as follows: For a 3D spatial point variable node associated with the visual reprojection constraint, its estimated value under the current nonlinear optimization iteration is taken as the 3D coordinate of the spatial point in the world coordinate system; the pose node of the corresponding keyframe connected by the visual reprojection constraint is taken to obtain the camera pose of the UAV at that keyframe, i.e., the rotation matrix and translation vector; based on the camera projection model, the 3D coordinates in the world coordinate system are transformed to the camera coordinate system through the camera pose, and then perspective projection is performed according to the pre-calibrated camera intrinsic parameter matrix and distortion model to obtain the 2D projection coordinates of the 3D spatial point on the image plane of the corresponding image frame. These 2D projection coordinates are the predicted feature values.

[0081] Here, the laser odometry constraint residual is used to characterize the error of the laser odometry constraint. The laser odometry constraint residual is the registration residual between the geometric features of the current frame and the geometric features of the previous frame in the laser point cloud sequence.

[0082] The process of obtaining the registration residual is as follows: For two adjacent frames of point clouds in a laser point cloud sequence, the geometric features extracted from the previous frame are used as reference geometric features, and the similar geometric features extracted from the current frame are used as the geometric features to be registered. Based on the adjacent pose nodes connected by laser odometry constraints, the pose node of the previous keyframe is taken as the reference coordinate system, and the relative pose transformation of the current keyframe pose node is taken as the relative pose transformation to be optimized. The geometric features to be registered are mapped to the reference frame coordinate system through the relative pose transformation to be optimized, and the minimum target distance between the mapped geometric features and the reference geometric features is calculated as the registration residual. When the geometric feature is a linear structure, the target distance is taken as the distance from the spatial point to the fitted line; when the geometric feature is a planar structure, the target distance is taken as the distance from the spatial point to the fitted plane. In practical implementation, local neighborhood covariance analysis is typically used to extract and fit geometric primitives from the reference frame point cloud. For linear structures, a set of neighboring points around a feature point is selected, and the unit eigenvector corresponding to the smallest eigenvalue of its covariance matrix is ​​used as the direction of the line, with the centroid of the neighboring points as the base point of the line. For planar structures, a neighborhood where the smallest eigenvalue is much smaller than the other two eigenvalues ​​is selected, and the unit eigenvector corresponding to the smallest eigenvalue is used as the plane normal, with the centroid as a point on the plane to construct the plane equation. During residual construction, each feature point to be registered is associated only with the unique geometric primitive matched in the reference frame. This one-to-one tight coupling strategy effectively avoids robustness problems caused by incorrect matching. Finally, the target distances of all corresponding features are squared and summed, and weighting coefficients are introduced according to feature confidence or point cloud density to form the final laser odometry constrained registration residual, which is used to drive the backend nonlinear optimization to solve for the optimal pose transformation.

[0083] Here, the inertial pre-integration constraint residual is used to characterize the error of the inertial pre-integration constraint. The inertial pre-integration constraint residual is determined based on the relative motion and the inertial pre-integration quantity. The relative motion is determined based on two adjacent pose nodes in the target factor graph, and the inertial pre-integration quantity is determined based on the inertial pre-integration constraint.

[0084] The specific process for obtaining relative motion volume is as follows: For two adjacent keyframe pose nodes connected by the inertial pre-integration constraint, read the rotation matrix and translation vector corresponding to the previous keyframe pose node, and the rotation matrix and translation vector corresponding to the current keyframe pose node; at the same time, read the velocity variables associated with the above pose nodes and the estimated values ​​of gyroscope bias and accelerometer bias under the current nonlinear optimization iteration.

[0085] Based on the kinematic relationship of the inertial measurement unit, the rotation matrix of the pose node of the previous keyframe is used to perform coordinate transformation and gravity compensation on the rotation, velocity and displacement changes of the pose node of the current keyframe relative to the pose node of the previous keyframe: the rotation difference between the two frames is converted into a relative rotation increment; the velocity difference between the two frames is deducted from the influence of gravity during the sampling time and then projected onto the coordinate system of the previous keyframe to obtain the relative velocity increment; the displacement difference between the two frames is deducted from the translational term caused by the velocity of the previous keyframe and the second-order translational term caused by gravity and then projected onto the coordinate system of the previous keyframe to obtain the relative displacement increment.

[0086] The relative rotation increment, relative velocity increment, and relative displacement increment, which are constructed from adjacent pose nodes and their associated velocity variables, together constitute the relative motion quantity.

[0087] The specific process for obtaining the inertial pre-integral quantity is as follows: The raw observation data (angular velocity and linear acceleration) of the inertial measurement unit between two adjacent keyframe pose nodes associated with the inertial pre-integration constraint are integrated in the carrier coordinate system corresponding to the pose node of the previous keyframe to obtain the pre-integrated quantities describing the change in relative motion state between the two keyframes. These include at least pre-integrated relative rotation, pre-integrated relative velocity, and pre-integrated relative displacement. The covariance matrix is ​​propagated synchronously during the integration process to characterize the measurement uncertainty. This set of pre-integrated relative rotation, pre-integrated relative velocity, and pre-integrated relative displacement constitutes the inertial pre-integrated quantities.

[0088] The inertial pre-integration constraint residual is the relative motion quantity obtained by back-calculation from the above adjacent pose nodes and their associated velocity variables, and the deviation between it and the aforementioned inertial pre-integration quantity. It usually includes three components: rotation residual, velocity residual, and displacement residual.

[0089] After determining the objective cost function, the pose information of the UAV can be obtained by minimizing the cost function. Iterative optimization algorithms, such as the Gauss-Newton method or the Levenberg-Marquardt method, are used to minimize the objective cost function. When the objective function converges to its minimum value, the corresponding value of the state variable x represents the optimal UAV pose information.

[0090] The method provided in this invention explicitly constructs the multi-source fusion localization problem as a weighted nonlinear least squares optimization problem, which can fuse the constraints of various sensors in a statistically optimal manner. This method uses the covariance matrix to scientifically quantify and process the measurement noise and uncertainty of different sensors, so that constraints with low noise and high accuracy dominate the optimization, thereby maximizing the advantages of each sensor, suppressing noise interference, and obtaining more accurate and robust UAV pose information than any single sensor solution.

[0091] Based on the above embodiments, step 160 further includes the following prior steps: Step 21: Obtain the signal strength of the signal in the Global Positioning System; Step 22: When the signal strength is greater than or equal to a preset strength threshold, obtain the location observation value in the global positioning system; Step 23: Determine the pose node to be associated corresponding to the keyframe in the target factor graph that matches the timestamp of the position observation; Step 24: Extract the position components from the pose nodes to be associated, and determine the new edges of the pose nodes to be associated based on the observation residuals between the position observations and the position components, so as to constrain the pose nodes to be associated.

[0092] Specifically, firstly, the signal strength of the GPS signal can be obtained. Signal strength is a physical quantity that reflects the quality of the GPS satellite signal received by the UAV. Signal strength can include signal-to-noise ratio or carrier-to-noise ratio, and the unit is usually decibel (dB) or decibel milliwatt (dBm).

[0093] Then, when the signal strength is greater than or equal to a preset strength threshold, the location observation value in the Global Positioning System (GPS) is acquired. The preset strength threshold is a pre-defined critical value used to determine the reliability of the GPS signal. For example, the preset strength threshold can be set to 30 dBm. When the signal strength reaches or exceeds the preset strength threshold, the current GPS positioning result is considered reliable.

[0094] Here, the position observation value is the three-dimensional position coordinate of the UAV in the global coordinate system calculated by the GPS receiver when the GPS signal is reliable.

[0095] Furthermore, the pose nodes to be associated corresponding to the keyframes in the target factor map that match the timestamps of the location observations can be identified. These pose nodes are those in the constructed target factor map whose corresponding keyframe timestamps are closest to the timestamps of the acquired GPS location observations.

[0096] Finally, the position components in the pose nodes to be associated can be extracted, and the new edges of the pose nodes to be associated can be determined based on the observation residuals between the position observations and the position components, so as to constrain the pose nodes to be associated.

[0097] The position component refers to the three-dimensional translation component in the six-degree-of-freedom pose, namely the (x, y, z) coordinates.

[0098] Here, adding an edge involves adding a new factor node to the target factor graph. This factor node connects only to the pose nodes to be associated, and its mathematical model is a univariate constraint. The residual of the univariate constraint is defined as the difference between the position observation provided by GPS and the currently estimated position component of the pose node.

[0099] After adding new edges, i.e., GPS global constraints, the updated target factor map is then subjected to global nonlinear optimization. When the GPS signal is valid, such as when the signal strength is ≥30dBm, the GPS observations are added to the target factor map as global constraints to further improve positioning accuracy. When the GPS signal is lost, such as when the signal strength is <30dBm, the system automatically switches to a pure vision-laser-IMU fusion mode, continuously optimizing the target factor map to output the UAV's six-degree-of-freedom pose, achieving centimeter-level continuous positioning.

[0100] Based on the above embodiments, step 160 further includes: Step 1601: Generate a baseline route based on railway line design data; Step 1602: Based on the pose information of the UAV, control the UAV to fly along the reference flight path, and perform dynamic obstacle perception based on the multi-source perception data during the flight to obtain the obstacle perception result. Step 1603: When the obstacle perception result indicates the presence of a dynamic obstacle, adaptive obstacle avoidance control is performed based on the pose information of the UAV and the obstacle perception result to obtain a detour route, and the UAV is controlled to fly along the detour route.

[0101] Specifically, Figure 2 This is a schematic diagram of the process for controlling a drone to fly along a bypass route provided by the present invention, as shown below. Figure 2 As shown, firstly, a baseline flight path can be generated based on railway line design data. This baseline flight path is a pre-planned, unobstructed 3D flight path for UAV inspection missions. In one specific implementation, by importing computer-aided design (CAD) or Keyhole Markup Language (KML) data of the railway line, key facilities such as roadbed, bridges, tunnels, and overhead contact lines can be automatically identified. Different flight parameters can be set for different facilities; for example, the overhead contact line inspection flight path has a height of 10-15m and a flight speed of 3-5m / s; the roadbed inspection flight path has a height of 15-20m and a flight speed of 5-8m / s, etc., generating a baseline flight path with centimeter-level accuracy.

[0102] Then, based on the UAV's pose information, the system can control the UAV to fly along a reference flight path and perform dynamic obstacle perception based on multi-source sensing data during flight to obtain obstacle perception results. The flight control system can compare the UAV's pose information with the pose of the target point on the reference flight path in real time and generate control commands to reduce deviations. At the same time, the UAV uses its onboard sensors, such as LiDAR and cameras, to continuously perceive the surrounding environment to detect potential dynamic obstacles.

[0103] Finally, when the obstacle perception result indicates the presence of a dynamic obstacle, adaptive obstacle avoidance control is performed based on the UAV's pose information and the obstacle perception result to obtain a detour route, and the UAV is controlled to fly along the detour route.

[0104] Here, the detour route is a dynamically generated, safe local flight path designed to cope with sudden obstacles. In one specific implementation, the detour route can be planned using an improved artificial potential field method based on obstacle information and the UAV's pose information, ensuring that the safe distance between the detour route and the obstacle is no less than 5 meters. After the detour is completed, the route automatically returns to the baseline route. If the obstacle is detected to be too close, for example, less than 10 meters, emergency deceleration or hovering is automatically triggered to ensure flight safety.

[0105] Furthermore, all flight control and planning algorithms can undergo extensive stress testing in highly realistic railway simulation environments, such as Unity, to verify their safety boundaries and reliability.

[0106] The method provided in this invention addresses the pain point of traditional drone inspections, which rely on manual remote control and cannot cope with emergencies, by combining pre-generated baseline routes with real-time dynamic obstacle avoidance.

[0107] Based on the above embodiments, step 1602, which involves performing dynamic obstacle perception based on the multi-source sensing data during flight to obtain obstacle perception results, includes: During flight, the multi-source sensing data is input into a multi-scale target detection model to obtain the obstacle sensing results output by the multi-scale target detection model. The multi-scale target detection model includes a feature extraction backbone network, a feature pyramid, an attention module, and a classification layer; The feature extraction backbone network is used to extract texture semantic features from the visual image sequence and geometric features from the laser point cloud sequence; The feature pyramid is used to extract the texture semantic features at multiple scales to obtain multi-scale texture semantic features, and to fuse the multi-scale texture semantic features across scales to obtain multi-scale visual features. The attention module is used to fuse the multi-scale visual features and the geometric features to obtain fused features; The classification layer is used to generate the obstacle perception result based on the fusion features.

[0108] Specifically, during flight, multi-source perception data is input into a multi-scale target detection model to obtain the obstacle perception results output by the multi-scale target detection model.

[0109] The multi-scale object detection model includes a feature extraction backbone network, a feature pyramid, an attention module, and a classification layer. The feature extraction backbone network is used to extract texture semantic features from visual image sequences and geometric features from laser point cloud sequences.

[0110] Here, texture semantic features are used to reflect the object category and semantic attribute information carried by pixel grayscale, color and texture pattern in a visual image sequence.

[0111] Here, the feature pyramid is used to extract texture semantic features at multiple scales, resulting in multi-scale texture semantic features. These multi-scale texture semantic features are then fused across scales to obtain multi-scale visual features. The multi-scale visual features are used to represent the structure, contours, and semantic context information of objects at different scales in a visual image sequence.

[0112] Here, the attention module is used to fuse multi-scale visual features and geometric features to obtain fused features. For example, the Convolutional Block Attention Module (CBAM) can be used to enhance the channel and spatial attention weights of multi-scale features, and a feature-level weighted fusion strategy can be used to perform channel-wise weighted fusion of the geometric features extracted from the laser point cloud and the texture semantic features of the visible light image.

[0113] The classification layer is used to generate obstacle perception results based on fused features. For example, it can identify dynamic obstacles such as temporary construction vehicles, birds, and cables, and output information such as their position, size, and speed.

[0114] Based on the above embodiments, the method further includes: Step 31: Identify risk targets based on the multi-source sensing data; the risk targets include indicators of impact range, probability of occurrence, degree of harm, and difficulty of handling. Step 32: Match the risk target with the pre-constructed railway operation and maintenance knowledge graph to obtain the safety rules corresponding to the risk target; Step 33: Based on the safety rules, determine the first quantitative score of the impact range index, the second quantitative score of the occurrence probability index, the third quantitative score of the hazard level index, and the fourth quantitative score of the handling difficulty index, respectively. Step 34: Based on the target weight, perform a weighted summation of the first quantitative score, the second quantitative score, the third quantitative score, and the fourth quantitative score to obtain the risk quantification value; Step 35: Determine the risk level based on the risk quantification value, and determine the treatment recommendations based on the risk level.

[0115] Specifically, firstly, risk targets can be identified based on multi-source sensing data. This step can employ a multimodal target detection model embedded with multi-scale attention, for example, using YOLOv8 as a baseline and embedding a CBAM module in the backbone network to enhance the feature extraction capability for small targets and camouflaged targets, such as foreign objects covered by vegetation. The multimodal target detection model is trained on a dedicated sample library of railway external environmental risks containing massive amounts of real samples to identify risks such as illegal construction, foreign object encroachment, vegetation encroachment, and slope collapse. During the model training and optimization phase, a dedicated sample library of railway external environmental risks covering nineteen types of risks is constructed, containing over 100,000 real samples. The sample collection covers various meteorological conditions such as sunny, rainy, and foggy days, as well as different time periods during the day and night, and covers typical railway scenarios such as bridges, tunnels, and roadbeds. During training, a transfer learning strategy is adopted to transfer the model parameters pre-trained on a general object detection dataset to this task. Combined with various data augmentation techniques such as random flipping, illumination transformation, and scale scaling, the generalization ability and robustness of the model under complex working conditions are effectively improved to meet the application requirements of real-time inspection by UAVs.

[0116] Here, risk targets can include indicators of impact scope, probability of occurrence, severity of harm, and difficulty of handling. These four indicators are abstractions and definitions of the risk attributes of identified physical targets from the perspective of railway operation and maintenance safety.

[0117] Furthermore, risk objectives can be matched with a pre-built railway operation and maintenance knowledge graph to obtain the corresponding safety rules. In a specific implementation, a railway operation and maintenance knowledge graph can be constructed based on the track patrol implementation of the engineering section and related railway maintenance specifications, design standards, equipment files, etc., containing nodes such as risk objectives, safety protection zone scope, risk level determination rules, and historical disaster records.

[0118] After obtaining the safety rules, the first quantitative score for the impact range indicator, the second quantitative score for the occurrence probability indicator, the third quantitative score for the hazard level indicator, and the fourth quantitative score for the handling difficulty indicator can be determined based on the safety rules.

[0119] Furthermore, based on the target weights, the first, second, third, and fourth quantitative scores can be weighted and summed to obtain the risk quantification value. This risk quantification process can employ the fuzzy comprehensive evaluation method, and its quantification formula can be expressed as: ; Where R is the risk quantification value, Represents the target weight, i.e., the first... The comprehensive weight of each risk assessment indicator, For the first The quantitative score of each evaluation indicator.

[0120] Finally, risk levels can be determined based on risk quantification values, and response recommendations can be made based on these risk levels. For example, risk levels can be divided into four categories based on the range of risk quantification values ​​R: general risk (R<25), significant risk (25≤R<50), major risk (50≤R<75), and extremely major risk (R≥75). Targeted response recommendations, such as regular monitoring or immediate work stoppage, can then be automatically generated based on the risk level and the response rules in the knowledge graph.

[0121] In addition, the system is based on the calculations obtained The system automatically classifies risks and directly triggers corresponding hardware control and alarm commands without human intervention. For example, general risks automatically trigger regular cloud monitoring commands; significant risks automatically trigger real-time edge tracking and audible and visual alarm commands; major risks automatically trigger drone hovering and push emergency alarm commands to the operation and maintenance center; and particularly serious risks automatically trigger emergency drone return and link to the railway dispatch system for early warning.

[0122] Based on the above embodiments, the target weights include the weights of the influence range index, the occurrence probability index, the severity index, and the handling difficulty index; The steps for determining the target weight include: Step 41, determine the static basic weights; the static basic weights include the first initial weight of the influence range index, the second initial weight of the occurrence probability index, the third initial weight of the hazard level index, and the fourth initial weight of the handling difficulty index; Step 42: When the risk target is detected to be within a preset distance range of the railway mainline, the third initial weight in the static basic weight is increased to obtain the target weight; Step 43: When the risk target is detected to be in motion, the second initial weight in the static basic weight is increased to obtain the target weight.

[0123] Specifically, the target weights may include the weights of the impact range indicator, the probability of occurrence indicator, the severity of harm indicator, and the difficulty of handling indicator.

[0124] Accordingly, the first step is to determine the static basic weights. These static basic weights are a set of initial weight values ​​pre-set based on expert experience and historical data, used to reflect the relative importance of each risk indicator under normal circumstances. Specifically, the static basic weights include the first initial weight for the scope of impact indicator, the second initial weight for the probability of occurrence indicator, the third initial weight for the severity of harm indicator, and the fourth initial weight for the difficulty of handling indicator.

[0125] In one specific implementation, the Analytic Hierarchy Process (AHP) can be used. This involves inviting multiple experts in railway operation and maintenance to construct a judgment matrix and calculating a set of static basic weights. For example, the initial weights for the scope of impact, probability of occurrence, severity of harm, and difficulty of handling are 0.20, 0.25, 0.35, and 0.20, respectively, which are the static basic weights. .

[0126] When a risk target is detected within a preset distance from the railway mainline, the third initial weight in the static basic weighting is increased to obtain the target weight. This step reflects the risk weighting based on spatial proximity. For example, when an excavator is detected within 5 meters of the railway mainline, its potential hazard level increases sharply. Therefore, the system will automatically increase its corresponding third initial weight, for example, by 10%, and normalize the remaining weights proportionally.

[0127] When a risk target is detected to be in motion, the second initial weight in the static base weights is increased to obtain the target weight. This step reflects the weighting of dynamic risks. For example, when a boulder is detected to be in motion on a slope, the probability of its occurrence is much higher than in a static state. Therefore, the system will automatically increase the second initial weight corresponding to its probability of occurrence, for example, by 8%, and normalize the remaining weights proportionally.

[0128] The method provided by this invention can adaptively adjust the focus of risk assessment based on key contextual information such as the relative position of the risk target and the core railway facilities, and its own motion state. This makes the risk quantification results closer to the dynamic changes in the real world and the on-site judgment logic of experts, greatly improving the accuracy and timeliness of risk assessment.

[0129] Based on any of the above embodiments Figure 3 This is a block diagram of the overall architecture of the railway scene unmanned aerial vehicle autonomous inspection system provided by the present invention, as shown below. Figure 3As shown, the system's workflow begins with the UAV perception and data acquisition module, which is responsible for acquiring multi-source perception data synchronously collected by the UAV in the railway scenario. The acquired multi-source perception data is then transmitted to the multi-source tightly coupled fusion navigation module, which contains cooperative navigation and positioning sub-modules and path planning and obstacle avoidance sub-modules. These sub-modules are used to calculate high-precision UAV pose information and achieve autonomous flight control. Simultaneously, the multi-source perception data acquired by the UAV perception and data acquisition module, along with the pose information determined by the navigation and positioning sub-module, is input into the multimodal macroscopic risk identification system. The system is divided into several modules: a risk identification module for identifying various risk targets; and a risk target identification module for identifying risk targets for further transmission to the railway operation and maintenance knowledge graph and risk reasoning module. The railway operation and maintenance knowledge graph and risk reasoning module completes the quantitative assessment and reasoning of risks by matching with the knowledge graph. Subsequently, the risk reasoning results are sent to the cloud-edge collaboration and model iteration module for continuous model optimization. Finally, the processing results from the railway operation and maintenance knowledge graph and risk reasoning module and the cloud-edge collaboration and model iteration module are summarized to the result output and data storage module for generating standardized inspection reports and archiving and storing the data.

[0130] The cloud-edge collaboration and model iteration module aims to construct a closed-loop data system for UAV-based edge data collection, analysis, and cloud-based model iteration verification. This addresses the technical bottleneck of the existing system's inability to continuously evolve model performance, enabling the autonomous evolution of AI model capabilities and significantly reducing false positive and false negative rates. At the edge, the high-performance edge computing unit on the UAV executes navigation calculations and risk identification algorithms in real time. High-confidence alarms (with a confidence level of at least 90%) are directly pushed to the cloud control platform, while low-confidence alarms are sent back to the cloud for secondary analysis, effectively alleviating data transmission pressure while ensuring real-time performance. A human-machine collaborative review interface is deployed in the cloud, where railway maintenance experts review and confirm the validity of high-confidence and low-confidence alarms. All confirmed valid false negatives and false negatives are automatically marked as high-value, difficult samples and stored in a database. Furthermore, the system adopts an incremental learning mechanism, fine-tuning the artificial intelligence model weekly based on difficult samples in the training database. By introducing a learning rate decay strategy to suppress overfitting, the system achieves continuous improvement in model performance while controlling training costs and cycles. The fine-tuned optimized model is automatically distributed to the edge computing units of all drones in the network via a secure channel, completing clustered batch updates.

[0131] The results output and data storage module is responsible for presenting, archiving, and managing the system's final results. It provides users with legally valid inspection and evaluation reports and supports the review and traceability work of railway operations and maintenance, while also meeting the data confidentiality standards for infrastructure operations and maintenance. The visualization output of the results output and data storage module includes a comprehensive evaluation report covering inspection line attributes, time windows, UAV full trajectory, risk target geographic coordinates, risk category level classification, handling suggestions, on-site real-scene images, and feature heatmaps. The feature heatmap uses inspection images as a base, employing black-and-white grayscale gradients to represent the model's focus on potential risk areas; darker grayscale indicates higher focus. The entire process is without color rendering to meet confidentiality requirements. All inspection metadata, including route information, flight control logs, multi-source sensing data, feature extraction results, risk identification records, expert review logs, and model iteration trajectories, is structured and stored in a secure database. The system supports multi-dimensional combined queries, filtering, and report export based on route time, risk level, and type. A three-level encrypted access control system for administrators, operations and maintenance experts, and ordinary users ensures data access security. It also provides standardized application programming interfaces to achieve seamless integration with the railway operations and maintenance management system's emergency response platform and digital twin base, enabling multi-scenario business integration.

[0132] The following describes the unmanned aerial vehicle (UAV) inspection device for railway scenarios provided by the present invention. The UAV inspection device for railway scenarios described below and the UAV inspection method for railway scenarios described above can be referred to and correspond to each other.

[0133] Based on any of the above embodiments, the present invention provides a drone inspection device for railway scenarios. Figure 4 This is a structural schematic diagram of the unmanned aerial vehicle (UAV) inspection device for railway scenarios provided by the present invention, as shown below. Figure 4 As shown, the device includes: The acquisition unit 410 is used to acquire multi-source perception data synchronously collected by the UAV in the railway scene; the multi-source perception data includes at least visual image sequences, laser point cloud sequences and inertial measurement unit data; The first extraction unit 420 is used to extract visual features from the visual image sequence and determine visual reprojection constraints based on the reprojection residuals of visual features between different image frames in the visual image sequence. The second extraction unit 430 is used to extract geometric features from the laser point cloud sequence and determine the laser odometry constraint based on the registration residual of the geometric features between consecutive point cloud frames in the laser point cloud sequence. The pre-integration unit 440 is used to perform pre-integration processing on the inertial measurement unit data between the acquisition times of two key frames in the visual image sequence, to obtain the pre-integration processing result, and to determine the inertial pre-integration constraint based on the pre-integration processing result. The construction unit 450 is used to construct a target factor graph by using the six-degree-of-freedom pose of the UAV at multiple key frames as the pose nodes of the factor graph, and using the visual reprojection constraint, the laser odometry constraint and the inertial pre-integration constraint as the edges of the factor graph. The optimization unit 460 is used to perform nonlinear optimization on the target factor map to obtain the pose information of the UAV.

[0134] The device provided in this invention acquires three types of multi-source sensing data with different physical principles and applicable scenarios: visual image sequences, laser point cloud sequences, and inertial measurement unit data. It extracts independent visual reprojection constraints, laser odometer constraints, and inertial pre-integration constraints from the multi-source sensing data. Then, by constructing a target factor map, it tightly couples and fuses the visual reprojection constraints, laser odometer constraints, and inertial pre-integration constraints and performs global nonlinear optimization. Thus, even in railway scenarios where GPS signals are attenuated or even interrupted due to strong electromagnetic interference or physical obstruction, it can perform continuous and accurate pose calculation by fusing constraints from multi-source sensing information. This solves the problem of poor positioning reliability and easy loss of control caused by traditional UAV inspection schemes that rely on a single global positioning system signal, ensuring the navigation stability and positioning accuracy of UAVs in complex railway environments.

[0135] Based on any of the above embodiments, the first extraction unit 420 is specifically used for: Descriptor features are extracted from the visual image sequence as the visual features.

[0136] Based on any of the above embodiments, the second extraction unit 430 is specifically used for: Line and surface features are extracted from the laser point cloud sequence as the geometric features.

[0137] Based on any of the above embodiments, the optimization unit 460 is specifically used for: The target cost function of the target factor map is determined; the target cost function is determined based on a first weight of the visual reprojection constraint residual, a second weight of the laser odometry constraint residual, and a third weight of the inertial pre-integration constraint residual, as well as the visual reprojection constraint residual, the laser odometry constraint residual, and the inertial pre-integration constraint residual; the first weight is determined based on the covariance matrix of the visual reprojection constraint; the second weight is determined based on the covariance matrix of the laser odometry constraint; the third weight is determined based on the covariance matrix of the inertial pre-integration constraint. The visual reprojection constraint residual is used to characterize the error of the visual reprojection constraint. The visual reprojection constraint residual is determined based on the difference between the observed value and the predicted value of the feature points extracted from the visual image sequence. The predicted value of the feature is determined based on the pose node and the camera projection model. The laser odometry constraint residual is used to characterize the error of the laser odometry constraint. The laser odometry constraint residual is the registration residual between the geometric features of the current frame and the geometric features of the previous frame in the laser point cloud sequence. The inertial pre-integration constraint residual is used to characterize the error of the inertial pre-integration constraint. The inertial pre-integration constraint residual is determined based on the relative motion and the inertial pre-integration quantity. The relative motion is determined based on two adjacent pose nodes in the target factor graph, and the inertial pre-integration quantity is determined based on the inertial pre-integration constraint. The pose information of the UAV is obtained by minimizing the objective cost function.

[0138] Based on any of the above embodiments, a constraint unit is further included, wherein the constraint unit is specifically used for: Obtain the signal strength of signals in the Global Positioning System; When the signal strength is greater than or equal to a preset strength threshold, the position observation value in the global positioning system is obtained; Determine the pose nodes to be associated corresponding to the keyframes in the target factor graph that match the timestamps of the location observations; Extract the position components from the pose nodes to be associated, and determine new edges for the pose nodes to be associated based on the observation residuals between the position observations and the position components, so as to constrain the pose nodes to be associated.

[0139] Based on any of the above embodiments, a flight unit is further included, wherein the flight unit specifically includes: Generate route units, used to generate baseline routes based on railway line design data; The control unit is used to control the UAV to fly along the reference flight path based on the pose information of the UAV, and to perform dynamic obstacle perception based on the multi-source perception data during the flight to obtain obstacle perception results. The detour unit is used to perform adaptive obstacle avoidance control based on the pose information of the UAV and the obstacle perception result when the obstacle perception result indicates the existence of a dynamic obstacle, obtain a detour route, and control the UAV to fly along the detour route.

[0140] Based on any of the above embodiments, the control unit is specifically used for: During flight, the multi-source sensing data is input into a multi-scale target detection model to obtain the obstacle sensing results output by the multi-scale target detection model. The multi-scale target detection model includes a feature extraction backbone network, a feature pyramid, an attention module, and a classification layer; The feature extraction backbone network is used to extract texture semantic features from the visual image sequence and geometric features from the laser point cloud sequence; The feature pyramid is used to extract the texture semantic features at multiple scales to obtain multi-scale texture semantic features, and to fuse the multi-scale texture semantic features across scales to obtain multi-scale visual features. The attention module is used to fuse the multi-scale visual features and the geometric features to obtain fused features; The classification layer is used to generate the obstacle perception result based on the fusion features.

[0141] Based on any of the above embodiments, a processing unit is further included, wherein the processing unit is specifically used for: Risk targets are identified based on the multi-source sensing data; the risk targets include indicators of impact range, probability of occurrence, degree of harm, and difficulty of handling. The risk objectives are matched with a pre-constructed railway operation and maintenance knowledge graph to obtain the safety rules corresponding to the risk objectives; Based on the aforementioned safety rules, a first quantitative score for the impact range indicator, a second quantitative score for the occurrence probability indicator, a third quantitative score for the severity of harm indicator, and a fourth quantitative score for the handling difficulty indicator are determined respectively. Based on the target weight, the first quantitative score, the second quantitative score, the third quantitative score, and the fourth quantitative score are weighted and summed to obtain the risk quantitative value; The risk level is determined based on the risk quantification value, and the treatment recommendations are determined based on the risk level.

[0142] Based on any of the above embodiments, the target weight includes the weight of the influence range indicator, the weight of the occurrence probability indicator, the weight of the hazard level indicator, and the weight of the handling difficulty indicator; It also includes a target weight determination unit, which is specifically used for: Determine the static basic weights; the static basic weights include the first initial weight of the influence range indicator, the second initial weight of the occurrence probability indicator, the third initial weight of the hazard degree indicator, and the fourth initial weight of the handling difficulty indicator; When the risk target is detected to be within a preset distance range of the railway mainline, the third initial weight in the static basic weight is increased to obtain the target weight; When the risk target is detected to be in motion, the second initial weight in the static basic weight is increased to obtain the target weight.

[0143] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a UAV inspection method for a railway scenario. This method includes: acquiring multi-source perception data synchronously collected by the UAV in a railway scenario; the multi-source perception data includes at least a visual image sequence, a laser point cloud sequence, and inertial measurement unit data; extracting visual features from the visual image sequence and determining visual reprojection constraints based on the reprojection residuals of visual features between different image frames in the visual image sequence; extracting geometric features from the laser point cloud sequence and determining visual reprojection constraints based on the geometric features between consecutive point cloud frames in the laser point cloud sequence. The registration residuals of the features are used to determine the laser odometry constraints; the inertial measurement unit data between the acquisition times of two key frames in the visual image sequence are pre-integrated to obtain the pre-integration results, and the inertial pre-integration constraints are determined based on the pre-integration results; the six-degree-of-freedom poses of the UAV at multiple key frames are used as pose nodes of the factor graph, and the visual reprojection constraints, the laser odometry constraints, and the inertial pre-integration constraints are used as edges of the factor graph to construct a target factor graph; the target factor graph is nonlinearly optimized to obtain the pose information of the UAV.

[0144] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the UAV inspection method for railway scenarios provided by the above methods. The method includes: acquiring multi-source sensing data synchronously collected by the UAV in the railway scenario; the multi-source sensing data includes at least a visual image sequence, a laser point cloud sequence, and inertial measurement unit data; extracting visual features from the visual image sequence, and determining visual reprojection constraints based on the reprojection residuals of visual features between different image frames in the visual image sequence; and extracting visual features from the laser point cloud sequence. Geometric features are extracted from the cloud sequence, and laser odometry constraints are determined based on the registration residuals of geometric features between consecutive point cloud frames in the laser point cloud sequence. Pre-integration processing is performed on the inertial measurement unit data between the acquisition times of two key frames in the visual image sequence to obtain the pre-integration processing result, and inertial pre-integration constraints are determined based on the pre-integration processing result. Using the six-degree-of-freedom pose of the UAV at multiple key frames as pose nodes of the factor graph, the visual reprojection constraints, the laser odometry constraints, and the inertial pre-integration constraints are used as edges of the factor graph to construct a target factor graph. Nonlinear optimization is performed on the target factor graph to obtain the pose information of the UAV.

[0146] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a UAV inspection method for railway scenarios provided by the methods described above. This method includes: acquiring multi-source sensing data synchronously collected by the UAV in a railway scenario; the multi-source sensing data includes at least a visual image sequence, a laser point cloud sequence, and inertial measurement unit data; extracting visual features from the visual image sequence and determining visual reprojection constraints based on the reprojection residuals of visual features between different image frames in the visual image sequence; extracting geometric features from the laser point cloud sequence and based on... The registration residuals of geometric features between consecutive point cloud frames in the laser point cloud sequence are used to determine the laser odometry constraints. Pre-integration processing is performed on the inertial measurement unit data between the acquisition times of two key frames in the visual image sequence to obtain the pre-integration processing result. Inertial pre-integration constraints are determined based on the pre-integration processing result. Using the six-degree-of-freedom pose of the UAV at multiple key frames as pose nodes of the factor graph, the visual reprojection constraints, the laser odometry constraints, and the inertial pre-integration constraints are used as edges of the factor graph to construct a target factor graph. Nonlinear optimization is performed on the target factor graph to obtain the pose information of the UAV.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for unmanned aerial vehicle (UAV) inspection in a railway scenario, characterized in that, include: Acquire multi-source perception data synchronously collected by drones in railway scenarios; The multi-source sensing data includes at least visual image sequences, laser point cloud sequences, and inertial measurement unit data; Visual features are extracted from the visual image sequence, and visual reprojection constraints are determined based on the reprojection residuals of visual features between different image frames in the visual image sequence. Geometric features are extracted from the laser point cloud sequence, and laser odometry constraints are determined based on the registration residuals of geometric features between consecutive point cloud frames in the laser point cloud sequence. The inertial measurement unit data between the acquisition times of two key frames in the visual image sequence is pre-integrated to obtain the pre-integration processing result, and the inertial pre-integration constraint is determined based on the pre-integration processing result; Using the six-degree-of-freedom pose of the UAV at multiple keyframes as the pose nodes of the factor graph, and using the visual reprojection constraint, the laser odometry constraint, and the inertial pre-integration constraint as the edges of the factor graph, a target factor graph is constructed. The pose information of the UAV is obtained by performing nonlinear optimization on the target factor map.

2. The unmanned aerial vehicle (UAV) inspection method for railway scenarios according to claim 1, characterized in that, Extracting visual features from the visual image sequence includes: Descriptor features are extracted from the visual image sequence as the visual features.

3. The unmanned aerial vehicle (UAV) inspection method for railway scenarios according to claim 1, characterized in that, Extracting geometric features from the laser point cloud sequence includes: Line and surface features are extracted from the laser point cloud sequence as the geometric features.

4. The unmanned aerial vehicle (UAV) inspection method for railway scenarios according to any one of claims 1 to 3, characterized in that, The nonlinear optimization of the target factor map to obtain the pose information of the UAV includes: The target cost function of the target factor map is determined; the target cost function is determined based on a first weight of the visual reprojection constraint residual, a second weight of the laser odometry constraint residual, and a third weight of the inertial pre-integration constraint residual, as well as the visual reprojection constraint residual, the laser odometry constraint residual, and the inertial pre-integration constraint residual; the first weight is determined based on the covariance matrix of the visual reprojection constraint; the second weight is determined based on the covariance matrix of the laser odometry constraint; the third weight is determined based on the covariance matrix of the inertial pre-integration constraint. The visual reprojection constraint residual is used to characterize the error of the visual reprojection constraint. The visual reprojection constraint residual is determined based on the difference between the observed value and the predicted value of the feature points extracted from the visual image sequence. The predicted value of the feature is determined based on the pose node and the camera projection model. The laser odometry constraint residual is used to characterize the error of the laser odometry constraint. The laser odometry constraint residual is the registration residual between the geometric features of the current frame and the geometric features of the previous frame in the laser point cloud sequence. The inertial pre-integration constraint residual is used to characterize the error of the inertial pre-integration constraint. The inertial pre-integration constraint residual is determined based on the relative motion and the inertial pre-integration quantity. The relative motion is determined based on two adjacent pose nodes in the target factor graph, and the inertial pre-integration quantity is determined based on the inertial pre-integration constraint. The pose information of the UAV is obtained by minimizing the objective cost function.

5. The unmanned aerial vehicle (UAV) inspection method for railway scenarios according to any one of claims 1 to 3, characterized in that, The step of performing nonlinear optimization on the target factor map to obtain the pose information of the UAV also includes: Obtain the signal strength of signals in the Global Positioning System; When the signal strength is greater than or equal to a preset strength threshold, the position observation value in the global positioning system is obtained; Determine the pose nodes to be associated corresponding to the keyframes in the target factor graph that match the timestamps of the location observations; Extract the position components from the pose nodes to be associated, and determine new edges for the pose nodes to be associated based on the observation residuals between the position observations and the position components, so as to constrain the pose nodes to be associated.

6. The unmanned aerial vehicle (UAV) inspection method for railway scenarios according to any one of claims 1 to 3, characterized in that, The process of performing nonlinear optimization on the target factor map to obtain the pose information of the UAV further includes: Generate a baseline route based on railway line design data; Based on the pose information of the UAV, the UAV is controlled to fly along the reference route, and during the flight, dynamic obstacle perception is performed based on the multi-source perception data to obtain obstacle perception results. When the obstacle perception result indicates the presence of a dynamic obstacle, adaptive obstacle avoidance control is performed based on the UAV's pose information and the obstacle perception result to obtain a detour route, and the UAV is controlled to fly along the detour route.

7. The unmanned aerial vehicle (UAV) inspection method for railway scenarios according to claim 6, characterized in that, The process of dynamically sensing obstacles based on the multi-source sensing data during flight to obtain obstacle sensing results includes: During flight, the multi-source sensing data is input into a multi-scale target detection model to obtain the obstacle sensing results output by the multi-scale target detection model. The multi-scale target detection model includes a feature extraction backbone network, a feature pyramid, an attention module, and a classification layer; The feature extraction backbone network is used to extract texture semantic features from the visual image sequence and geometric features from the laser point cloud sequence; The feature pyramid is used to extract the texture semantic features at multiple scales to obtain multi-scale texture semantic features, and to fuse the multi-scale texture semantic features across scales to obtain multi-scale visual features. The attention module is used to fuse the multi-scale visual features and the geometric features to obtain fused features; The classification layer is used to generate the obstacle perception result based on the fusion features.

8. The unmanned aerial vehicle (UAV) inspection method for railway scenarios according to any one of claims 1 to 3, characterized in that, The method further includes: Risk targets are identified based on the multi-source sensing data; the risk targets include indicators of impact range, probability of occurrence, degree of harm, and difficulty of handling. The risk objectives are matched with a pre-constructed railway operation and maintenance knowledge graph to obtain the safety rules corresponding to the risk objectives; Based on the aforementioned safety rules, a first quantitative score for the impact range indicator, a second quantitative score for the occurrence probability indicator, a third quantitative score for the severity of harm indicator, and a fourth quantitative score for the handling difficulty indicator are determined respectively. Based on the target weight, the first quantitative score, the second quantitative score, the third quantitative score, and the fourth quantitative score are weighted and summed to obtain the risk quantitative value; The risk level is determined based on the risk quantification value, and the treatment recommendations are determined based on the risk level.

9. The unmanned aerial vehicle (UAV) inspection method for railway scenarios according to claim 8, characterized in that, The target weights include the weights of the impact range indicator, the occurrence probability indicator, the severity of harm indicator, and the handling difficulty indicator. The steps for determining the target weight include: Determine the static basic weights; the static basic weights include the first initial weight of the influence range indicator, the second initial weight of the occurrence probability indicator, the third initial weight of the hazard degree indicator, and the fourth initial weight of the handling difficulty indicator; When the risk target is detected to be within a preset distance range of the railway mainline, the third initial weight in the static basic weight is increased to obtain the target weight; When the risk target is detected to be in motion, the second initial weight in the static basic weight is increased to obtain the target weight.

10. A drone inspection device for railway scenarios, characterized in that, include: The acquisition unit is used to acquire multi-source perception data synchronously collected by the UAV in the railway scenario; The multi-source sensing data includes at least visual image sequences, laser point cloud sequences, and inertial measurement unit data; The first extraction unit is used to extract visual features from the visual image sequence and determine visual reprojection constraints based on the reprojection residuals of visual features between different image frames in the visual image sequence. The second extraction unit is used to extract geometric features from the laser point cloud sequence and determine the laser odometry constraint based on the registration residual of the geometric features between consecutive point cloud frames in the laser point cloud sequence. The pre-integration unit is used to perform pre-integration processing on the inertial measurement unit data between the acquisition times of two key frames in the visual image sequence, to obtain the pre-integration processing result, and to determine the inertial pre-integration constraint based on the pre-integration processing result. The construction unit is used to construct a target factor graph by using the six-degree-of-freedom pose of the UAV at multiple key frames as the pose nodes of the factor graph, and using the visual reprojection constraint, the laser odometry constraint and the inertial pre-integration constraint as the edges of the factor graph. An optimization unit is used to perform nonlinear optimization on the target factor map to obtain the pose information of the UAV.

Images