A method and apparatus for runway estimation under GNSS denied conditions for unmanned aerial vehicles (UAVs)

Abstract

A method and apparatus for runway estimation in GNSS-denied environments for unmanned aerial vehicles (UAVs). When GNSS signals are denied to the UAV, motion data is acquired, a runway image sequence is captured, and runway distance data is obtained. The runway image sequence is input into a convolutional neural network to extract the pixel coordinates of the runway boundary, runway centerline, and runway landing zone markers, and to identify the runway number. A body coordinate system is constructed. The motion data, runway boundary pixel coordinates, runway centerline pixel coordinates, runway landing zone marker pixel coordinates, and runway distance data are transformed into the body coordinate system to generate initial runway geometric parameters. Environmental features of the target area are scanned to generate a 3D point cloud, which is matched with the standard coordinates of runway marker points, and a scale constraint factor is output. The initial runway geometric parameters are optimized, and the runway magnetic heading angle is fused to output the target runway geometric parameters. Implementing this technical solution can meet the landing requirements of UAVs in GNSS-denied environments.

Description

Technical Field

[0001] This application relates to the technical field of data processing, specifically to a method and apparatus for runway estimation in a GNSS denied environment for unmanned aerial vehicles (UAVs). Background Technology

[0002] When drones land at unknown airports, they typically employ satellite-enhanced positioning (such as GBAS), visual template matching, and lidar 3D reconstruction techniques. Satellite-enhanced positioning relies on ground-based base stations to correct GPS signals for meter-level positioning; visual template matching generates a rough runway boundary by comparing satellite maps with real-time images; and lidar 3D reconstruction scans the environment to construct a point cloud model and fits the runway's geometric parameters. These techniques can meet basic landing requirements when GNSS (Global Navigation Satellite System) is available, but their failure risk is significant in satellite-denied scenarios.

[0003] In GNSS-denied environments, existing runway width estimation methods lack sufficient accuracy and are highly dependent on environmental conditions. For example, vision-based methods are susceptible to factors such as lighting and weather, while inertial navigation-based methods accumulate errors over time, making it difficult to maintain high accuracy over extended periods. These methods cannot meet the landing requirements of UAVs in unknown or complex environments. Summary of the Invention

[0004] To meet the landing requirements of UAVs in unknown or complex environments, this application provides a runway estimation method and apparatus for UAVs in GNSS denied environments.

[0005] The first aspect of this application provides a runway estimation method for UAVs in GNSS denied environments: When the GNSS signal of the UAV is denied, motion data is obtained through the inertial measurement unit on the UAV, runway image sequence is captured through the visual sensor on the UAV, and runway distance data is obtained through the millimeter-wave radar on the UAV. The runway image sequence is input into a preset convolutional neural network to extract the pixel coordinates of the runway boundary, the pixel coordinates of the runway centerline, the pixel coordinates of the runway junction area marker, and to identify the runway number. A body coordinate system is established based on the airframe structure of the UAV, which includes the wing mounting plane, fuselage, center of gravity position, and right wing reference point. The motion data, the pixel coordinates of the runway boundary, the pixel coordinates of the runway centerline, the pixel coordinates of the runway landing zone marker, and the runway distance data are converted to the body coordinate system to generate initial runway geometric parameters. The millimeter-wave radar scans the target area to generate a three-dimensional point cloud, which is then matched with the standard coordinates of runway marker points in a pre-stored airport environment database, and a scale constraint factor is output. The initial runway geometry parameters are optimized based on the scale constraint factor, and the runway magnetic heading angle determined based on the runway number is fused to output the target runway geometry parameters.

[0006] By adopting the above technical solution, integrating visual, millimeter-wave radar and inertial navigation data, and combining convolutional neural network feature extraction and point cloud matching to generate scale constraints, the shortcomings of single sensors in GNSS denied environments, such as susceptibility to light interference, error accumulation or insufficient accuracy, are effectively overcome. This achieves high-precision and robust estimation of runway geometric parameters under complex conditions, meeting the requirements for safe landing of UAVs in GNSS denied environments.

[0007] Optionally, the step of inputting the runway image sequence into a preset convolutional neural network to extract the runway boundary pixel coordinates, runway centerline pixel coordinates, runway touchdown area marker pixel coordinates, and identify the runway number includes: Motion compensation is performed on the runway image sequence based on the angular velocity data in the motion data to obtain a corrected image sequence; Based on the corrected image sequence, a multi-scale fusion feature map and the runway number are obtained through the convolutional neural network; Based on the multi-scale fused feature map, a runway overall region mask, a centerline region mask, and a junction area marker region mask are generated by a parallel feature decoder. Morphological closing operations are performed on the overall runway region mask to obtain an optimized overall runway region mask. Based on the optimized overall runway region mask, the pixel coordinates of the runway boundary are extracted using an edge detection operator. Based on the centerline region mask, the pixel coordinates of the runway centerline are calculated using a skeleton extraction algorithm. Based on the grounding area marker mask, the pixel coordinates of the runway grounding area marker are calculated using a geometric center positioning algorithm.

[0008] By adopting the above technical solutions, the perception and recognition accuracy of key elements of the runway under different distances and viewing angles have been improved. After post-processing such as morphology, edge detection and skeleton extraction, the accuracy and robustness of the pixel coordinate extraction of boundary, center line and junction area markers are ensured, thereby greatly enhancing the reliability and output quality of the vision subsystem under complex conditions.

[0009] Optionally, establishing the body coordinate system based on the UAV's airframe structure includes: Set the center of gravity position as the origin of the body coordinate system; The X-axis of the body coordinate system is established based on the fuselage, and the direction in which the nose of the UAV points is the positive direction of the X-axis; The Z-axis of the body coordinate system is established based on the wing mounting plane, with the positive direction of the Z-axis being perpendicular to the wing plane and pointing upwards. The Y-axis of the body coordinate system is established based on the right wing reference point, and the direction from the origin of the coordinate system to the right wing reference point is the positive direction of the Y-axis. The body coordinate system is established based on the origin, the X-axis, the Y-axis, and the Z-axis.

[0010] By adopting the above technical solution, a coordinate system strongly coupled with the airframe is constructed, with the UAV's center of gravity as the origin, the X-axis defined based on the fuselage direction, the Z-axis defined vertically to the wing plane, and the Y-axis defined by the right wing reference point. This coordinate system ensures that sensor data is directly correlated with flight dynamics, makes runway data projection conform to aerodynamic layout, eliminates installation deviations of traditional coordinate systems, provides a precise geometric framework for multi-source data fusion, and significantly improves the spatial consistency of runway width estimation and the physical interpretability of control commands in GNSS denied environments.

[0011] Optionally, the step of converting the motion data, the track boundary pixel coordinates, the track centerline pixel coordinates, the track contact area marker pixel coordinates, and the track distance data to the body coordinate system to generate initial track geometric parameters includes: Based on the camera intrinsic parameter matrix of the UAV and the motion data, the pixel coordinates of the runway boundary, the pixel coordinates of the runway centerline, and the pixel coordinates of the runway landing zone marker are uniformly calculated to a three-dimensional coordinate point set in the body coordinate system. The three-dimensional coordinate point set is compensated and corrected based on the motion data; Based on the compensated and corrected set of three-dimensional coordinate points, the runway centerline equation and runway plane equation are generated by fitting using a preset random sampling consensus algorithm. Based on the runway centerline equation and the runway plane equation, the initial runway geometric parameters are calculated and generated. The initial runway geometric parameters include the initial runway width, the initial runway heading angle, the initial aircraft distance from the runway extension line XTK, and the initial aircraft distance from the landing area.

[0012] By adopting the above technical solutions, the accumulation of errors caused by changes in UAV attitude in visual measurements is effectively suppressed; and the RANSAC algorithm is used to robustly fit the runway geometry model, which can accurately estimate the initial parameters such as runway width, heading angle and aircraft relative position in noisy and outlier data, providing a highly reliable data foundation for subsequent scale optimization and fusion.

[0013] Optionally, the compensation and correction of the three-dimensional coordinate point set based on the motion data includes: Extract the timestamp sequence from the motion data and align it with the capture time of the runway image sequence; Based on the aligned timestamp sequence, the UAV attitude angle at the corresponding moment is extracted from the motion data, and a rotation matrix is ​​generated based on the UAV attitude angle; The attitude rotation compensation is performed on the three-dimensional coordinate point set based on the rotation matrix; The acceleration values ​​in the motion data are integrated to obtain the displacement vector, and the displacement vector is used to perform translation compensation on the three-dimensional coordinate point set after attitude rotation compensation, and the compensated and corrected three-dimensional coordinate point set is output.

[0014] By adopting the above technical solution, precise spatiotemporal synchronization between visual coordinate points and UAV pose is achieved. The rotation matrix and displacement vector generated by attitude angle and acceleration integral are used to perform rotation and translation compensation on the point set, effectively eliminating visual measurement deviations caused by UAV maneuvering flight and significantly improving the spatial consistency and positional accuracy of the three-dimensional point set.

[0015] Optionally, the step of generating a three-dimensional point cloud using the environmental features of the target area scanned by the millimeter-wave radar, matching it with the standard coordinates of runway marker points in a pre-stored airport environment database, and outputting scale constraint factors includes: The millimeter-wave radar acquires the original echo signal of the target area; The original echo signal is digitally processed to generate three-dimensional point cloud data containing environmental features; The three-dimensional point cloud data is processed by ground point cloud segmentation and runway marker point identification to obtain the measured spatial coordinates of the runway marker points; The measured spatial location coordinates are matched with the standard coordinates of the target runway markers of the corresponding airport in the pre-stored airport environment database to generate a set of matching point pairs. For the set of matching point pairs, at least one measurement distance value is calculated based on the measurement spatial location coordinates, and a standard distance value corresponding to the measurement distance value is calculated based on the standard coordinates of the target runway marker point; Calculate the ratio of the standard distance value to the measured distance value, and determine the ratio as the scale constraint factor.

[0016] By adopting the above technical solution, a three-dimensional point cloud is generated and the runway marker points are segmented and identified. The points are then matched with pre-stored standard coordinates, thereby providing key absolute scale geometric constraints for the initial geometric parameters generated by pure vision and inertial navigation fusion. This effectively solves the scale uncertainty problem existing in monocular vision and significantly improves the accuracy and reliability of the final runway parameter estimation.

[0017] Optionally, the optimization of the initial runway geometry parameters based on the scale constraint factor, and the fusion of the runway magnetic heading angle determined based on the runway number, to output the target runway geometry parameters include: The initial runway width, the initial aircraft distance from the runway extension line XTK (Cross-Track Error, a relative value of the lateral deviation distance), and the initial aircraft distance from the landing area in the initial runway geometry parameters are multiplied by the scale constraint factor to obtain the corrected runway width, the corrected aircraft distance from the runway extension line XTK, and the corrected aircraft distance from the landing area. Based on a preset weighted fusion algorithm, the initial runway heading angle and the runway magnetic heading angle in the initial runway geometric parameters are weighted and calculated to generate the target runway heading angle; The corrected runway width, the corrected distance from the aircraft to the runway extension line (XTK), the corrected distance from the aircraft to the landing area, and the target runway heading angle are combined into the target runway geometric parameters and output.

[0018] By adopting the above technical solution, the scale uncertainty defect caused by the fusion of monocular vision and inertial navigation is solved. At the same time, the initial heading angle estimated by geometry is weighted and fused with the magnetic heading angle obtained based on the number lookup, which effectively combines the advantages of geometric measurement and prior information, and finally outputs high-precision, scale-accurate and heading-reliable target runway geometric parameters.

[0019] A second aspect of this application provides a runway estimation system for UAVs in GNSS denied environments, specifically including: The data acquisition module is used to acquire motion data through the inertial measurement unit on the UAV, capture runway image sequences through the visual sensor on the UAV, and acquire runway distance data through the millimeter-wave radar on the UAV when the UAV's GNSS signal is denied. The runway boundary extraction module is used to input the runway image sequence into a preset convolutional neural network to extract the runway boundary pixel coordinates, the runway centerline pixel coordinates, the runway junction area marker pixel coordinates, and identify the runway number. The body coordinate system construction module is used to establish a body coordinate system based on the airframe structure of the UAV; The initial geometric parameter estimation module is used to convert the motion data, the pixel coordinates of the runway boundary, the pixel coordinates of the runway centerline, the pixel coordinates of the runway contact area marker, and the runway distance data to the body coordinate system to generate initial runway geometric parameters. The scale constraint generation module is used to generate a three-dimensional point cloud by using the environmental features of the target area scanned by the millimeter-wave radar, match the standard coordinates of runway marker points in the pre-stored airport environment database, and output the scale constraint factor. The target geometry parameter generation module is used to optimize the initial runway geometry parameters based on the scale constraint factor, and integrate the runway magnetic heading angle determined based on the runway number to output the target runway geometry parameters.

[0020] A third aspect of this application provides an electronic device including a processor, a memory, a user interface, and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any of the foregoing.

[0021] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed, perform the method described in any of the preceding descriptions.

[0022] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: This approach leverages motion compensation and Convolutional Neural Networks (CNNs) to enhance the robustness of visual perception. Geometric consistency is ensured through a unified ontological coordinate system and motion correction. The RANSAC algorithm robustly fits initial parameters, and millimeter-wave radar point cloud matching is innovatively introduced to provide absolute scale constraints. Finally, high-precision results are output through scale correction and heading fusion. This scheme effectively overcomes the inherent limitations of single technologies, such as susceptibility to environmental interference, error accumulation, and scale ambiguity. It achieves all-weather, highly reliable runway parameter estimation, fundamentally guaranteeing the safety and accuracy of UAVs' autonomous landing in unknown and complex airports. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the system architecture of an embodiment of a runway estimation method or system for UAVs in GNSS denied environment, which applies this application. Figure 2 This is a flowchart illustrating a runway estimation method for UAVs in GNSS denied environments disclosed in an embodiment of this application. Figure 3 This is a schematic diagram of a runway estimation system for UAVs in GNSS denied environments, as disclosed in an embodiment of this application. Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application.

[0024] Explanation of reference numerals in the attached figures: 100, System architecture; 101, First terminal device; 102, Second terminal device; 103, Third terminal device; 104, Network; 105, Server; 301, Data acquisition module; 302, Runway boundary extraction module; 303, Body coordinate system construction module; 304, Initial geometric parameter estimation module; 305, Scale constraint generation module; 306, Target geometric parameter generation module; 401, Processor; 402, Communication bus; 403, User interface; 404, Network interface; 405, Memory. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0026] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0027] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0028] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0029] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as model training applications, video recognition applications, web browser applications, social platform software, etc.

[0030] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptops, and desktop computers, etc. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are imposed here.

[0031] This embodiment discloses a runway estimation method for UAVs in GNSS denied environments. Figure 2 This is a flowchart illustrating a runway estimation method for UAVs in GNSS denied environments, as disclosed in an embodiment of this application. Figure 2 As shown, the method includes the following steps: S201. When the GNSS signal of the UAV is denied, motion data is acquired through the inertial measurement unit on the UAV, runway image sequence is captured through the visual sensor on the UAV, and runway distance data is acquired through the millimeter-wave radar on the UAV.

[0032] Specifically, when the GNSS signal rejection of the UAV is detected to be continuous for more than a set duration threshold, the multi-sensor collaborative acquisition process is activated: the onboard inertial measurement unit acquires motion data streams in real time, including three-axis angular velocity, three-axis acceleration, and pitch / roll / yaw attitude angles. All data is embedded with microsecond-level hardware timestamps generated by a high-precision crystal oscillator; the synchronously triggered global shutter vision sensor captures a sequence of grayscale or RGB (Red-Green-Blue) images of the runway within the forward-downward field of view at a fixed frame rate, based on preset runway identification area parameters. Each frame of the image is processed... The automatic exposure control loop dynamically adjusts exposure parameters to suppress interference from sudden changes in illumination and marks timestamps that are strictly synchronized with the motion data. The millimeter-wave radar switches to landing scanning mode and acquires distance, azimuth, and radial velocity information of key targets on the runway surface through electronic beam scanning, generating a raw distance data packet with timestamps and confidence markers based on dynamic signal-to-noise ratio calculation. The above motion data, image sequences, and distance data are transmitted to the cloud server in real time via a high-speed serial bus. After being aligned by a timestamp-driven interpolation algorithm, a spatiotemporally unified perception dataset is formed for subsequent runway estimation processes.

[0033] S202. Input the runway image sequence into a preset convolutional neural network to extract the runway boundary pixel coordinates, runway centerline pixel coordinates, runway junction area marker pixel coordinates, and identify the runway number.

[0034] In this embodiment, the motion-compensated runway image sequence is first input frame by frame into a pre-set multi-task convolutional neural network trained on a large amount of aerial imagery data. This network can employ an encoder-decoder architecture, such as a U-Net++ model using ResNet or EfficientNet as the feature extraction backbone. During the network's forward propagation, the encoder extracts multi-scale depth features of the image; subsequently, the network processes these features in parallel through different "head" structures to complete multiple tasks: a fully connected classification head connected by a global average pooling layer is responsible for analyzing high-level semantic features and directly outputting the recognition result of the runway number (e.g., 22R); simultaneously, three parallel decoder branches upsample and segment the feature map at the pixel level, generating binary masks for the overall runway region, centerline region, and junction area marker region, respectively. Finally, the system performs a series of refined post-processing steps on the CNN output mask: morphological closing operations are applied to the overall runway region mask to fill internal holes and smooth boundaries, and then the pixel coordinate set of the runway boundary is accurately extracted using the Canny edge detection operator; a skeleton extraction algorithm is applied to the centerline region mask to refine it into a single-pixel-width curve, thereby obtaining the pixel coordinates of the runway centerline; and for each connected component of the docking area sign mask, the precise pixel coordinates of each sign are determined by calculating its geometric center (centroid), thus efficiently and accurately extracting all key visual information.

[0035] Optionally, the step of inputting the runway image sequence into a preset convolutional neural network to extract the runway boundary pixel coordinates, runway centerline pixel coordinates, runway landing zone marker pixel coordinates, and identify the runway number includes: performing motion compensation on the runway image sequence based on the angular velocity data in the motion data to obtain a corrected image sequence; based on the corrected image sequence, obtaining a multi-scale fusion feature map and the runway number through the convolutional neural network; based on the multi-scale fusion feature map, generating a runway overall region mask, a centerline region mask, and a landing zone marker region mask respectively through a parallel feature decoder; performing morphological closing operations on the runway overall region mask to obtain an optimized runway overall region mask, and extracting the runway boundary pixel coordinates based on the optimized runway overall region mask using an edge detection operator; calculating the runway centerline pixel coordinates based on the centerline region mask using a skeleton extraction algorithm; and calculating the runway landing zone marker pixel coordinates based on the landing zone marker region mask using a geometric center localization algorithm.

[0036] Specifically, before feeding the image sequence into the CNN, the system performs a motion compensation step based on motion data, aiming to eliminate image blurring and geometric distortion caused by rapid attitude changes (especially roll and pitch) during UAV flight. In practice, the system precisely synchronizes high-frequency (e.g., 200Hz) angular velocity data from the inertial measurement unit (IMU) with the image frame capture timestamps from the visual sensor. For two adjacent image frames (e.g., frame t and frame (t-1), the system extracts all angular velocity samples ω(t) within this time interval Δt. By integrating these angular velocity samples, the attitude rotation Δθ that occurred during this period can be accurately calculated. Based on this rotation, the system generates a 3x3 rotation matrix or a corresponding homography transformation matrix H. Subsequently, this transformation matrix H is applied to the frame t image, and through image warping, its pixels are aligned to the attitude reference frame of frame (t-1). This process effectively eliminates the blurring introduced by rotational motion, generating a clearer and more stable corrected image sequence, providing high-quality input for subsequent neural network recognition, and significantly improving the stability and accuracy of the recognition.

[0037] Furthermore, the obtained corrected image sequence is fed frame by frame into a pre-designed and trained multi-task CNN. This network can employ an advanced encoder-decoder architecture, such as using ResNet-50 or EfficientNet as the feature extraction backbone (encoder) and combining it with U-Net++ or DeepLabv3+ as the decoder structure. The encoder is responsible for extracting multi-scale features from the input image, ranging from low-level texture to high-level semantics. Based on this, the network is designed with parallel task processing branches. First, after the highest-level semantic feature map output by the encoder, a classification "head" is connected. This "head" typically consists of a global average pooling layer and several fully connected layers, and its task is to identify the runway number in the image, outputting classification results such as "04L" or "22R". Second, the decoder utilizes the feature maps from each stage of the encoder, performing feature fusion through skip connections to generate a multi-scale fused feature map rich in detail and semantic information.

[0038] Furthermore, based on this multi-scale fusion feature map, three parallel, structurally independent feature decoders are used to perform pixel-level semantic segmentation tasks. Each decoder branch is specifically trained to identify different targets and generates three independent binary masks: (1) a runway overall region mask, identifying the complete outline of the runway surface; (2) a centerline region mask, identifying the area where the dashed or solid centerline of the runway is located; and (3) a junction area marker region mask, identifying the zebra crossing and other marker areas at the runway entrance. This parallel decoding design ensures that the tasks do not interfere with each other, and each branch can focus more on learning the fine features of its specific target, thereby improving the overall accuracy of segmentation.

[0039] Furthermore, the system performs targeted post-processing on the three masks to extract precise pixel coordinates. For the overall runway region mask, a morphological closing operation (i.e., dilation followed by erosion) is first performed. This operation effectively fills small voids inside the mask caused by uneven lighting or marker occlusion and connects potentially broken boundaries, resulting in an optimized overall runway region mask with a more complete topological structure and smoother edges. Subsequently, an edge detection operator, such as the robust and accurate Canny operator, is applied to this optimized mask to extract a single-pixel-width runway contour. The set of all pixels on this contour is the required runway boundary pixel coordinates. For the centerline region mask, since it is usually a strip-shaped region with a certain pixel width, the system uses a skeleton extraction algorithm (such as the classic Zhang-Suen thinning algorithm) to process it. This algorithm iteratively peels away the boundary pixels of the region until the region is thinned to a single-pixel-width curve that cannot be further reduced, while maintaining the topological connectivity of the original region. All pixels on this finally generated centerline curve constitute the precise runway centerline pixel coordinates. For the runway landing zone marker mask, which may contain multiple unconnected white areas (corresponding to multiple zebra crossings), the system first applies connected component analysis to identify each independent marker region. Then, for each independent connected component, a geometric center localization algorithm is used to determine its unique representative location by calculating the average coordinates of all pixels within that region (i.e., the centroid). The set of all calculated centroid coordinates represents the pixel coordinates of the runway landing zone markers. Through this series of refined post-processing steps, the system transforms the coarse segmentation results output by the neural network into high-precision pixel coordinate data that can be directly used for subsequent geometric calculations.

[0040] S203. Establish a body coordinate system based on the airframe structure of the UAV, wherein the airframe structure includes the wing mounting plane, fuselage, center of gravity position, and right wing reference point.

[0041] Specifically, the origin O of the coordinate system is taken as the center of gravity of the UAV; the X-axis is parallel to the fuselage axis and passes through the origin O of the coordinate system. The direction from the origin O to the nose of the UAV is defined as the positive direction of the X-axis; the positive direction of the Z-axis is defined based on the direction of the normal vector of the wing mounting plane; the positive direction of the Y-axis is defined based on the position vector of the right wing reference point; finally, a right-handed rectangular coordinate system is established—the origin O is bound to the center of gravity, the X-axis is aligned with the fuselage heading, the Z-axis is perpendicular to the wing plane, and the Y-axis is uniquely determined by the right wing reference point. All coordinate axes are rigidly bound to the physical structure of the UAV.

[0042] Optionally, establishing a body coordinate system based on the UAV's airframe structure includes: setting the center of gravity position as the origin of the body coordinate system; establishing the X-axis of the body coordinate system based on the airframe, with the direction the UAV's nose points being the positive direction of the X-axis; establishing the Z-axis of the body coordinate system based on the wing mounting plane, with the positive direction of the Z-axis being perpendicular to the wing plane and pointing upwards; establishing the Y-axis of the body coordinate system based on the right wing reference point, with the positive direction of the Y-axis being the direction from the origin to the right wing reference point; and establishing the body coordinate system based on the origin, the X-axis, the Y-axis, and the Z-axis.

[0043] Specifically, given the known structural parameters of the UAV, the spatial position of the center of gravity is determined through pre-calibration or real-time state estimation: First, the theoretical coordinates of the center of gravity under the standard configuration are obtained by calling the onboard mass distribution database (usually located at the intersection of the longitudinal axis of the fuselage and the focal point of the wing chord); if the UAV carries a variable payload (such as a pod or a drop object), the load distribution at each support point is measured in real time by the pressure sensor array installed on the landing gear, and the actual coordinates of the center of gravity are calculated online using the moment balance equation in combination with the three-dimensional model of the fuselage (accuracy up to ±2cm); finally, the coordinates of the center of gravity are set as the origin O of the body coordinate system. This operation ensures that all subsequent sensor data conversions are based on the core kinematics of the UAV, which not only ensures that the coordinate system is rigidly bound to the body when the attitude changes, but also provides a unified dynamic reference point for motion compensation, fundamentally eliminating the cumulative error caused by reference system drift.

[0044] Furthermore, high-precision laser reflection markers are fixed at the nose and tail of the UAV, and the three-dimensional coordinates of the two points are acquired in real time by an airborne laser tracker. The connecting vector is calculated and normalized to obtain the unit direction vector of the X-axis. With the center of gravity of the aircraft as the origin of the coordinate system, the direction along this vector pointing to the nose is defined as the positive direction of the X-axis. At the same time, the pitch rate data monitored in real time by the IMU is integrated to dynamically compensate for the axial deviation caused by the bending deformation of the fuselage during flight, ensuring that the X-axis is always rigidly bound to the physical longitudinal axis of the fuselage, and the heading stability error is controlled within ±0.1°.

[0045] Furthermore, optical targets were deployed at key mounting holes at the wing root (L1-L3 on the left wing and R1-R3 on the right wing). Their three-dimensional coordinates were acquired in real time using an airborne total station, and the least squares method was used to fit the wing mounting plane equation. Extracting normal vectors =(A, B, C), the constant term D and the coefficients A, B, C together determine the position of the plane in space; based on the gravitational acceleration vector Determine the vertical upward direction: Calculate If the result is negative, then take If the Z-axis is positive, then take the positive Z-axis; otherwise, take the positive Z-axis. After normalization, we obtain the unit vector. (Plane fitting accuracy ±0.5mm); Synchronously fuse wing strain sensor data, and when aerodynamic loads are detected causing wing surface deformation to exceed a threshold (e.g., curvature change >0.1rad / m), dynamically update plane parameters and recalculate. Ensure that the Z-axis is always perpendicular to the physical wing plane and pointing upwards, with an orthogonality error of ≤0.05 radians with the X-axis.

[0046] Furthermore, a high-precision titanium alloy reference pin is fixedly installed at the root of the right wing of the UAV as a physical reference point. The three-dimensional spatial position of this point relative to the origin of the center of gravity is measured in real time using an airborne laser tracker. First, the original direction vector from the center of gravity to this reference point is calculated, and then geometric orthogonalization is performed: the part of the vector that overlaps with the established X-axis direction is removed, generating a component perpendicular to the X-axis. This component is verified against the Z-axis direction using the right-hand rule (ensuring that when the thumb points in the positive X-direction, the four fingers are bent and rotated from the Y-axis to the Z-axis). After confirming the consistency of the direction, the perpendicular component is normalized to obtain the unit direction vector of the Y-axis. Throughout the process, the data from the wing root strain sensor is monitored in real time. When aerodynamic deformation causes the displacement of the reference point to exceed 0.5 mm, the position is immediately remeasured and the vector calculation is updated. Ultimately, it is ensured that the Y-axis strictly maintains the direction from the origin of the center of gravity to the physical reference point of the right wing, and the orthogonality error of the right-hand coordinate system formed by the X-axis and Z-axis does not exceed 0.04 radians.

[0047] Furthermore, a rotation matrix is ​​formed by combining three axial unit vectors in real time, and the origin coordinates are used as translation components to construct a 4×4 homogeneous transformation matrix. A dynamic binding mechanism is initiated simultaneously—using FPGA (Field Programmable Gate Array) hardware to read IMU angular velocity data at a frequency of 1000Hz, and updating the coordinate system attitude in real time through an exponential mapping algorithm to ensure that the coordinate system is rigidly synchronized with the body during UAV maneuvers. Orthogonality verification (calculating the deviation magnitude of the axial cross product from the theoretical vector < 0.000001) and right-hand rule detection (the determinant value is strictly kept at +1) are performed in each cycle. Finally, a spatiotemporally unified body coordinate system reference is output, providing centimeter-level spatial consistency assurance for multi-source sensor data (position error ≤ 3cm, heading angle error ≤ 0.1°).

[0048] S204. Convert the motion data, the track boundary pixel coordinates, the track centerline pixel coordinates, the track contact area marker pixel coordinates, and the track distance data to the body coordinate system to generate initial track geometric parameters.

[0049] In this embodiment, the system fuses the pixel coordinates of the runway boundary, centerline, and touchdown area markers obtained from the vision processing module with runway distance data from millimeter-wave radar and real-time motion data (attitude angles) from the IMU to calculate the initial runway geometric parameters. First, using a pre-calibrated camera intrinsic parameter matrix and the attitude angles provided by the IMU, combined with depth information measured by the millimeter-wave radar, the system uses the inverse operation of the perspective projection model to convert the two-dimensional pixel coordinates of each feature point into three-dimensional spatial coordinates in the UAV's body coordinate system, forming an initial, but potentially motion-distorted, three-dimensional point cloud. Subsequently, the system uses continuous motion data (angular velocity and acceleration integral) provided by the IMU to compensate and correct this point cloud, eliminating inter-frame displacement and rotation errors caused by the UAV's own motion, generating a high-density, highly consistent three-dimensional point cloud under a unified spatiotemporal reference. Finally, the system employs the highly robust Random Sample Consensus (RANSAC) algorithm to fit a geometric model to the corrected 3D point cloud: the 3D centerline equation of the runway is fitted using the centerline point set, and the runway plane equation is fitted using all runway point sets. Based on these two precise mathematical models, the system can directly calculate key initial runway geometric parameters, including the initial runway width, the initial heading angle of the UAV relative to the runway, the initial lateral deviation distance (XTK), and the initial distance along the runway direction to the landing zone marker.

[0050] Optionally, the step of converting the motion data, runway boundary pixel coordinates, runway centerline pixel coordinates, runway touchdown marker pixel coordinates, and runway distance data to the body coordinate system to generate initial runway geometric parameters includes: based on the UAV's camera intrinsic parameter matrix and the motion data, uniformly calculating the runway boundary pixel coordinates, runway centerline pixel coordinates, and runway touchdown marker pixel coordinates to a three-dimensional coordinate point set in the body coordinate system; compensating and correcting the three-dimensional coordinate point set based on the motion data; based on the compensated and corrected three-dimensional coordinate point set, fitting and generating the runway centerline equation and runway plane equation using a preset random sampling consensus algorithm; and calculating and generating the initial runway geometric parameters based on the runway centerline equation and runway plane equation, wherein the initial runway geometric parameters include the initial runway width, initial runway heading angle, initial aircraft distance from the runway extension line (XTK), and initial aircraft distance from the touchdown area.

[0051] Specifically, the system calls a pre-calibrated offline UAV camera intrinsic parameter matrix, which describes the projection relationship of the camera from the 3D world to the 2D image. For each pixel identified as a runway boundary, centerline, or landing zone marker, the system uses this intrinsic parameter matrix to perform inverse calculations, back-projecting it into a 3D spatial ray originating from the camera's optical center and passing through that pixel. Simultaneously, the system obtains the UAV's current precise attitude from motion data (e.g., high-frequency attitude angles provided by the inertial measurement unit, IMU) and combines this with altitude information from the barometer or GPS to construct a mathematical description of the ground plane in the body coordinate system. By calculating the intersection of the aforementioned 3D ray with the ground plane, the 3D spatial coordinates of that pixel in the UAV's body coordinate system can be uniquely determined. The system repeats this process for all feature pixels, thereby generating an initial 3D point cloud containing complete runway structure information.

[0052] Furthermore, the system performs motion data-based compensation and correction on the 3D coordinate point set generated in the previous step. The purpose of this step is to eliminate geometric distortions caused by the continuous motion of the UAV during data acquisition and the minute delays between the data streams of various sensors. Specifically, the system uses the timestamp of the latest frame as a unified time reference. For each 3D coordinate point acquired at a historical moment, the system utilizes high-frequency IMU data (including angular velocity and acceleration) to accurately reconstruct the displacement and attitude changes experienced by the UAV from that historical moment to the current reference moment by integrating it over the corresponding time difference. This change can be represented as a rigid body transformation matrix (including rotation and translation components). The system applies this transformation matrix to the corresponding historical 3D coordinate point, thereby "moving" it to the current coordinate system. By performing such compensation and correction on the entire point set, a precise 3D point cloud that is perfectly aligned in space and time and has a highly consistent geometric shape can be obtained.

[0053] Furthermore, based on the compensated and corrected set of 3D coordinate points, the system uses a pre-defined RANSAC algorithm to fit and generate the key geometric model of the runway. The core advantage of choosing the RANSAC algorithm lies in its extremely high robustness to noise and outliers (which may be caused by image recognition errors or environmental interference). This process can be further subdivided into two independent fitting tasks: First, the system uses all 3D points belonging to the runway centerline as input data, and iteratively uses RANSAC to find and ultimately determine the 3D straight line model that can be supported by the most "inliers," which is the runway centerline equation. Second, the system uses all 3D points related to the runway (including boundaries, centerline, etc.) as input data, and uses the same method to fit a 3D planar model that best describes the runway surface, i.e., the runway planar equation.

[0054] Furthermore, after obtaining the precise runway centerline equation and runway plane equation, the system generates the final initial runway geometric parameters through a series of deterministic geometric calculations. These parameters can be calculated in the following ways: calculating the initial runway width by solving for and summing the average vertical distances from the left and right boundary points of the runway to the fitted runway centerline on the runway plane; calculating the initial runway heading angle by projecting the direction vector of the runway centerline equation onto the horizontal plane and calculating the angle between it and the UAV's nose direction (usually the forward axis of the body coordinate system); calculating the initial aircraft distance from the runway extension (XTK) by solving for the vertical distance from the UAV origin ((0, 0, 0) in the body coordinate system) to the three-dimensional straight line of the runway centerline; and calculating the initial aircraft distance from the landing zone by simultaneously projecting the UAV's position and the three-dimensional coordinates of the landing zone marker onto the runway centerline and then calculating the distance between these two projected points. This series of precise initial parameters provides a reliable decision-making basis for subsequent autonomous landing trajectory planning.

[0055] Optionally, the compensation and correction of the three-dimensional coordinate point set based on the motion data includes: extracting the timestamp sequence from the motion data and aligning it with the capture time of the runway image sequence; extracting the UAV attitude angle at the corresponding moment from the motion data based on the aligned timestamp sequence, and generating a rotation matrix based on the UAV attitude angle; performing attitude rotation compensation on the three-dimensional coordinate point set based on the rotation matrix; integrating the acceleration values ​​in the motion data to obtain a displacement vector, and performing translation compensation on the attitude rotation-compensated three-dimensional coordinate point set using the displacement vector, and outputting the compensated and corrected three-dimensional coordinate point set.

[0056] Specifically, the system extracts high-frequency timestamp sequences from the motion data stream (typically generated by an inertial measurement unit, IMU) in parallel, and extracts the capture time (i.e., the timestamp of the exposure moment) of each frame from the runway image sequence. Since the clocks of different sensors (camera and IMU) may have slight asynchrony, the system needs to establish a precise correspondence. One preferred implementation is to calculate the precise motion state at each image frame's capture time using an interpolation algorithm (e.g., linear interpolation or spline interpolation) from the IMU's timestamp sequence; another approach is to use a nearest neighbor search method to find the temporally closest IMU data point. This step ensures that each frame and its calculated 3D coordinates match a temporally synchronized UAV motion state (attitude and acceleration), thus providing high-precision data input for subsequent compensation calculations.

[0057] Furthermore, based on the aligned timestamps, the system performs attitude rotation compensation on the set of 3D coordinate points. For each motion data point time-aligned with the image frame, the system extracts the UAV attitude angles at that moment, namely roll, pitch, and yaw. Based on these attitude angles, the system generates a 3x3 rotation matrix using the standard Euler angle to rotation matrix conversion formula. This rotation matrix precisely describes the spatial orientation of the UAV's body coordinate system relative to a fixed reference coordinate system (e.g., the coordinate system at the start of the mission or the latest image frame). The system applies this rotation matrix to all 3D coordinate points calculated for that image frame, transforming these points from their respective acquisition coordinate systems to a unified reference coordinate system through matrix multiplication. This step aims to eliminate inconsistencies in point cloud orientation caused by attitude changes such as roll and pitch during UAV flight.

[0058] Furthermore, after completing attitude rotation compensation, the system performs translation compensation to eliminate the effects of UAV displacement. The system extracts acceleration values ​​(typically triaxial acceleration) from motion data time-aligned with each image frame. To obtain displacement information, the system performs a double integration of the acceleration values ​​over the corresponding time interval, or more preferably, integrates the angular velocity to obtain the attitude, and integrates the acceleration after attitude rotation in the reference coordinate system to obtain the displacement vector. This displacement vector describes the spatial positional movement of the UAV from the moment the image frame was acquired to the reference time. The system applies the calculated displacement vector, in the form of vector addition, to the 3D coordinate point set after attitude rotation compensation. After translation compensation, the point clouds acquired at all historical moments are not only attitude-aligned, but their spatial positions are also unified to the position at the reference time. Finally, the system outputs a 3D coordinate point set after dual attitude rotation and translation compensation correction. This point set is geometrically highly consistent, laying a solid foundation for subsequent accurate runway model fitting.

[0059] S205. Using the environmental features of the target area scanned by the millimeter-wave radar, a three-dimensional point cloud is generated, which is matched with the standard coordinates of the runway marker points in the pre-stored airport environment database, and the scale constraint factor is output.

[0060] Specifically, the airborne millimeter-wave radar actively scans and detects the target area ahead, especially the runway and its surrounding environment. Based on the time-of-flight principle of electromagnetic waves, the radar generates a 3D point cloud with true physical dimensions and no scale ambiguity. This point cloud accurately describes the geometric distribution of hard targets such as runway lights, signs, or navigation facilities. Simultaneously, the system loads a pre-stored airport environment database containing the standard 3D coordinates of key runway markers (such as runway threshold lights and landing zone markers). Subsequently, the system uses a point cloud registration algorithm (e.g., the Iterative Closest Point (ICP) algorithm or its variants) to match the real-time generated radar point cloud with the standard coordinate point set in the database to identify at least two corresponding markers. By calculating the standard distance between these two corresponding points in the database and the measured distance in the radar point cloud, the system obtains the ratio between the two and outputs this ratio as a high-precision scale constraint factor. This ratio is used for subsequent global scale correction of the visual solution results, ensuring the authenticity and accuracy of all geometric parameters.

[0061] Optionally, the step of generating a three-dimensional point cloud by scanning the environmental features of the target area using the millimeter-wave radar, matching it with the standard coordinates of runway marker points in a pre-stored airport environment database, and outputting a scale constraint factor includes: acquiring the original echo signal of the target area through the millimeter-wave radar; digitizing the original echo signal to generate three-dimensional point cloud data containing environmental features; performing ground point cloud segmentation and runway marker point identification processing on the three-dimensional point cloud data to obtain the measured spatial coordinates of the runway marker points; matching the measured spatial coordinates with the standard coordinates of the target runway marker points of the corresponding airport in the pre-stored airport environment database to generate a set of matching point pairs; calculating at least one measured distance value based on the measured spatial coordinates for the set of matching point pairs, and calculating a standard distance value corresponding to the measured distance value based on the standard coordinates of the target runway marker points; calculating the ratio of the standard distance value to the measured distance value, and determining the ratio as the scale constraint factor.

[0062] Specifically, the system actively acquires raw echo signals from the target area using an onboard millimeter-wave radar. In practice, the radar antenna transmits an electromagnetic beam of a predetermined waveform (e.g., frequency-modulated continuous wave, FMCW) towards the runway approach direction. These beams are reflected when they encounter environmental features such as the runway surface, runway lighting system, navigation equipment, or other obstacles. The radar receiver captures these raw echo signals carrying target information. This raw signal is an analog signal containing rich physical information such as the target's distance, velocity, angle, and reflection intensity relative to the radar, providing the initial data foundation for subsequently generating a high-precision 3D point cloud.

[0063] Furthermore, the system performs digital processing on the acquired raw echo signals to generate 3D point cloud data containing environmental features. This processing is typically performed on the radar's internal digital signal processor (DSP) or a dedicated airborne computing unit. A typical processing flow includes: first, performing a Fast Fourier Transform (FFT) on the echo signals to extract the target's range information by analyzing the frequency difference; then, performing a second FFT on the signals from different receiving antenna elements or using other angle measurement algorithms (such as MUSIC or ESPRIT) to calculate the target's azimuth and elevation angles. Combining the range and angle information, the system can generate spherical coordinates for each valid echo point in the radar coordinate system, which can be further converted to Cartesian coordinates (X, Y, Z), thereby constructing a 3D point cloud data capable of representing the external environment in three dimensions.

[0064] Furthermore, the system performs ground point cloud segmentation and runway marker identification processing on the generated 3D point cloud data to accurately extract the locations of key landmarks. To reduce interference, the system first uses ground segmentation algorithms (such as RANSAC plane fitting or grid-based elevation difference methods) to filter out point clouds representing flat runway surfaces or ground, retaining only non-ground point clouds. Then, a clustering algorithm (such as DBSCAN) is applied to the non-ground point clouds to aggregate spatially adjacent points into different target clusters. The system further analyzes the geometric features (such as size, height, and shape) and physical features (such as radar cross section (RCS) values) of each target cluster and compares them with a preset runway marker feature model to identify specific markers such as runway threshold lights, PAPI light clusters, or runway edge lights. Once identification is successful, the system calculates the precise centroid location of these markers in the radar measurement space, i.e., the measurement space coordinates.

[0065] Furthermore, the system performs location matching between the extracted measured spatial coordinates and a pre-stored airport environment database to establish a correspondence between the measured points and real-world landmarks. This database precisely stores the official geodetic coordinates or standard coordinates in the airport coordinate system for all key runway markers at the target airport. The matching process utilizes the UAV's current coarse GPS positioning information to limit the search area. Within this area, the system compares the spatial topology of the measured marker cluster (e.g., the arrangement shape and spacing of a group of lights) with the layout of standard markers in the database, employing a point pattern matching algorithm to find the optimal match. Upon successful matching, the system generates a set of matched point pairs, where each element contains a measured spatial coordinate and a corresponding standard coordinate.

[0066] Furthermore, the system determines the final scale constraint factor based on this set of matched point pairs through geometric calculations. To improve robustness, the system selects at least two matched point pairs from the set. Then, based on the measured spatial coordinates of these two point pairs, the system calculates the Euclidean distance to obtain a measured distance value. Similarly, based on the standard coordinates of the target runway markers corresponding to these two point pairs, the system calculates the standard distance between them, which represents the true physical distance between these two landmarks in the real world. Finally, the system determines the high-precision scale constraint factor by calculating the ratio of the standard distance value to the measured distance value (i.e., scale constraint factor = standard distance / measured distance). This factor can accurately correct the entire visual localization and mapping results from a relative scale space to the true physical scale space, thereby eliminating the scale ambiguity of visual measurements.

[0067] S206. Optimize the initial runway geometry parameters based on the scale constraint factor, and integrate the runway magnetic heading angle determined based on the runway number to output the target runway geometry parameters.

[0068] Specifically, the system performs final optimization and fusion of the initial runway geometry parameters to output target runway geometry parameters with physical realism and high accuracy. Specifically, the system first applies the scale constraint factor determined by millimeter-wave radar in the preceding steps as a global scaling factor directly to all length-related initial runway geometry parameters obtained through visual analysis. This includes multiplying the relative position vector between the UAV and runway reference points (such as the entrance or center point), as well as dimensional parameters such as the estimated runway width, by this scale constraint factor, thereby accurately correcting the entire visual solution from a relative scale space to a true metric unit space. Simultaneously, the system uses the standard runway magnetic heading angle (e.g., 40 degrees) resolved from the runway number (e.g., 04R) as a high-precision external truth constraint. The system converts this magnetic heading angle into a direction vector and uses it to directly replace or correct the runway centerline direction vector initially estimated by the visual algorithm, thereby accurately correcting the runway's spatial attitude. Through the dual processing of scale optimization and heading angle fusion, the system finally outputs a complete set of target runway geometric parameters. This parameter set accurately describes the runway's three-dimensional position in the real-world coordinate system, its precise orientation aligned with Earth's magnetic north, and its width that conforms to actual physical dimensions, among other key information.

[0069] Optionally, optimizing the initial runway geometry parameters based on the scale constraint factor and fusing the runway magnetic heading angle determined based on the runway number to output the target runway geometry parameters includes: multiplying the initial runway width, initial aircraft distance from the runway extension line (XTK), and initial aircraft distance from the landing area in the initial runway geometry parameters by the scale constraint factor to obtain the corrected runway width, corrected aircraft distance from the runway extension line (XTK), and corrected aircraft distance from the landing area; performing a weighted calculation on the initial runway heading angle and the runway magnetic heading angle in the initial runway geometry parameters based on a preset weighted fusion algorithm to generate the target runway heading angle; and combining the corrected runway width, the corrected aircraft distance from the runway extension line (XTK), the corrected aircraft distance from the landing area, and the target runway heading angle into the target runway geometry parameters and outputting them.

[0070] Specifically, the system uses the scale constraint factor obtained from radar calibration as a multiplier, applying it to each dimension of the initial runway geometry parameters output by the vision algorithm. This includes, but is not limited to: multiplying the initial runway width (a value expressed in pixels or other relative units) by the scale constraint factor to obtain the corrected runway width in meters; multiplying the initial aircraft distance from the runway extension (XTK) by the scale constraint factor to obtain the corrected aircraft distance from the runway extension (XTK) in meters; and multiplying the initial aircraft distance from the landing zone (a relative value of longitudinal distance) by the scale constraint factor to obtain the corrected aircraft distance from the landing zone in meters. This step fundamentally eliminates the scale uncertainty inherent in monocular vision, giving all distance-related parameters physical meaning in the real world.

[0071] Furthermore, the system performs high-precision fusion of the runway heading angle. The system employs a pre-defined weighted fusion algorithm to optimally combine the initial runway heading angle calculated in real-time from visual perception with the runway magnetic heading angle from prior knowledge. One specific implementation uses a weighted average method, which can be expressed as: Target runway heading angle = w1 × Initial runway heading angle + w2 × Runway magnetic heading angle, where the sum of weights w1 and w2 is 1. These weights can be dynamically adjusted according to the flight phase or the confidence level of visual recognition; for example, when long-range visual features are unstable, the runway magnetic heading angle is given a higher weight w2. In another preferred embodiment, a Kalman filter can be used for fusion, taking the initial runway heading angle as a measurement and the runway magnetic heading angle as part of the system model or a strong prior. Through the prediction and update iteration of the filter, a smoother and more robust target runway heading angle that reflects real-time dynamic changes and is constrained by high-precision ground truth is output.

[0072] Furthermore, the system integrates and outputs all the corrected and fused parameters. The system combines the corrected runway width, the corrected aircraft-to-runway extension (XTK) distance, the corrected aircraft-to-landing zone distance obtained in the preceding steps, and the target runway heading angle obtained through weighted fusion into a structured dataset. This dataset constitutes the final target runway geometric parameters. This parameter set may further include other relevant geometric state quantities, such as the aircraft's altitude relative to the runway threshold and the runway slope angle. The system outputs this complete parameter set to the flight control system or navigation display system as the core basis for subsequent autonomous landing decisions, trajectory planning, and control, ensuring the safety and accuracy of the entire landing process.

[0073] This embodiment also discloses a runway estimation system for UAVs in GNSS denied environments. Figure 3 This is a schematic diagram of a runway estimation system for UAVs in GNSS denied environments, as disclosed in an embodiment of this application. Figure 3 As shown, the system includes: The data acquisition module 301 is used to acquire motion data through the inertial measurement unit on the UAV, capture runway image sequences through the visual sensor on the UAV, and acquire runway distance data through the millimeter-wave radar on the UAV when the UAV's GNSS signal is denied. The runway boundary extraction module 302 is used to input the runway image sequence into a preset convolutional neural network to extract the runway boundary pixel coordinates, the runway centerline pixel coordinates, the runway junction area marker pixel coordinates, and identify the runway number. The body coordinate system construction module 303 is used to establish a body coordinate system based on the body structure of the UAV; The initial geometric parameter estimation module 304 is used to convert the motion data, the pixel coordinates of the runway boundary, the pixel coordinates of the runway centerline, the pixel coordinates of the runway contact area marker, and the runway distance data to the body coordinate system to generate initial runway geometric parameters. The scale constraint generation module 305 is used to generate a three-dimensional point cloud by using the environmental features of the target area scanned by the millimeter-wave radar, match the standard coordinates of the runway marker points in the pre-stored airport environment database, and output the scale constraint factor. The target geometry parameter generation module 306 is used to optimize the initial runway geometry parameters based on the scale constraint factor, and integrate the runway magnetic heading angle determined based on the runway number to output the target runway geometry parameters.

[0074] Optionally, the runway boundary extraction module 302 is specifically used for: performing motion compensation on the runway image sequence based on the angular velocity data in the motion data to obtain a corrected image sequence; obtaining a multi-scale fusion feature map and the runway number through the convolutional neural network based on the corrected image sequence; generating a runway overall region mask, a centerline region mask, and a junction area marker region mask respectively through a parallel feature decoder based on the multi-scale fusion feature map; performing morphological closing operations on the runway overall region mask to obtain an optimized runway overall region mask; and extracting the runway boundary pixel coordinates through an edge detection operator based on the optimized runway overall region mask; calculating the runway centerline pixel coordinates through a skeleton extraction algorithm based on the centerline region mask; and calculating the runway junction area marker pixel coordinates through a geometric center localization algorithm based on the junction area marker region mask.

[0075] Optionally, the body coordinate system construction module 303 is specifically used for: setting the center of gravity position as the origin of the body coordinate system; establishing the X-axis of the body coordinate system based on the fuselage, with the direction in which the nose of the UAV points being the positive direction of the X-axis; establishing the Z-axis of the body coordinate system based on the wing mounting plane, with the positive direction of the Z-axis being perpendicular to the wing plane and pointing upwards; establishing the Y-axis of the body coordinate system based on the right wing reference point, with the positive direction of the Y-axis being the direction from the origin to the right wing reference point; and establishing the body coordinate system based on the origin, the X-axis, the Y-axis, and the Z-axis.

[0076] Optionally, the initial geometric parameter estimation module 304 is specifically used for: based on the camera intrinsic parameter matrix of the UAV and the motion data, uniformly calculating the runway boundary pixel coordinates, the runway centerline pixel coordinates, and the runway touchdown area marker pixel coordinates to a three-dimensional coordinate point set in the body coordinate system; compensating and correcting the three-dimensional coordinate point set based on the motion data; based on the compensated and corrected three-dimensional coordinate point set, fitting and generating the runway centerline equation and runway plane equation using a preset random sampling consensus algorithm; and calculating and generating the initial runway geometric parameters based on the runway centerline equation and runway plane equation, wherein the initial runway geometric parameters include the initial runway width, the initial runway heading angle, the initial aircraft distance from the runway extension line XTK, and the initial aircraft distance from the touchdown area.

[0077] Optionally, the initial geometric parameter estimation module 304 is specifically used for: extracting the timestamp sequence from the motion data and aligning it with the capture time of the runway image sequence; extracting the UAV attitude angle at the corresponding moment from the motion data based on the aligned timestamp sequence, and generating a rotation matrix based on the UAV attitude angle; performing attitude rotation compensation on the three-dimensional coordinate point set based on the rotation matrix; integrating the acceleration values ​​in the motion data to obtain a displacement vector, and performing translation compensation on the attitude rotation-compensated three-dimensional coordinate point set using the displacement vector, and outputting the compensated and corrected three-dimensional coordinate point set.

[0078] Optionally, the scale constraint generation module 305 is specifically used for: acquiring the original echo signal of the target area through the millimeter-wave radar; digitizing the original echo signal to generate three-dimensional point cloud data containing environmental features; performing ground point cloud segmentation and runway marker point identification processing on the three-dimensional point cloud data to obtain the measured spatial coordinates of the runway marker points; matching the measured spatial coordinates with the standard coordinates of the target runway marker points of the corresponding airport in the pre-stored airport environment database to generate a set of matching point pairs; calculating at least one measured distance value based on the measured spatial coordinates for the set of matching point pairs, and calculating a standard distance value corresponding to the measured distance value based on the standard coordinates of the target runway marker points; calculating the ratio of the standard distance value to the measured distance value, and determining the ratio as the scale constraint factor.

[0079] Optionally, the target geometry parameter generation module 306 is specifically used to: multiply the initial runway width, the initial aircraft distance from the runway extension line XTK, and the initial aircraft distance from the landing area in the initial runway geometry parameters by the scale constraint factor to obtain the corrected runway width, the corrected aircraft distance from the runway extension line XTK, and the corrected aircraft distance from the landing area; perform weighted calculation on the initial runway heading angle and the runway magnetic heading angle in the initial runway geometry parameters based on a preset weighted fusion algorithm to generate the target runway heading angle; and combine the corrected runway width, the corrected aircraft distance from the runway extension line XTK, the corrected aircraft distance from the landing area, and the target runway heading angle into the target runway geometry parameters and output them.

[0080] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0081] This embodiment also discloses an electronic device, as shown in the reference. Figure 4 The electronic device may include: at least one processor 401, at least one communication bus 402, user interface 403, network interface 404, and at least one memory 405.

[0082] The communication bus 402 is used to enable communication between these components.

[0083] The user interface 403 may include a display screen and a camera. Optionally, the user interface 403 may also include a standard wired interface and a wireless interface.

[0084] The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0085] The processor 401 may include one or more processing cores. The processor 401 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 405, and by calling data stored in memory 405. Optionally, the processor 401 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 401 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 401.

[0086] The memory 405 may include random access memory (RAM) or read-only memory. Optionally, the memory 405 may include a non-transitory computer-readable storage medium. The memory 405 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 405 may also be at least one storage device located remotely from the aforementioned processor 401. Figure 4As shown, the memory 405, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for a runway estimation method under GNSS denial environment for unmanned aerial vehicles.

[0087] exist Figure 4 In the electronic device shown, the user interface 403 is mainly used to provide an input interface for the user and to obtain the user input data; while the processor 401 can be used to call the application stored in the memory 405 for a runway estimation method under GNSS denial environment of a UAV. When executed by one or more processors 401, the electronic device performs one or more methods as described in the above embodiments.

[0088] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps may be performed in other orders or simultaneously.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some service interfaces; indirect couplings or communication connections between apparatuses or units may be electrical or other forms.

[0090] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory 405 and includes several instructions to cause a computer device to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned memory 405 includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

Claims

1. A runway estimation method for UAVs in GNSS denied environments, characterized in that, The method includes: When the GNSS signal of the UAV is denied, motion data is obtained through the inertial measurement unit on the UAV, runway image sequence is captured through the visual sensor on the UAV, and runway distance data is obtained through the millimeter-wave radar on the UAV. The runway image sequence is input into a preset convolutional neural network to extract the pixel coordinates of the runway boundary, the pixel coordinates of the runway centerline, the pixel coordinates of the runway junction area marker, and to identify the runway number. A body coordinate system is established based on the airframe structure of the UAV, which includes the wing mounting plane, fuselage, center of gravity position, and right wing reference point. The motion data, the pixel coordinates of the runway boundary, the pixel coordinates of the runway centerline, the pixel coordinates of the runway landing zone marker, and the runway distance data are converted to the body coordinate system to generate initial runway geometric parameters. The millimeter-wave radar scans the target area to generate a three-dimensional point cloud, which is then matched with the standard coordinates of runway marker points in a pre-stored airport environment database, and a scale constraint factor is output. The initial runway geometry parameters are optimized based on the scale constraint factor, and the runway magnetic heading angle determined based on the runway number is fused to output the target runway geometry parameters.

2. The method according to claim 1, characterized in that, The step of inputting the runway image sequence into a preset convolutional neural network to extract the runway boundary pixel coordinates, runway centerline pixel coordinates, runway touchdown area marker pixel coordinates, and identify the runway number includes: Motion compensation is performed on the runway image sequence based on the angular velocity data in the motion data to obtain a corrected image sequence; Based on the corrected image sequence, a multi-scale fusion feature map and the runway number are obtained through the convolutional neural network; Based on the multi-scale fused feature map, a runway overall region mask, a centerline region mask, and a junction area marker region mask are generated by a parallel feature decoder. Morphological closing operations are performed on the overall runway region mask to obtain an optimized overall runway region mask. Based on the optimized overall runway region mask, the pixel coordinates of the runway boundary are extracted using an edge detection operator. Based on the centerline region mask, the pixel coordinates of the runway centerline are calculated using a skeleton extraction algorithm. Based on the grounding area marker mask, the pixel coordinates of the runway grounding area marker are calculated using a geometric center positioning algorithm.

3. The method according to claim 1, characterized in that, The establishment of the body coordinate system based on the airframe structure of the UAV includes: Set the center of gravity position as the origin of the body coordinate system; The X-axis of the body coordinate system is established based on the fuselage, and the direction in which the nose of the UAV points is the positive direction of the X-axis; The Z-axis of the body coordinate system is established based on the wing mounting plane, with the positive direction of the Z-axis being perpendicular to the wing plane and pointing upwards. The Y-axis of the body coordinate system is established based on the right wing reference point, and the direction from the origin of the coordinate system to the right wing reference point is the positive direction of the Y-axis. The body coordinate system is established based on the origin, the X-axis, the Y-axis, and the Z-axis.

4. The method according to claim 1, characterized in that, The step of converting the motion data, the track boundary pixel coordinates, the track centerline pixel coordinates, the track contact area marker pixel coordinates, and the track distance data to the body coordinate system to generate initial track geometric parameters includes: Based on the camera intrinsic parameter matrix of the UAV and the motion data, the pixel coordinates of the runway boundary, the pixel coordinates of the runway centerline, and the pixel coordinates of the runway landing zone marker are uniformly calculated to a three-dimensional coordinate point set in the body coordinate system. The three-dimensional coordinate point set is compensated and corrected based on the motion data; Based on the compensated and corrected set of three-dimensional coordinate points, the runway centerline equation and runway plane equation are generated by fitting using a preset random sampling consensus algorithm. Based on the runway centerline equation and the runway plane equation, the initial runway geometric parameters are calculated and generated. The initial runway geometric parameters include the initial runway width, the initial runway heading angle, the initial aircraft distance from the runway extension line XTK, and the initial aircraft distance from the landing area.

5. The method according to claim 4, characterized in that, The compensation and correction of the three-dimensional coordinate point set based on the motion data includes: Extract the timestamp sequence from the motion data and align it with the capture time of the runway image sequence; Based on the aligned timestamp sequence, the UAV attitude angle at the corresponding moment is extracted from the motion data, and a rotation matrix is ​​generated based on the UAV attitude angle; The attitude rotation compensation is performed on the three-dimensional coordinate point set based on the rotation matrix; The acceleration values ​​in the motion data are integrated to obtain the displacement vector, and the displacement vector is used to perform translation compensation on the three-dimensional coordinate point set after attitude rotation compensation, and the compensated and corrected three-dimensional coordinate point set is output.

6. The method according to claim 1, characterized in that, The process involves generating a 3D point cloud using environmental features of the target area scanned by the millimeter-wave radar, matching it with the standard coordinates of runway marker points in a pre-stored airport environment database, and outputting scale constraint factors including: The millimeter-wave radar acquires the original echo signal of the target area; The original echo signal is digitally processed to generate three-dimensional point cloud data containing environmental features; The three-dimensional point cloud data is processed by ground point cloud segmentation and runway marker point identification to obtain the measured spatial coordinates of the runway marker points; The measured spatial location coordinates are matched with the standard coordinates of the target runway markers of the corresponding airport in the pre-stored airport environment database to generate a set of matching point pairs. For the set of matching point pairs, at least one measurement distance value is calculated based on the measurement spatial location coordinates, and a standard distance value corresponding to the measurement distance value is calculated based on the standard coordinates of the target runway marker point; Calculate the ratio of the standard distance value to the measured distance value, and determine the ratio as the scale constraint factor.

7. The method according to claim 1, characterized in that, The optimization of the initial runway geometry parameters based on the scale constraint factor, combined with the runway magnetic heading angle determined based on the runway number, outputs the target runway geometry parameters, including: The initial runway width, initial aircraft distance from the runway extension line XTK, and initial aircraft distance from the landing zone in the initial runway geometry parameters are multiplied by the scale constraint factor to obtain the corrected runway width, corrected aircraft distance from the runway extension line XTK, and corrected aircraft distance from the landing zone. Based on a preset weighted fusion algorithm, the initial runway heading angle and the runway magnetic heading angle in the initial runway geometric parameters are weighted and calculated to generate the target runway heading angle; The corrected runway width, the corrected distance from the aircraft to the runway extension line (XTK), the corrected distance from the aircraft to the landing area, and the target runway heading angle are combined into the target runway geometric parameters and output.

8. A runway estimation system for UAVs in GNSS denied environments, characterized in that, Specifically, it includes: The data acquisition module is used to acquire motion data through the inertial measurement unit on the UAV, capture runway image sequences through the visual sensor on the UAV, and acquire runway distance data through the millimeter-wave radar on the UAV when the UAV's GNSS signal is denied. The runway boundary extraction module is used to input the runway image sequence into a preset convolutional neural network to extract the runway boundary pixel coordinates, the runway centerline pixel coordinates, the runway junction area marker pixel coordinates, and identify the runway number. The body coordinate system construction module is used to establish a body coordinate system based on the airframe structure of the UAV; The initial geometric parameter estimation module is used to convert the motion data, the pixel coordinates of the runway boundary, the pixel coordinates of the runway centerline, the pixel coordinates of the runway contact area marker, and the runway distance data to the body coordinate system to generate initial runway geometric parameters. The scale constraint generation module is used to generate a three-dimensional point cloud by using the environmental features of the target area scanned by the millimeter-wave radar, match the standard coordinates of runway marker points in the pre-stored airport environment database, and output the scale constraint factor. The target geometry parameter generation module is used to optimize the initial runway geometry parameters based on the scale constraint factor, and integrate the runway magnetic heading angle determined based on the runway number to output the target runway geometry parameters.

9. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions. The user interface and the network interface are both used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1-7.

Images