Method, system, device and medium for visual measurement of aircraft rolling distance based on multi-mode fusion

Abstract

The application discloses a method for visual measurement of aircraft running distance based on multi-mode fusion, relates to the technical field of visual measurement, acquires video data, calibrates the video data with a pre-constructed runway digital map, performs geometric ranging and depth correction based on the calibration result, generates two-way distance estimation and its reliability, fuses the two-way distance estimation and its reliability with ADS-B messages after space-time alignment to obtain fusion data, adopts a double-constraint mechanism of ADS-B long-term scale drift and zero-point offset, uses displacement consistency constraint as a main means to eliminate scale drift, uses a low-weight position constraint as an auxiliary means to slowly correct zero-point offset, and avoids error propagation caused by traditional direct correction, and finally uses the fusion data to perform target tracking and event detection to obtain the running distance and confidence level.

Description

Technical Field

[0001] This invention relates to the field of visual measurement technology, specifically to a visual measurement method, system, device, and medium for aircraft takeoff distance based on multi-mode fusion. Background Technology

[0002] Currently, aircraft takeoff distances are primarily obtained through onboard equipment, air traffic control surveillance equipment, or manual analysis. Common technical solutions include: recording aircraft takeoff trajectories using onboard GPS / INS integrated navigation systems and calculating takeoff distances using high-precision positioning data; obtaining aircraft position reports from Automatic Dependent Surveillance-Broadcast (ADS-B) systems and tracking the takeoff process via air traffic control networks; and performing post-flight calculations based on manual interpretation of video recordings or analysis of Quick Access Recorder (QAR) data.

[0003] In recent years, with the development of computer vision technology, visual measurement of aircraft motion using fixed cameras has gradually become a new technical approach. This scheme, through techniques such as target detection, target tracking, camera calibration, and geometric measurement, can achieve continuous estimation of aircraft position and further calculate runway distance. For example, monocular vision-based measurement methods use a single camera for target localization, recovering the target's planar position information from a two-dimensional image; while multi-view vision measurement schemes expand the monitoring coverage by deploying multiple fixed cameras around the runway, aiming to achieve continuous tracking across the entire runway. Furthermore, the development of deep learning has also driven advancements in monocular depth estimation technology. Depth estimation models can be used to recover scene depth information from ordinary RGB images, providing a new technical means for target distance estimation.

[0004] However, the aforementioned existing technical solutions still have several shortcomings in practical applications, specifically including:

[0005] 1. The observation range of a single-camera visual measurement solution is limited, and it is difficult to cover the entire runway when the aircraft's takeoff distance is long. If multiple cameras are used for stitching coverage, problems such as the accumulation of calibration errors, discontinuity of target handover, and inconsistent measurement scales may easily occur between different cameras, affecting the accuracy of continuous measurement of the entire runway.

[0006] 2. Existing visual measurement methods typically rely on one-time offline calibration. However, during long-term operation, the installation posture of a camera may slowly drift due to factors such as temperature changes, wind load vibration, foundation settlement, or maintenance adjustments. This causes the calibration parameters to gradually become invalid, and measurement errors accumulate continuously, making it difficult to meet the requirements for long-term unattended operation.

[0007] 3. Deep learning-based distance measurement methods typically use monocular depth estimation results directly as the distance output. However, depth estimation models are easily affected by factors such as weather changes, lighting conditions, target occlusion, and scene changes, resulting in significant random fluctuations and scale drift in the depth estimation results. Furthermore, existing solutions lack effective reliability evaluation and correction mechanisms, making it difficult to guarantee the stability and absolute accuracy of the measurements.

[0008] 4. Existing methods involving multi-source data fusion typically employ fixed-weight fusion or simple averaging fusion. These methods fail to dynamically adjust based on the reliability of different information sources, such as visual geometric measurements, depth estimation, and ADS-B data, under different operating conditions. They also lack long-term scale constraint mechanisms, making it difficult to balance short-term measurement accuracy with long-term operational stability.

[0009] 5. Some methods rely on manual interpretation of video, resulting in low automation and high labor costs; some methods rely solely on changes in target position or altitude, which can easily lead to misjudgments in complex airport environments such as oblique camera installation, target obstruction, and changes in lighting, thus affecting the accuracy of the taxiing distance measurement results.

[0010] In summary, current technologies lack an aircraft runway distance measurement solution that can simultaneously achieve continuous coverage of the entire runway, long-term calibration stability, adaptive fusion of multi-source information, and accurate detection of critical events. Summary of the Invention

[0011] Based on the problems mentioned above, the purpose of this invention is to provide a visual measurement method, system, device, and medium for aircraft runway distance based on multi-mode fusion. This invention solves the problems of existing methods involving multi-source data fusion, which typically employ fixed-weight fusion or simple averaging fusion. These methods fail to dynamically adjust based on the reliability of different information sources such as visual geometric measurement, depth estimation, and ADS-B data under different operating conditions, and also lack long-term scale constraint mechanisms, making it difficult to balance short-term measurement accuracy with long-term operational stability.

[0012] This invention is achieved through the following technical solution:

[0013] The first aspect of this invention provides a visual measurement method for aircraft takeoff distance based on multi-mode fusion, comprising the following steps:

[0014] Acquire video data, calibrate the video data with a pre-constructed digital map of the runway, and perform geometric ranging and depth correction based on the calibration results to generate two distance estimates and their reliability.

[0015] The two distance estimates and their reliability are fused with the spatiotemporally aligned ADS-B message to obtain fused data.

[0016] The fused data is used for target tracking and event detection to obtain the gliding distance and confidence level.

[0017] In the above technical solution, the calibration process not only establishes the mapping relationship between image pixel coordinates and runway physical coordinates, but also dynamically corrects parameter drift caused by camera changes due to environmental variations. Based on this, the method simultaneously performs geometric ranging based on calibration parameters and depth correction based on a depth estimation model, thereby generating two parallel distance estimates. A confidence index is assigned to each of these estimates to quantify their reliability under the current image conditions.

[0018] The two distance estimates and their reliability are then fused with the spatiotemporally aligned ADS-B messages. During this process, the global location information provided by the ADS-B data is used as a baseline reference. A reliability-driven weighted fusion strategy dynamically adjusts the weight ratios of visual geometry measurement and depth estimation in the fusion process. Simultaneously, the displacement constraints of the ADS-B data over long periods are utilized to perform long-term corrections for scale and zero-point offset in the fusion results, eliminating systematic errors that may accumulate in visual measurements. This results in fused data that combines short-term sensitivity with long-term stability.

[0019] Based on the fused high-precision motion parameters and the target detection results from multiple camera video streams, a smooth handover of target identity and trajectory relay is achieved between different camera fields of view to ensure continuous coverage of the entire runway. On this basis, the aircraft's touchdown / takeoff times are accurately identified, and a complete taxiing trajectory is generated in a unified coordinate system. The final taxiing distance and its corresponding confidence level are calculated, providing a quantitative evaluation basis for the validity of the measurement results.

[0020] In one optional embodiment, the video data is calibrated against a pre-built digital map of the runway, including:

[0021] Extract the pixel coordinates of the runway markings from the video data;

[0022] Determine the area of ​​responsibility of the current camera, and generate a subset constraint of map feature points based on the area of ​​responsibility of the current camera;

[0023] Within the constraints of the subset of map feature points, establish a matching relationship between the pixel coordinates and the pre-constructed digital map of the runway;

[0024] Based on the matching relationship, a homography matrix is ​​established.

[0025] In one optional embodiment, geometric ranging and depth correction are performed based on the calibration results, including:

[0026] The pixel coordinates of each frame of the video data are mapped to runway plane world coordinates using the homography matrix.

[0027] The x-coordinate of the runway plane world coordinates is used as the geometric distance estimate;

[0028] The geometric ranging confidence level is calculated based on the extracted pixel coordinates.

[0029] The video data is input into a monocular depth estimation model to obtain the original depth value. The original depth value is then filtered to obtain the filtered original depth value.

[0030] A correction function is established using the geometric distance estimate and the original depth value. The filtered original depth value is then corrected using the correction function to obtain the depth distance estimate.

[0031] The original depth value is used to calculate the depth confidence level.

[0032] In an optional embodiment, the two distance estimates and their reliability are fused with the spatiotemporally aligned ADS-B message, including:

[0033] The activation decision is based on the credibility of the two-way distance estimation, and activation weights are generated.

[0034] Calculate the credibility of ADS-B messages after spatiotemporal alignment;

[0035] Based on the activation weight and the credibility of the ADS-B message, the two distance estimates and the spatiotemporally aligned ADS-B message are fused to generate a weighted fused distance.

[0036] The weighted fused distance is used as the observation value and input into the Kalman filter to obtain the smoothed distance estimate, velocity estimate and acceleration estimate.

[0037] In an optional embodiment, the smoothed distance estimation is used to constrain the long-term scale drift and zero-point offset of the two distance estimates and their reliability; wherein, the long-term scale drift and zero-point offset constraints include:

[0038] Calculate the ADS-B displacement and the fusion displacement of the weighted fusion distance of the ADS-B message within the time window;

[0039] A global optimization objective function is constructed to optimize the ADS-B displacement and the fused displacement;

[0040] Calculate the ratio of the optimized ADS-B displacement to the fused displacement within the period, and calculate the scale correction factor based on the ratio.

[0041] The scale correction factor is updated, and the updated scale correction factor is used to correct the two-way distance estimates. The corrected two-way distance estimates are then fed back into the fusion activation decision.

[0042] In one optional embodiment, the fused data is used for target tracking and event detection.

[0043] When the aircraft enters the overlapping field of view of adjacent cameras, the same target is determined based on the first fused data and the second fused data of the adjacent cameras.

[0044] After being identified as the same target, the first fused data and the second fused data are smoothly weighted and fused within the overlapping field of view to obtain the fused distance and form a continuous distance sequence.

[0045] The credibility of tire contact evidence, aircraft attitude evidence, and kinematic evidence is calculated separately, and the credibility of each piece of evidence is weighted and fused to obtain the ground contact probability or ground takeoff probability.

[0046] The ground contact probability or ground lift probability is used to drive a finite state machine to perform state transitions. When the preset duration condition is met, the ground contact event or ground lift event is confirmed, and the ground contact time and position, and the ground lift time and position are recorded.

[0047] Based on the touchdown time and position and the takeoff time and position, combined with the continuous distance sequence, the landing run distance and takeoff run distance are calculated;

[0048] The distances output by each camera within its designated area are stitched together according to timestamps, and overlapping areas are smoothed to obtain the complete trajectory of the entire runway. Then, the complete trajectory is optimized at the back end to output the optimized distance sequence.

[0049] In one optional embodiment, the credibility of tire contact evidence, aircraft attitude evidence, and kinematic evidence is calculated separately, including:

[0050] The landing gear region of interest is extracted from the bottom of the target detection bounding box and image segmentation is performed. The tire foreground pixel coverage and the mean gray-level difference between consecutive frames in this region are calculated as motion response. The tire contact confidence is obtained by combining the relationship between higher coverage and lower motion response.

[0051] The credibility of the aircraft attitude evidence was obtained by estimating the pitch angle and its inter-frame change rate through target detection and contour fitting.

[0052] The velocity and acceleration estimates in the fused data are calculated to determine the credibility of the kinematic evidence.

[0053] A second aspect of the present invention provides a visual measurement system for aircraft takeoff distance based on multi-mode fusion, comprising:

[0054] The visual perception and calibration module is used to acquire video data, calibrate the video data with a pre-constructed digital map of the runway, and perform geometric ranging and depth correction based on the calibration results to generate two distance estimates and their reliability.

[0055] The multi-source data fusion and calibration module is used to fuse the two distance estimates and their reliability with the spatiotemporally aligned ADS-B message to obtain fused data.

[0056] The target tracking and event detection module is used to perform target tracking and event detection using the fused data, and to obtain the gliding distance and confidence level.

[0057] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a visual measurement method for aircraft takeoff distance based on multi-mode fusion.

[0058] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a visual measurement method for aircraft takeoff distance based on multi-mode fusion.

[0059] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0060] 1. A three-layer collaborative measurement architecture of "geometric measurement - depth verification - global constraint" is proposed. Geometric measurement is used as the only primary measurement source, depth estimation is used only for verification and compensation, and ADS-B is used only for long-term constraints, forming a master-slave collaborative relationship with separated responsibilities rather than a traditional parallel and integrated relationship.

[0061] 2. A method for automatic online calibration of runways with spatial prior constraints is proposed. This method avoids mismatches by limiting each camera to searching for corner points of the calibrated lines only within its own area of ​​responsibility, and continuously performs online self-calibration by utilizing the runway idle window.

[0062] 3. A dual constraint mechanism for long-term scale drift and zero-point offset in ADS-B is proposed. The displacement consistency constraint is used as the main means to eliminate scale drift, and the low-weight position constraint is used as an auxiliary means to slowly correct the zero-point offset, thus avoiding the error propagation caused by traditional direct correction.

[0063] 4. A credibility-driven multi-source dynamic fusion method is proposed, which evaluates each data source in real time through a unified credibility evaluation system, so that the fusion weight changes dynamically with credibility rather than using fixed weights.

[0064] 5. A cross-camera continuous measurement and trajectory seamless relay mechanism is proposed, which comprehensively utilizes appearance features, motion consistency and ranging continuity to confirm the target identity, and uses an S-shaped curve to smoothly transition to achieve trajectory connection in the overlapping area, while using the runway centerline direction constraint to suppress lateral jumps. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0066] Figure 1 A schematic diagram of the architecture for visual measurement of aircraft takeoff distance provided in an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of camera deployment geometry parameters provided in an embodiment of the present invention;

[0068] Figure 3 A flowchart illustrating the visual measurement method for aircraft takeoff distance based on multi-mode fusion provided in an embodiment of the present invention;

[0069] Figure 4 This is a schematic diagram of the visual perception and calibration processing flow provided in an embodiment of the present invention;

[0070] Figure 5 This is a schematic diagram of the multi-source data fusion and calibration processing flow provided in an embodiment of the present invention;

[0071] Figure 6 This is a schematic diagram of the target tracking and event detection processing procedure provided in an embodiment of the present invention;

[0072] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0074] Example 1

[0075] Visual measurement of aircraft takeoff distance, such as Figure 1 As shown, cameras are arranged at equal intervals on the same side of the runway, with a spacing of D between adjacent cameras. Each camera is fixed at the height of the pole. The lens is mounted diagonally forward along the runway on the pole, with the optical axis forming a horizontal angle with the runway centerline. Ensure that the field of view covers the runway centerline and at least one sideline, with an overlap rate of at least P between adjacent cameras. The deployment geometry is as follows: Figure 2 As shown, each camera is assigned a responsible area, the first... Camera #1 is responsible for the area behind the runway entrance. to The spatial range is defined as a meter interval. This spatial constraint will serve as prior information, playing a crucial role in automatic calibration, ranging verification, and cross-camera fusion. (Based on runway length) Taking width W as an example, the total number of cameras required on one side The camera is used to capture images containing aircraft runway markings, which are then used for subsequent visual perception and calibration.

[0076] ADS-B receivers are deployed near airport control towers or runways, with antennas erected to ensure line-of-sight coverage of the entire runway. Decoded ADS-B messages are sent to the central processing server via Ethernet. All devices are synchronized using PTP or GPS clocks to ensure time alignment of multi-source data. The ADS-B receivers receive ADS-B messages, and the spatiotemporally aligned ADS-B messages are then used for multi-source data fusion with visually calibrated images. This fused multi-source data is then used for target tracking and event detection.

[0077] like Figure 3 As shown, the visual measurement method for aircraft takeoff distance based on multi-mode fusion includes the following steps:

[0078] Acquire video data, calibrate the video data with a pre-constructed digital map of the runway, and perform geometric ranging and depth correction based on the calibration results to generate two distance estimates and their reliability.

[0079] The two distance estimates and their reliability are fused with the spatiotemporally aligned ADS-B message to obtain fused data.

[0080] The fused data is used for target tracking and event detection to obtain the gliding distance and confidence level.

[0081] It should be noted that this calibration process not only establishes the mapping relationship between image pixel coordinates and runway physical coordinates, but also dynamically corrects parameter drift caused by camera changes due to environmental variations. Based on this, the method simultaneously performs geometric ranging based on calibration parameters and depth correction based on a depth estimation model, thereby generating two parallel distance estimates. A confidence index is assigned to each of these estimates to quantify their reliability under the current image conditions.

[0082] The two distance estimates and their reliability are then fused with the spatiotemporally aligned ADS-B messages. During this process, the global location information provided by the ADS-B data is used as a baseline reference. A reliability-driven weighted fusion strategy dynamically adjusts the weight ratios of visual geometry measurement and depth estimation in the fusion process. Simultaneously, the displacement constraints of the ADS-B data over long periods are utilized to perform long-term corrections for scale and zero-point offset in the fusion results, eliminating systematic errors that may accumulate in visual measurements. This results in fused data that combines short-term sensitivity with long-term stability.

[0083] Based on the fused high-precision motion parameters and the target detection results from multiple camera video streams, a smooth handover of target identity and trajectory relay is achieved between different camera fields of view to ensure continuous coverage of the entire runway. On this basis, the aircraft's touchdown / takeoff times are accurately identified, and a complete taxiing trajectory is generated in a unified coordinate system. The final taxiing distance and its corresponding confidence level are calculated, providing a quantitative evaluation basis for the validity of the measurement results.

[0084] In one optional embodiment, the video data is calibrated against a pre-built digital map of the runway, including:

[0085] Extract the pixel coordinates of the runway markings from the video data;

[0086] Determine the area of ​​responsibility of the current camera, and generate a subset constraint of map feature points based on the area of ​​responsibility of the current camera;

[0087] Within the constraints of the subset of map feature points, establish a matching relationship between the pixel coordinates and the pre-constructed digital map of the runway;

[0088] Based on the matching relationship, a homography matrix is ​​established.

[0089] It should be noted that the pre-built digital runway map is constructed from airport design CAD drawings or through on-site measurements using differential GPS / total station. When CAD drawings exist, the world coordinate system coordinates of all runway marking corner points on the CAD drawings are recorded. This includes the endpoints of the dashed centerline, the endpoints of the sidelines, the corner points of the touchdown zone markers, and the corner points of the distance markers. The origin is the center of the runway entrance. The X-axis is defined along the runway direction, the Y-axis is perpendicular to the runway direction, and the runway plane is the Z=0 plane. The dashed lines of the runway centerline are arranged according to standard geometric relationships, which include: the length of the solid line segment... ,interval The two sidelines extend along the X direction, with a spacing equal to the runway width W. These corner coordinates constitute the runway marking corner point database, denoted as set: , where M is the total number of corner points.

[0090] When CAD drawings are unavailable, differential GPS or a total station can be used to measure the coordinates of key marking corner points on-site.

[0091] The video data is then compared with a pre-built digital map of the runway, such as... Figure 4 As shown, the pixel coordinates of runway markings are extracted from video data. Specifically, color thresholding, line detection, and sub-pixel corner point extraction are performed sequentially on the image in the video data to obtain the pixel coordinates of the marking corner points. .

[0092] First, determine the area of ​​responsibility of the current camera, and let the first... Camera #1 is responsible for the area behind the runway entrance. Within a meter interval, the range of the pre-constructed runway digital map is limited to the area currently covered by the camera. The subset of map feature points used for matching is constrained as follows:

[0093] ;

[0094] Obtain the pixel coordinates of the datum corner points from the video data image. Construct as a collection In the feature point subset constraint of the graph Within the scope, establish The matching relationship is defined as follows: k is the index of the corner point extracted from the current camera image, used to traverse all extracted corner points. Let the i-th camera extract K tracing corner points in the current frame image, then k = 1, 2, ..., K. This represents the pixel coordinates of the k-th corner point, and correspondingly, It is a spatially constrained subset The world coordinates of the corner point that matches this pixel.

[0095] This spatial prior fundamentally eliminates cross-regional mismatches and significantly improves the robustness of automatic calibration in scenarios with dense markings or similar features.

[0096] When the runway plane constraint Z=0, the world coordinates to pixel coordinates The projection relationship degenerates into a planar homography transformation. At this point, the matching relationship is established as a homography matrix. :

[0097] ;

[0098] in, The scaling factor is used. The homography matrix is ​​solved using at least four pairs of matching points through direct linear transformation or least squares methods. , where h 11 to h 33 The elements in the homography matrix have no independent physical meaning. They are determined by the camera's intrinsic and extrinsic parameters and the runway plane (Z=0). During calibration, they are directly calculated from at least 4 sets of image-runway coordinate corresponding points and used as a whole to complete the perspective transformation from the runway plane to the image.

[0099] Furthermore, during system operation, runway idle windows occur at a frequency Repeat the above calibration process and continue to update. This enables automatic tracking and correction of slow camera drift.

[0100] In one optional embodiment, geometric ranging and depth correction are performed based on the calibration results, including:

[0101] The pixel coordinates of each frame of the video data are mapped to runway plane world coordinates using the homography matrix.

[0102] The x-coordinate of the runway plane world coordinates is used as the geometric distance estimate;

[0103] The geometric ranging confidence level is calculated based on the extracted pixel coordinates.

[0104] The video data is input into a monocular depth estimation model to obtain the original depth value. The original depth value is then filtered to obtain the filtered original depth value.

[0105] A correction function is established using the geometric distance estimate and the original depth value. The filtered original depth value is then corrected using the correction function to obtain the depth distance estimate.

[0106] The original depth value is used to calculate the depth confidence level.

[0107] It should be noted that this embodiment uses two channels to achieve geometric ranging and depth correction respectively. Channel A is used for geometric ranging, which serves as the main measurement method and has the highest measurement accuracy when the runway markings are clearly visible. Channel B is used for depth ranging.

[0108] Specifically, in channel A, for each frame of the video data, the pixel coordinates of each frame (i.e., the pixel coordinates of the aircraft target touchdown point) are determined using the homography matrix maintained in practice. Mapped to world coordinates on the runway plane:

[0109] ;

[0110] Solving the above equation yields the world coordinates of the runway plane. The x-coordinate of the runway plane world coordinates is used as the geometric distance estimate: .

[0111] After obtaining the geometric distance estimate, it is necessary to calculate the reliability of the geometric distance measurement. The number of corner points extracted depends on the quality of the current frame's corner point extraction. Reprojection error As a basis for evaluation.

[0112] The calculation of reprojection error involves first backprojecting the world coordinates to the image coordinates using the homography matrix, and then calculating the average Euclidean distance between the backprojected image coordinates and the pixel coordinates of all matching current frame datum corner points. The calculation is as follows:

[0113] ;

[0114] in, World coordinates through Mapped pixel coordinates These are the actual extracted corner pixel coordinates.

[0115] The number of extracted corner points is compared with the number of corner points where geometric ranging failed. Compare:

[0116] ;

[0117] in, To meet the corner threshold, This is the acceptable reprojection error threshold. When the number of corner points is too small, geometric ranging is unreliable. Otherwise, the reliability increases with the number of corner points and decreases with the increase of reprojection error.

[0118] In channel B, video data is input into a monocular depth estimation model. In this embodiment, the monocular depth estimation model is the existing Video-Depth-Anything model or its equivalent model, generating a metric depth map for each frame. Locate the pixel coordinates of the aircraft target in the depth map. Substitute the target pixel coordinates of the aircraft into the depth map to determine the original depth value. .

[0119] Using camera intrinsics, pixel coordinates and depth values ​​are converted into 3D coordinates in the camera coordinate system through a pinhole camera model:

[0120] ;

[0121] Among them, camera intrinsic parameters include focal length. , Main point , .

[0122] Then, by combining the camera's extrinsic parameters, the camera coordinate system is transformed into the world coordinate system:

[0123] ;

[0124] Among them, the external components of the camera include a rotating matrix. Translation vector .

[0125] Pick The axial component represents the longitudinal distance of the depth estimation channel: .

[0126] The depth estimation output is subjected to sliding window mid-value filtering to obtain the filtered original depth value. .

[0127] Using geometric distance estimation and original depth value Establish a correction function; specifically, during the runway idle window, synchronously acquire geometric distance estimates for the same aircraft target via channel A. The raw depth value output by the channel B depth estimation model Since the runway markings are clearly visible at this time, and the ranging accuracy of channel A is the highest, the geometric distance estimate is used as the reference true value. Multiple sets of paired data from the same time and the same target are used to form a calibration sample set. .

[0128] On this sample set, a regression method is used to fit the correction function. In this embodiment, quadratic polynomial regression is used for fitting:

[0129] ;

[0130] in, The coefficients are determined by minimizing the mean square error between the correction value and the reference value:

[0131] ;

[0132] After fitting, the correction function is used. The filtered original depth values ​​are corrected to obtain the depth distance estimate. The correction parameters are continuously updated with new idle window data to adapt to slow changes in the environment and camera status.

[0133] The usage strategy for channel B prioritizes verification, followed by compensation. This applies to geometric ranging reliability. Below a first threshold (e.g., 0.4) and with high depth estimation confidence. When the value is above the second threshold (e.g., 0.6), the depth estimation is activated to participate in the fusion; if both channels are unreliable, the current state is maintained or the filtered prediction value is output.

[0134] Furthermore, this embodiment does not rely on the confidence level output by the depth estimation model, but instead constructs its own evaluation mechanism to calculate the depth confidence level of the original depth value.

[0135] Specifically, for the first The frame evaluates the target location from three dimensions: temporal stability, edge consistency, and local continuity. The depth of credibility.

[0136] For temporal stability, it calculates the current frame and the previous frame. The rate of change of the depth value at the target location, taken as the normalized value of the mean absolute deviation:

[0137] ;

[0138] in, To prevent division by zero for small constants, the timing stability score is... .

[0139] For local continuity, in the neighborhood of the target location Standard deviation of internal calculated depth values:

[0140] ;

[0141] in, The mean depth of the neighborhood is given by the local continuity score. .

[0142] For edge consistency, the degree of matching between the depth edges of the target region and the semantic edges of the image is detected, and the normalized cross-correlation between the depth gradient and the image gradient is used as a measure:

[0143] ;

[0144] in, For depth gradient, The image grayscale gradient has an edge consistency score of [value missing]. .

[0145] The scores from the three dimensions mentioned above are combined into a comprehensive evaluation score:

[0146] ;

[0147] in, , .

[0148] The confidence level of the depth estimation for this frame is set to . .

[0149] It should be noted that the above The three parameters are attenuation coefficients (sensitivity coefficients), used to control the sensitivity of mapping each dimension indicator to the credibility score. For example: The larger the change rate, the better. This will lead to a higher time series stability score. The faster the descent, the more sensitive that dimension is to depth fluctuations. Similarly, adjust the contribution of local continuity and edge consistency to the final credibility. These three parameters can be calibrated based on the actual scenario or set empirically.

[0150] In an optional embodiment, the two distance estimates and their reliability are fused with the spatiotemporally aligned ADS-B message, including:

[0151] The activation decision is based on the credibility of the two-way distance estimation, and activation weights are generated.

[0152] Calculate the credibility of ADS-B messages after spatiotemporal alignment;

[0153] Based on the activation weight and the credibility of the ADS-B message, the two distance estimates and the spatiotemporally aligned ADS-B message are fused to generate a weighted fused distance.

[0154] The weighted fused distance is used as the observation value and input into the Kalman filter to obtain the smoothed distance estimate, velocity estimate and acceleration estimate.

[0155] like Figure 5 As shown, the purpose of this step is to perform spatiotemporal alignment of multi-source data, credibility-driven weighted fusion, and long-term scale and zero-point offset correction based on ADS-B displacement constraints, ultimately outputting a highly reliable fusion distance.

[0156] Specifically, the ADS-B ground receiver receives ADS-B messages broadcast by the aircraft in real time and decodes them to obtain the aircraft's latitude and longitude in the WGS-84 coordinate system. Barometric altitude or geometrical altitude Ground speed, heading, and message transmission time Convert the WGS-84 latitude and longitude to a local Cartesian coordinate system with the runway threshold as the origin, to obtain the three-dimensional position in the runway coordinate system. Let the WGS-84 coordinates of the runway threshold be... The azimuth of the runway centerline is Then the longitudinal distance of the aircraft in the local coordinate system is:

[0157] ;

[0158] in The radius is the Earth's radius (WGS-84 ellipsoid approximation), and the angles are in radians. For simplification, a plane approximation can also be used for direct calculation.

[0159] There is a transmission delay from when the message is sent from the aircraft to when it is received and decoded on the ground. ,in, This is the receiving time.

[0160] Rewind the position to the time of transmission:

[0161] ;

[0162] in, This is the projection of the ground speed component in the message onto the runway direction.

[0163] ADS-B data update frequency, classic value is approximately The visual system uses frame rate Run. Upscale the ADS-B data to the visual frame rate using Hermite interpolation (or its equivalent algorithm) to obtain a timestamp aligned with each frame's image timestamp. All devices are synchronized with a unified time base via PTP or GPS clocks.

[0164] After obtaining the spatiotemporally aligned ADS-B message, its reliability also needs to be calculated. In this embodiment, the reliability of the ADS-B message includes the signal strength index. Motion model consistency and delay indicators .

[0165] The signal strength index is obtained by normalizing the signal strength or signal-to-noise ratio output by the ADS-B receiver to [0,1].

[0166] Compare the current ADS-B position with the Kalman filter's predicted position. The consistency of the motion model is calculated through comparison.

[0167] ;

[0168] in, This is the allowable deviation threshold.

[0169] Through transmission delay and maximum allowable delay Calculate latency metrics:

[0170] ;

[0171] Among them, transmission delay The smaller the value, the higher the credibility.

[0172] Comprehensive signal strength index Motion model consistency and delay indicators The reliability of the ADS-B message can be obtained as follows:

[0173] ;

[0174] in, This is the upper limit constraint factor for ADS-B confidence, with a typical value of 0.7. It reflects the positioning of ADS-B as a global constraint rather than the true location value. Its confidence limit is lower than that of geometric ranging and depth estimation, avoiding the direct use of the absolute position of ADS-B as the fusion benchmark.

[0175] By default, the depth estimation channel only performs verification and does not participate in distance fusion. It only participates in fusion when activation conditions are met. Therefore, it is necessary to first determine the fusion activation based on the credibility of the two distance estimates and generate activation weights. The fusion activation determination is as follows:

[0176] ;

[0177] Depth estimation is activated for fusion only if the reliability of geometric ranging is less than the first threshold and the reliability of depth estimation is greater than the second threshold, and the reliability of depth estimation is used as the activation weight; otherwise, it is set to 0.

[0178] It should be noted that if both visual channels are unreliable ( and If the current Kalman filter prediction state is not maintained, no fusion update is performed, and the process waits for any channel to become reliable.

[0179] Based on activation weights and ADS-B message reliability, data fusion is performed on the two distance estimates and the spatiotemporally aligned ADS-B messages to generate a weighted fused distance. The fusion process is as follows:

[0180] ;

[0181] The weighted fusion distance is used as the observation input to the Kalman filter, and the state vector of the Kalman filter is taken as... (Longitudinal distance, velocity, acceleration), the state transition equation adopts a uniform acceleration model:

[0182] ;

[0183] The observation equation is ,in and These represent process noise and observation noise, respectively. The filtered output is a smoothed distance estimate. And velocity estimation and acceleration estimation.

[0184] In an optional embodiment, the smoothed distance estimation is used to constrain the long-term scale drift and zero-point offset of the two distance estimates and their reliability; wherein, the long-term scale drift and zero-point offset constraints include:

[0185] Calculate the ADS-B displacement and the fusion displacement of the weighted fusion distance of the ADS-B message within the time window;

[0186] A global optimization objective function is constructed to optimize the ADS-B displacement and the fused displacement;

[0187] Calculate the ratio of the optimized ADS-B displacement to the fused displacement within the period, and calculate the scale correction factor based on the ratio.

[0188] The scale correction factor is updated, and the updated scale correction factor is used to correct the two-way distance estimates. The corrected two-way distance estimates are then fed back into the fusion activation decision.

[0189] It should be noted that in this embodiment, the role of ADS-B information is positioned as long-term constraint rather than instantaneous positioning. Specifically, the accuracy of the absolute position of a single ADS-B frame is limited (typically between meters and tens of meters); however, over a shorter time span of tens of seconds to minutes, the relative accuracy of its cumulative displacement value is significantly improved. Utilizing this characteristic, effective constraints can be imposed on the long-term scale and zero-point drift caused by the slow drift of calibration parameters in visual odometry, thereby improving overall stability.

[0190] Specifically, the ADS-B offset of the ADS-B message within the time window is calculated as follows:

[0191] ;

[0192] And, the fusion displacement of the weighted fusion distance:

[0193] ;

[0194] It should be noted that, That is, the real-time output of the Kalman filter at the current moment. Both refer to the same physical quantity. Used here. This is to emphasize its role in displacement calculation and global optimization as a distance estimate for visual systems.

[0195] During the trajectory generation phase (or backtracking optimization after the taxiing is completed), a global optimization objective function is constructed to optimize the ADS-B displacement and the fused displacement:

[0196] ;

[0197] The first term is the weight of the displacement constraint term. Larger (e.g.) The first term constrains the scale drift of the vision system. The difference operation cancels out the effect of the fixed zero-point offset; this term primarily corrects scale errors. The second term is the weight of the position constraint term. Smaller (e.g.) The third term, which serves as a long-term soft constraint to suppress cumulative zero-point offset, does not participate in single-frame correction; the fourth term is the weight of the second-order smoothing term. (For example The acceleration rate of the constrained trajectory is used to ensure the temporal physical smoothness. The sample weights are determined only when the admission criteria are met. ,otherwise .

[0198] Furthermore, the admission criteria are: ADS-B location integrity rate. (For example Geometric distance measurement reprojection error (e.g., 2 pixels) and the number of corner points (e.g., 20) Depth estimation credibility score (e.g., 0.7) When the aircraft is in a constant speed or constant acceleration phase, its acceleration is estimated by the filter. (For example ).

[0199] Once a certain number of valid samples are accumulated or at regular intervals, the scale correction factor for geometric ranging is calculated using the displacement constraint term, i.e., the ratio of the optimized ADS-B displacement to the fused displacement within the period is calculated:

[0200] ;

[0201] The moving average of the effective samples is used as the scaling correction factor.

[0202] ;

[0203] in, This is the set of valid sample indexes within the current update cycle.

[0204] The scaling correction parameters are updated using an exponential moving average:

[0205] ;

[0206] in, To update the rate coefficient, a value of 0.1 can be used to ensure that the parameter smoothly tracks slow drifts and does not respond to instantaneous fluctuations.

[0207] Meanwhile, the parameters of the depth estimation correction model ( It also utilizes the newly acquired data through the least squares method. The data pairs are refitted and updated using the same exponential moving average method.

[0208] Geometric distance estimation using the updated scale correction factor Make corrections: ; using the revised Instead of the original geometric distance estimation Entering weighted fusion, this correction results in visual displacement. Gradually approaching the ADS-B displacement This constrains the long-term scale drift of the visual system.

[0209] Directly used for correcting subsequent frames To obtain more accurate The two corrected inputs are then weighted, fused, and Kalman filtered to produce subsequent frames with higher accuracy. This forms a closed loop of "output → accumulation → parameter update → input correction → improvement of subsequent output".

[0210] In one optional embodiment, the fused data is used for target tracking and event detection.

[0211] When the aircraft enters the overlapping field of view of adjacent cameras, the same target is determined based on the first fused data and the second fused data of the adjacent cameras.

[0212] After being identified as the same target, the first fused data and the second fused data are smoothly weighted and fused within the overlapping field of view to obtain the fused distance and form a continuous distance sequence.

[0213] The credibility of tire contact evidence, aircraft attitude evidence, and kinematic evidence is calculated separately, and the credibility of each piece of evidence is weighted and fused to obtain the ground contact probability or ground takeoff probability.

[0214] The ground contact probability or ground lift probability is used to drive a finite state machine to perform state transitions. When the preset duration condition is met, the ground contact event or ground lift event is confirmed, and the ground contact time and position, and the ground lift time and position are recorded.

[0215] Based on the touchdown time and position and the takeoff time and position, combined with the continuous distance sequence, the landing run distance and takeoff run distance are calculated;

[0216] The distances output by each camera within its designated area are stitched together according to timestamps, and overlapping areas are smoothed to obtain the complete trajectory of the entire runway. Then, the complete trajectory is optimized at the back end to output the optimized distance sequence.

[0217] like Figure 6 As shown, target determination is performed first. When the aircraft enters the overlapping field of view of adjacent cameras A and B, the first fused data and the second fused data are obtained from A and B simultaneously.

[0218] Calculate the first fused data Recorded as Second fusion data Recorded as The degree of difference within the overlapping region determines the ranging continuity. :

[0219] ;

[0220] in, This represents the maximum permissible distance difference.

[0221] Simultaneously, the appearance feature vectors of A and B are extracted respectively. , Measured by cosine similarity or Euclidean distance and The similarity is normalized to [0,1] as the appearance feature similarity.

[0222] Extract the target motion direction and velocity magnitude of A and B, and calculate their matching degree as the motion vector consistency. .

[0223] The above three factors are weighted and combined as follows:

[0224] ;

[0225] in, , , For the corresponding weights.

[0226] Within the overlapping field of view, the first and second fused data are weighted and fused using an S-shaped curve to obtain the fusion distance:

[0227] ;

[0228] in, The weighting function is S-shaped, and the normalized position of the aircraft within the overlapping region is... ( 1 represents the starting point of the overlapping area closest to A, and 1 represents the ending point closest to B.

[0229] Furthermore, the S-shaped weighting function:

[0230] ;

[0231] in, For the steepness coefficient of transition (e.g.) This allows the weights to switch smoothly near the midpoint of the overlapping area.

[0232] Furthermore, when there is a deviation of several meters between the geometric distance measurements of A and B, the depth variation trend of the depth estimation channel can be used as an auxiliary criterion: comparison For continuity in A and B, prioritize the geometric ranging on the side with smoother depth changes.

[0233] Furthermore, the target motion is constrained laterally by utilizing the runway centerline direction, thus limiting... The inter-frame jump amplitude of coordinates suppresses lateral anomalies caused by detection box jitter.

[0234] Accurate detection of the contact / takeoff moment is a prerequisite for calculating the skid distance. In this embodiment, the camera is installed obliquely along the runway (the angle between the optical axis and the runway centerline). Traditional single tire contact detection methods based on a frontal and side view are easily affected by factors such as differences in camera models, obstructions, shadows, and tire marks on the runway under oblique viewing conditions.

[0235] This embodiment employs a decision-making architecture of "multi-evidence fusion + finite state machine," integrating four types of evidence: tire contact, aircraft attitude, kinematics, and landing gear compression. The credibility of each piece of evidence is calculated separately, and then weighted and fused. The resulting contact-to-ground or takeoff probability is used as the event triggering criterion. This architecture aligns with the multi-source fusion design philosophy, ensuring robustness under oblique-look-ahead deployment conditions.

[0236] The state transition of a finite state machine is driven by the probability of touching the ground or the probability of leaving the ground. The states include: in the hole, bottoming candidate, ground sliding, and leaving the ground candidate.

[0237] The state transition includes: airborne to ground candidate. (e.g., 0.7) and continue Frames (e.g.) If the state transitions to grounding candidate, then the state changes to grounding candidate.

[0238] Touchdown candidate → Ground gliding: continuous Intra-frame All remain Confirm the ground contact event and mark the time of ground contact. (Taking the first frame of the candidate period) and its position ;

[0239] Touchdown candidate → Airborne: If within the candidate period Falling back to The following and continuing If a frame is identified as a false alarm, the status is returned to the air (anti-shake mechanism).

[0240] Ground gliding → Candidate for takeoff: (e.g., $0.7) and continue Frame, transition to off-ground candidate;

[0241] Candidate for takeoff → Airborne: Confirm takeoff event and mark takeoff time. (The moment when the contact condition is met in the last frame of the candidate period) and its position ;

[0242] Candidate for takeoff → Ground gliding: During the candidate period It falls back to its gliding state.

[0243] Probability of being off the ground Calculation and Similar, but the credibility of each piece of evidence is reversed ( The lower, The more obvious the wheel lift, The more significant the change in acceleration, hour, The higher (the higher).

[0244] Independent detection of left and right landing gear: Due to the angled installation of the camera, the left and right main landing gears are positioned differently in the image. and Extract and calculate separately Only when evidence of tire contact on both sides supports a ground-lift condition, middle The value of the direction of the vehicle leaving the ground is taken to avoid misjudgment caused by lifting the wheel on one side only.

[0245] When the preset continuous conditions are met, confirm the ground contact event or the ground lift event, and record the ground contact time and location, and the ground lift time and location.

[0246] Based on the touchdown time and location, and the takeoff time and location, combined with a continuous distance sequence, calculate the landing distance and takeoff distance:

[0247] ;

[0248] in, This represents the longitudinal distance at the moment the gliding begins.

[0249] Estimated by speed Determine, when from The number of nearby plants began to increase continuously and exceeded the threshold. (For example When the condition is first met, take the time when the condition is first met. for .

[0250] The distance output by each camera within its designated area. By stitching the data together according to timestamps and smoothing out overlapping areas, the complete trajectory of the entire runway is obtained.

[0251] Within the overlapping region, a weighted fusion is performed using an S-shaped weighting function. As a transition value, it forms a continuous distance sequence. .

[0252] Using a global optimization objective function based on ADS-B displacement constraints and low-weight position constraints, the stitched complete trajectory is optimized at the back end, and the optimized distance sequence is output. .

[0253] The average confidence level of each data source within the entire gliding interval is calculated, using the geometric distance average confidence level. and depth estimation average confidence Assess the confidence level of the gliding distance.

[0254] High confidence level: and ( For example, 0.8);

[0255] Medium confidence level: at least one path And the rest ( For example, 0.5);

[0256] All other cases are low confidence levels.

[0257] The confidence level and the running distance are output together to assess the reliability of the data.

[0258] In one optional embodiment, the credibility of tire contact evidence, aircraft attitude evidence, and kinematic evidence is calculated separately, including:

[0259] The landing gear region of interest is extracted from the bottom of the target detection bounding box and image segmentation is performed. The tire foreground pixel coverage and the mean gray-level difference between consecutive frames in this region are calculated as motion response. The tire contact confidence is obtained by combining the relationship between higher coverage and lower motion response.

[0260] The credibility of the aircraft attitude evidence was obtained by estimating the pitch angle and its inter-frame change rate through target detection and contour fitting.

[0261] The velocity and acceleration estimates in the fused data are calculated to determine the credibility of the kinematic evidence.

[0262] It should be noted that the tire contact evidence includes: extracting the landing gear region of interest from the bottom of the target detection bounding box and performing image segmentation processing; calculating the tire foreground pixel coverage and the mean gray-level difference between consecutive frames in this region as the motion response; and obtaining the tire contact confidence by combining the relationship between the higher the coverage and the smaller the motion response.

[0263] Specifically, the landing gear region of interest is extracted from the bottom region of the target detection bounding box, and adaptive thresholding and morphological processing are performed to obtain a binary image of the tire foreground. (Adaptive thresholding (such as the Otsu algorithm) separates the dark tire pixels from the light-colored runway background, resulting in a binary image where the tire is the foreground (white) and the runway is the background (black). Morphological opening operations (erosion followed by dilation) remove isolated noise and small breaks, and the remaining white pixels are the tire foreground pixels.)

[0264] Based on this, calculate the tire coverage rate. and inter-frame motion response The tire contact reliability is obtained by combining the two. .

[0265] Among them, the calculation of tire coverage :

[0266] ;

[0267] in, for Number of foreground pixels of the tire in the bottom edge area This represents the total number of pixels in the region.

[0268] The higher the tire, the closer it is to the ground.

[0269] Through The inter-frame motion response is obtained by calculating the mean inter-frame difference between two consecutive frames. :

[0270] ;

[0271] in, This represents the pixel grayscale value.

[0272] The landing gear area is relatively stable during contact. Smaller size; the landing gear moves as a whole after takeoff. Increase.

[0273] Combining the two:

[0274] ;

[0275] in, This is the coverage threshold (e.g., 0.3). These are the normalized parameters for the motion response.

[0276] The higher the value, the greater the likelihood of contact.

[0277] The reliability of aircraft attitude evidence is obtained by estimating the pitch angle and its inter-frame rate of change through target detection and contour fitting. After locating the aircraft using a target detection network, the fuselage centerline is extracted through keypoint detection or contour fitting to estimate the aircraft's pitch angle. (Trunk nose up is positive). Calculate the inter-frame pitch angle change rate:

[0278] ;

[0279] Attitude reliability is determined based on the inter-frame pitch angle change rate. Specifically, during the landing and touchdown phase, the pitch angle recovers from the horizontal orientation at the moment of touchdown. Increase. When (Pitch angle change rate threshold, for example) Credibility increases during takeoff. During the takeoff phase, the wheel-lifting action... And it continues to increase, exceeding the threshold. (For example The reliability of the attitude increases when the vehicle is off the ground.

[0280] Therefore, attitude reliability The calculation is as follows:

[0281] .

[0282] Velocity estimation in fused data and acceleration estimation Calculations are performed to determine the credibility of the kinematic evidence. Specifically, at the moment of landing touchdown, the aircraft transitions from flight to ground taxiing, and acceleration typically undergoes a sudden backward abrupt change (landing impact deceleration); at the moment of takeoff, the aircraft transitions from ground acceleration taxiing to climb, and the slope of the velocity-time curve changes.

[0283] First, abrupt changes in acceleration are detected; therefore, the kinematic reliability of the landing scenario is calculated first. This represents the longitudinal acceleration output by the Kalman filter in the current frame. The moving average of the acceleration before ground contact. Defined as before the ground contact candidate period Frames (e.g.) Moving average of acceleration estimation (frame):

[0284] ;

[0285] Next, the kinematic reliability of the landing scenario is calculated:

[0286] ;

[0287] in, The landing impact acceleration deviation normalization threshold represents the expected minimum acceleration change amplitude at the moment of ground contact.

[0288] Next, the kinematic reliability of the takeoff scenario, used to detect the rate of change of acceleration, i.e., the change in the slope of the velocity-time curve, is calculated. The first-order difference of the current frame's acceleration (i.e., jerk) reflects the rate of descent of the acceleration itself. During takeoff, the acceleration smoothly decreases from a large positive value, producing a negative jerk. ), while the acceleration during the gliding phase should be at Fluctuations in the vicinity.

[0289]

[0290] in The normalized threshold for takeoff jerk. The larger the value, the more drastic the change in the slope of the velocity-time curve, and the higher the probability of taking off.

[0291] In subsequent integration, Take based on the current flight phase (landing or takeoff) of the system or .

[0292] Furthermore, when the camera resolution is high enough and the pixel size of the landing gear area meets the requirements, evidence of landing gear compression can be added. After the aircraft touches the ground, the landing gear shock absorbers are compressed due to the weight of the fuselage, and the distance between the tires and the bottom of the fuselage decreases; after takeoff, the shock absorbers are released, and the distance is restored.

[0293] exist Extract the landing gear area and detect the pixel distance between the landing gear struts or tire top edge and the fixed reference point on the fuselage. Compression is defined as:

[0294] ;

[0295] in, This is the reference length under no-load conditions (statistical mean under off-ground conditions).

[0296] Compression credibility:

[0297] ;

[0298] in, This is a reference value for the full-load compression. The larger it is, the more certain it is to hit the ground. This indicates that the object is off the ground.

[0299] Introducing landing gear availability factor :

[0300] ;

[0301] when In this case, the evidence is not involved in the fusion, and the system automatically degenerates into the three-evidence mode, without forcibly relying on high-resolution cameras.

[0302] Based on the landing gear area resolution: the pixel size of the landing gear in the image directly affects the accuracy of compression measurement. Let the height of the landing gear area in the image be... (in pixels), when Below the preset threshold (For example, at 15 pixels) the compression measurement is unreliable and is also judged as... ,otherwise .

[0303] The above evidence is then fused together for status determination:

[0304] ;

[0305] in For the fusion weight of each piece of evidence, satisfying .

[0306] Furthermore, online anomaly detection, multi-source arbitration, physical status monitoring, and automatic recovery are incorporated into each of the above-mentioned stages to ensure the reliability and maintainability of the system during long-term operation.

[0307] in,

[0308] A consistency check is performed on the distance estimates between the two paths and the ADS-B messages, defining the pairwise deviations:

[0309] ;

[0310] ;

[0311] ;

[0312] like (Arbitration threshold, for example) If the two data sources are consistent, then they are considered to be consistent; otherwise, they are considered to be misaligned.

[0313] Two consistent paths, one deviant path: Using majority voting, two paths with consistent trust are used to isolate the deviant data source and mark it as suspicious. The credibility of the isolated data source is forcibly multiplied by a decay factor. (e.g., 0.1).

[0314] All three paths are inconsistent: Proceed to deep arbitration. Compare the distances of the three paths with the predicted values ​​from the Kalman filter. (Compare the positional components from the state vector) and select the path with the smallest deviation as the temporary source of trust:

[0315]

[0316] At the same time, anything that causes a sudden change in speed (For example ) or positional bounce ( , For example Data sources containing Kalman predictions are automatically rejected and not included in the arbitration. If a decision still cannot be made after the above filtering, the system outputs the Kalman prediction value. And trigger an alarm.

[0317] A certain data source is continuous If a frame (e.g., 10 frames) is determined to be untrustworthy (i.e., isolated or rejected by arbitration), then the data source is marked as faulty, and its weight... Forced to zero. For the camera, the corresponding... and Carried and All values ​​are reduced to 0, and alarm information (including camera ID, timestamp, and anomaly type) is sent to the maintenance terminal.

[0318] After the fault source is restored, its weights are gradually restored using a linear ramping method to avoid sudden weight restoration causing a jump in distance output. Let the recovery start frame number be... The recovery period is Frame (e.g., 10 frames), then during the recovery period, the th frame( The weights of ) are:

[0319]

[0320] in, This represents the credibility of the data source under normal conditions. The weights will return to normal after the recovery period.

[0321] Each camera base has a built-in tilt sensor that continuously monitors the pole's pitch direction. and roll direction The attitude angle. Let the reference attitude recorded during initial calibration be... The deviation between the current attitude and the baseline is:

[0322] ;

[0323] when (For example If the camera is detected to have undergone slight physical displacement, the system will pause operation of that camera. Output (its) (Forced zeroing) triggers the automatic online calibration process to immediately re-execute. After calibration, the new external parameters homography matrix... renew, Once the settings return to normal, the camera is reintegrated into the fusion process.

[0324] when (For example If the camera is detected as having a serious displacement, the system will mark it as a hardware fault, suspend all its outputs, and send an alarm (including camera ID, timestamp, and tilt angle change) to the maintenance terminal.

[0325] Once the camera is installed, precise measurements of the installation angle and position are not required; it is only necessary to ensure that the field of view roughly covers the runway area. During the initialization phase, the system automatically calculates precise extrinsic parameters through an automated online calibration process.

[0326] If a camera shifts due to an accidental collision, the system will detect this via a tilt sensor or an anomaly arbitration mechanism and automatically trigger recalibration. If recalibration is successful (i.e., a sufficient number of marker corner points are extracted), If the camera returns to normal operation, the weight is... Intra-frame linear ramp-up recovery. If recalibration fails ( (This indicates that the runway markings are completely invisible in the field of view), the system marks the camera as a hardware malfunction and issues an alarm.

[0327] After maintenance personnel arrived and roughly restored the camera to face the runway, the system automatically re-executed the calibration procedure during a runway idle window. Once calibration was successful, the camera's... and exist Intraframe from It linearly recovers to normal values ​​and re-participates in multi-source fusion.

[0328] Example 2

[0329] Embodiment 2 of the present invention provides a visual measurement system for aircraft takeoff distance based on multi-mode fusion, comprising:

[0330] The visual perception and calibration module is used to acquire video data, calibrate the video data with a pre-constructed digital map of the runway, and perform geometric ranging and depth correction based on the calibration results to generate two distance estimates and their reliability.

[0331] The multi-source data fusion and calibration module is used to fuse the two distance estimates and their reliability with the spatiotemporally aligned ADS-B message to obtain fused data.

[0332] The target tracking and event detection module is used to perform target tracking and event detection using the fused data, and to obtain the gliding distance and confidence level.

[0333] Example 3

[0334] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention, as shown below. Figure 7 As shown, the electronic device includes a processor 21, a memory 22, an input device 23, and an output device 24; the number of processors 21 in the computer device can be one or more. Figure 7 Taking a processor 21 as an example; the processor 21, memory 22, input device 23, and output device 24 in an electronic device can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0335] The memory 22, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules. The processor 21 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 22, thereby implementing the multi-mode fusion-based visual measurement method for aircraft takeoff distance in Embodiment 1.

[0336] The memory 22 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 22 may further include memory remotely located relative to the processor 21, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0337] Input device 23 can be used to receive user input such as ID and password. Output device 24 is used to output the network configuration page.

[0338] Example 4

[0339] Embodiment 4 of the present invention also provides a computer-readable storage medium, wherein the computer-executable instructions, when executed by a computer processor, are used to implement the multi-mode fusion-based visual measurement method for aircraft takeoff distance as provided in Embodiment 1.

[0340] The storage medium containing computer-executable instructions provided in the embodiments of the present invention is not limited to the method operation provided in Embodiment 1, but can also perform related operations in the visual measurement method for aircraft takeoff distance based on multi-mode fusion provided in any embodiment of the present invention.

[0341] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A visual measurement method for aircraft takeoff distance based on multi-mode fusion, characterized in that, Includes the following steps: Acquire video data, calibrate the video data with a pre-constructed digital map of the runway, and perform geometric ranging and depth correction based on the calibration results to generate two distance estimates and their reliability. The two distance estimates and their reliability are fused with the spatiotemporally aligned ADS-B message to obtain fused data. The fused data is used for target tracking and event detection to obtain the gliding distance and confidence level.

2. The visual measurement method for aircraft takeoff distance based on multi-mode fusion according to claim 1, characterized in that, The video data is then calibrated against a pre-constructed digital map of the runway, including: Extract the pixel coordinates of the runway markings from the video data; Determine the area of ​​responsibility of the current camera, and generate a subset constraint of map feature points based on the area of ​​responsibility of the current camera; Within the constraints of the subset of map feature points, establish a matching relationship between the pixel coordinates and the pre-constructed digital map of the runway; Based on the matching relationship, a homography matrix is ​​established.

3. The visual measurement method for aircraft takeoff distance based on multi-mode fusion according to claim 2, characterized in that, Based on the calibration results, geometric ranging and depth correction are performed, including: The pixel coordinates of each frame of the video data are mapped to runway plane world coordinates using the homography matrix. The x-coordinate of the runway plane world coordinates is used as the geometric distance estimate; The geometric ranging confidence level is calculated based on the extracted pixel coordinates. The video data is input into a monocular depth estimation model to obtain the original depth value. The original depth value is then filtered to obtain the filtered original depth value. A correction function is established using the geometric distance estimate and the original depth value. The filtered original depth value is then corrected using the correction function to obtain the depth distance estimate. The original depth value is used to calculate the depth confidence level.

4. The visual measurement method for aircraft takeoff distance based on multi-mode fusion according to claim 1, characterized in that, The two distance estimates and their reliability are fused with the spatiotemporally aligned ADS-B message, including: The activation decision is based on the credibility of the two-way distance estimation, and activation weights are generated. Calculate the credibility of ADS-B messages after spatiotemporal alignment; Based on the activation weight and the credibility of the ADS-B message, the two distance estimates and the spatiotemporally aligned ADS-B message are fused to generate a weighted fused distance. The weighted fused distance is used as the observation value and input into the Kalman filter to obtain the smoothed distance estimate, velocity estimate and acceleration estimate.

5. The visual measurement method for aircraft takeoff distance based on multi-mode fusion according to claim 4, characterized in that, The smoothed distance estimation is used to constrain the long-term scale drift and zero-point offset of the two distance estimates and their reliability; wherein, the long-term scale drift and zero-point offset constraints include: Calculate the ADS-B displacement and the fusion displacement of the weighted fusion distance of the ADS-B message within the time window; A global optimization objective function is constructed to optimize the ADS-B displacement and the fused displacement; Calculate the ratio of the optimized ADS-B displacement to the fused displacement within the period, and calculate the scale correction factor based on the ratio. The scale correction factor is updated, and the updated scale correction factor is used to correct the two-way distance estimates. The corrected two-way distance estimates are then fed back into the fusion activation decision.

6. The visual measurement method for aircraft takeoff distance based on multi-mode fusion according to claim 1, characterized in that, Target tracking and event detection are performed using the fused data. When the aircraft enters the overlapping field of view of adjacent cameras, the same target is determined based on the first fused data and the second fused data of the adjacent cameras. After being identified as the same target, the first fused data and the second fused data are smoothly weighted and fused within the overlapping field of view to obtain the fused distance and form a continuous distance sequence. The credibility of tire contact evidence, aircraft attitude evidence, and kinematic evidence is calculated separately, and the credibility of each piece of evidence is weighted and fused to obtain the ground contact probability or ground takeoff probability. The ground contact probability or ground lift probability is used to drive a finite state machine to perform state transitions. When the preset duration condition is met, the ground contact event or ground lift event is confirmed, and the ground contact time and position, and the ground lift time and position are recorded. Based on the touchdown time and position and the takeoff time and position, combined with the continuous distance sequence, the landing run distance and takeoff run distance are calculated; The distances output by each camera within its designated area are stitched together according to timestamps, and overlapping areas are smoothed to obtain the complete trajectory of the entire runway. Then, the complete trajectory is optimized at the back end to output the optimized distance sequence.

7. The visual measurement method for aircraft takeoff distance based on multi-mode fusion according to claim 6, characterized in that, Calculate the credibility of tire contact evidence, aircraft attitude evidence, and kinematic evidence separately, including: The landing gear region of interest is extracted from the bottom of the target detection bounding box and image segmentation is performed. The tire foreground pixel coverage and the mean gray-level difference between consecutive frames in this region are calculated as motion response. The tire contact confidence is obtained by combining the relationship between higher coverage and lower motion response. The credibility of the aircraft attitude evidence was obtained by estimating the pitch angle and its inter-frame change rate through target detection and contour fitting. The velocity and acceleration estimates in the fused data are calculated to determine the credibility of the kinematic evidence.

8. A multi-mode fusion-based visual measurement system for aircraft takeoff distance, used to implement the multi-mode fusion-based visual measurement method for aircraft takeoff distance as described in any one of claims 1 to 7, characterized in that, include: The visual perception and calibration module is used to acquire video data, calibrate the video data with a pre-constructed digital map of the runway, and perform geometric ranging and depth correction based on the calibration results to generate two distance estimates and their reliability. The multi-source data fusion and calibration module is used to fuse the two distance estimates and their reliability with the spatiotemporally aligned ADS-B message to obtain fused data. The target tracking and event detection module is used to perform target tracking and event detection using the fused data, and to obtain the gliding distance and confidence level.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the visual measurement method for aircraft runway distance based on multi-mode fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the visual measurement method for aircraft takeoff distance based on multi-mode fusion as described in any one of claims 1 to 7.

Images