A runway foreign object detection system based on image recognition

Abstract

The application relates to the runway safety detection technical field and discloses a runway foreign matter detection system based on image recognition. The system comprises a runway block division unit, a reference feature library construction unit, a real-time image analysis unit, a feature deviation analysis unit and an alarm decision unit. The system divides the runway into a plurality of independent detection blocks, constructs a reference texture feature set and an optical reflection feature set of each block in a foreign matter-free state, collects real-time image sequences by using a multispectral imaging device, extracts real-time texture and reflection feature vectors, compares the real-time feature vectors with the reference feature set, generates a texture deviation matrix and a reflection deviation matrix, and generates foreign matter positioning coordinates and alarm level instructions based on the joint analysis result. The system realizes fine and multidimensional detection of runway foreign matters, can accurately position the foreign matters and give corresponding alarms, and is suitable for runway safety detection scenes.

Description

Technical Field

[0001] This invention relates to the field of runway safety inspection technology, specifically to a runway foreign object detection system based on image recognition. Background Technology

[0002] Currently, runway debris is one of the most significant factors affecting aviation safety. These debris may include gravel, metal fragments, plastic waste, bird carcasses, etc., which can easily cause serious flight accidents if they are sucked into aircraft engines or collide with landing gear.

[0003] Existing methods for detecting foreign objects on runways largely rely on manual inspections, requiring personnel to periodically check along the runway. This approach is limited by labor costs and working hours, making it difficult to achieve 24 / 7, comprehensive monitoring. Especially in complex environments such as nighttime or inclement weather, the accuracy and efficiency of manual inspections drop significantly, resulting in noticeable blind spots.

[0004] Some airports have experimented with using single-spectrum imaging equipment for auxiliary detection, but such equipment can only capture some of the optical features of objects. For foreign objects whose color is similar to the runway surface or whose material is special, it is easy to miss or misjudge them. At the same time, traditional detection methods lack a systematic division of the runway area, making it difficult to quickly locate the specific location of foreign objects when they appear, thus prolonging the handling time.

[0005] Current technologies for analyzing foreign object features often remain at a single dimension, focusing either solely on texture changes or relying solely on reflective properties. They fail to establish a multi-feature collaborative judgment mechanism, leading to insufficient reliability of detection results. During peak flight arrival and departure times, this inefficient detection method may also affect the normal utilization efficiency of runways, indirectly increasing operational pressure. Summary of the Invention

[0006] The purpose of this invention is to provide a runway foreign object detection system based on image recognition to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides a runway foreign object detection system based on image recognition, the system comprising:

[0008] The runway block division unit is used to divide the runway surface into several independent detection blocks based on predefined runway partitioning rules.

[0009] The benchmark feature library construction unit is used to store the benchmark texture feature set and optical reflection feature set of each detection block in the absence of foreign objects;

[0010] The real-time image analysis unit is used to acquire real-time image sequences of the current detection block through multispectral imaging equipment deployed at the runway perimeter, and extract its real-time texture feature vector and reflection feature vector;

[0011] The feature deviation analysis unit is used to dynamically compare the real-time texture feature vector with the reference texture feature set to generate a texture deviation matrix, and at the same time compare the real-time reflection feature vector with the optical reflection feature set to generate a reflection deviation matrix.

[0012] The alarm decision unit is used to generate foreign object location coordinates and alarm level instructions based on the joint analysis results of the texture deviation matrix and the reflection deviation matrix.

[0013] Preferably, the runway block division unit performs the following operations:

[0014] Obtain the actual physical dimensions and surface material distribution data of the runway;

[0015] Using the runway centerline as the reference axis, the runway is divided into equal-area grid blocks according to a predefined block side length threshold.

[0016] Each grid block is assigned a corresponding material feature code based on its surface material type.

[0017] The material feature code is bound to the block coordinates and then transmitted to the benchmark feature library construction unit.

[0018] Preferably, the benchmark feature library construction unit performs the following operations:

[0019] Receive material feature codes and block coordinates from the runway block division unit;

[0020] During a predefined baseline acquisition period, a set of baseline images of each block under interference-free conditions is continuously acquired using a multispectral imaging device.

[0021] Multi-scale feature decomposition is performed on the baseline image set of each block to extract its spatial domain texture features and frequency domain reflectance features respectively;

[0022] The spatial domain texture features are stored as a reference texture feature set, and the frequency domain reflection features are stored as an optical reflection feature set.

[0023] Preferably, the real-time image parsing unit includes:

[0024] The image preprocessing subunit is used to perform illumination equalization and motion blur correction on the acquired real-time image sequence;

[0025] The feature mapping subunit is used to extract multi-level texture feature maps from the corrected image through a deep convolutional network;

[0026] The reflection analysis subunit is used to analyze infrared spectral data in an image to generate a histogram of reflection intensity distribution.

[0027] The coordinate association subunit is used to align the multi-level texture feature map with the reflection intensity distribution histogram according to pixel coordinates to generate a real-time feature vector set.

[0028] Preferably, the feature deviation analysis unit performs the following operations:

[0029] Receive the baseline texture feature set and optical reflection feature set from the baseline feature library construction unit;

[0030] Obtain the real-time feature vector set generated by the real-time image parsing unit;

[0031] The Euclidean distance matrix between the real-time texture feature vector and the corresponding block reference texture feature set is calculated as the texture deviation matrix;

[0032] The spectral angle difference between the real-time reflection feature vector and the corresponding block optical reflection feature set is calculated as the reflection deviation matrix;

[0033] The texture deviation matrix and reflection deviation matrix are transmitted to the alarm decision unit.

[0034] Preferably, the system further includes a multi-level detection model configuration unit, which includes:

[0035] The model initialization subunit is used to load the pre-trained foreign object detection neural network model;

[0036] The parameter adaptation subunit is used to dynamically adjust the model convolution kernel weights based on the benchmark feature set from the benchmark feature library construction unit.

[0037] The feature fusion subunit is used to perform tensor concatenation of the texture deviation matrix and the reflection deviation matrix from the feature deviation analysis unit;

[0038] The model inference subunit is used to generate a foreign object probability heatmap by processing the spliced ​​tensor through the adjusted foreign object detection neural network.

[0039] Preferably, the multi-level detection model configuration unit further includes:

[0040] The model integration control subunit is used to acquire the variance distribution data of the feature sets of each block in the benchmark feature library construction unit;

[0041] The weight allocation subunit is used to calculate the confidence weight of each block detection model based on the variance distribution data.

[0042] The decision fusion subunit is used to perform weighted fusion of foreign object probability heatmaps from different blocks to generate global detection results.

[0043] Preferably, the alarm decision unit performs the following operations:

[0044] Receive global detection results from the multi-level detection model configuration unit;

[0045] Obtain the texture deviation matrix and reflection deviation matrix transmitted by the feature deviation analysis unit;

[0046] When the probability of foreign objects in the global detection results exceeds a predefined threshold, the coordinates of the center of the foreign object are located based on the spatial distribution gradient of the texture deviation matrix and the reflection deviation matrix.

[0047] The foreign object threat level is calculated based on the deviation amplitude of pixels surrounding the center coordinates of the foreign object.

[0048] Trigger the corresponding alert protocol based on the threat level.

[0049] Preferably, the alarm decision unit further includes:

[0050] The time-series verification subunit is used to continuously acquire global detection results for multiple detection cycles;

[0051] The status continuous judgment subunit will initiate an emergency alarm command when the foreign object threat level continues to rise within a predefined duration.

[0052] The false alarm filtering subunit ignores alarm triggering when the foreign object threat level exceeds the threshold in a single cycle but disappears in the next cycle.

[0053] Preferably, the system further includes a data management unit, the data management unit comprising:

[0054] The storage optimization subunit is used to acquire the spatiotemporal distribution data of the texture deviation matrix and reflection deviation matrix generated by the feature deviation analysis unit;

[0055] The data compression subunit initiates lossy compression of the feature matrix when the variance of the spatiotemporal distribution data is continuously lower than a predefined compression threshold.

[0056] The storage space allocation subunit dynamically adjusts the historical data storage cycle based on the alarm frequency generated by the alarm decision unit.

[0057] Compared with the prior art, the beneficial effects of the present invention are:

[0058] The runway surface is divided into several independent detection blocks by using runway block segmentation units, making the detection process more targeted and enabling detailed monitoring of each block. This avoids the omission of details caused by the large area during overall detection. The detection data of each independent block is relatively concentrated, which facilitates subsequent feature analysis and comparison, and reduces the interference of data from irrelevant areas on the detection results.

[0059] The benchmark feature library construction unit stores the benchmark texture feature set and optical reflection feature set for each detection block under foreign object-free conditions, providing a precise reference standard for real-time detection. These benchmark feature sets are built separately for each block, fully considering the inherent differences that may exist between different blocks, such as local changes in runway material and fixed features formed by long-term effects of lighting conditions, making subsequent feature comparisons more accurate.

[0060] The real-time image analysis unit acquires real-time image sequences using multispectral imaging equipment deployed around the runway perimeter, and extracts real-time texture and reflectance feature vectors, enabling it to capture various features of foreign objects under different spectra. Compared to single-spectrum imaging, multispectral imaging can acquire richer information about objects; whether it's dark-colored gravel or transparent plastic fragments, they can be effectively identified through their unique texture and reflectance properties, thus broadening the range of detectable foreign objects.

[0061] The feature deviation analysis unit compares the real-time feature vector with the baseline feature set to generate a texture deviation matrix and a reflection deviation matrix, enabling multi-dimensional analysis of foreign object features. The texture deviation matrix reflects the differences in texture structure between the real-time image and the baseline image, such as surface unevenness caused by foreign objects; the reflection deviation matrix reflects changes in optical reflection characteristics, such as the difference in reflectivity between metallic foreign objects and the runway surface. The combination of the two matrices presents potential foreign object features from different perspectives, reducing the limitations of single-feature analysis.

[0062] The alarm decision unit generates foreign object location coordinates and alarm level commands based on the joint analysis results of the texture deviation matrix and reflection deviation matrix, making the location of foreign objects more accurate. Due to the prior block division and the specific location information of feature changes in the deviation matrix, the precise coordinates of the foreign object can be quickly pinpointed, facilitating rapid response by relevant personnel. Simultaneously, the alarm level commands derived from multi-feature joint analysis more reasonably reflect the potential hazard level of the foreign object, helping relevant departments to take appropriate measures according to the level, thus improving the response efficiency to foreign object issues. Attached Figure Description

[0063] Figure 1 This is a schematic diagram illustrating the working principle of the image recognition-based runway foreign object detection system described in this invention.

[0064] Figure 2 A flowchart illustrating the execution of runway block division units;

[0065] Figure 3 The flowchart for the characteristic deviation analysis unit;

[0066] Figure 4 Flowchart for configuring the multi-level detection model unit;

[0067] Figure 5 The flowchart is executed by the alarm decision unit. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Please see Figure 1 This invention provides a runway foreign object detection system based on image recognition. The system includes: a runway block division unit, a baseline feature library construction unit, a real-time image parsing unit, a feature deviation analysis unit, and an alarm decision unit, with each unit operating collaboratively. Specific implementation details are as follows:

[0070] The runway block division unit divides the runway surface into independent detection blocks. The benchmark feature library construction unit stores the foreign object-free state features of each block. The real-time image analysis unit collects and analyzes the image data of the current block. The feature deviation analysis unit compares the real-time features with the benchmark features to generate a deviation matrix. The alarm decision unit locates the coordinates of the foreign object based on the deviation matrix analysis results and generates an alarm command.

[0071] Example 1: See Figure 2 Before executing its operations, the runway segmentation unit needs to acquire the runway's basic physical data. This unit extracts the actual physical dimensions of the runway from the airport's digital elevation model, including runway length, width, and surface curvature parameters. Simultaneously, it collects runway surface material distribution data using a vehicle-mounted infrared spectral scanner to identify the material types in different areas, such as standard concrete, modified asphalt, or reflective marking coatings. Data input is stored in a structured format, including a mapping table between geographic coordinate boundaries and material physical properties. During processing, the unit uses the geographic coordinates of the runway centerline as the reference axis, with the axis definition generated according to the International Civil Aviation Organization (ICAO) runway marking specifications.

[0072] The grid-based partitioning algorithm is based on a predefined threshold for the side length of each block. With a side length threshold set to 5 meters, the runway surface is divided into a rectangular grid array using a spatial geometric algorithm. The partitioning process employs two-dimensional coordinate transformation technology, establishing a local coordinate system based on the runway centerline. The grid is divided at equal intervals along its length and into segments according to the threshold along its width. Each grid block generates a unique coordinate identifier using a binary encoding format of "longitude offset - latitude offset". The partitioning results in a set of equal-area blocks, with the number of blocks dynamically calculated based on the runway dimensions; for example, a standard 3600-meter runway is divided into 1728 independent blocks.

[0073] Material feature encoding and assignment is a core operation of the unit. The system has a built-in material encoding lookup table, with each encoding being a 4-digit fixed-length string: the first digit identifies the material category (C for concrete, A for asphalt), and the last three digits identify the reflectivity grade (e.g., 045 represents 45% reflectivity). The unit uses spatial overlay analysis to georeference the material distribution data with the grid block boundaries, matching the dominant material type to each block and assigning it an encoding. After encoding, the unit establishes a mapping table between block coordinates and materials. The data structure is in key-value pair format, where the key is the block coordinate identifier and the value is the material encoding string. This mapping table is transmitted to the baseline feature library construction unit via a distributed message queue, with timestamps and data verification codes appended during transmission.

[0074] The baseline feature library construction unit initiates baseline acquisition during low-altitude activity periods. The predefined baseline acquisition period is set from 02:00 to 04:00 local time, during which runway clearance status is verified by the airport air traffic control system. The unit controls a multispectral imaging equipment group deployed on towers on both sides of the runway, including visible light cameras and short-wave infrared sensors. The acquisition process executes automated sequential imaging for each block: a 30-second image stream is acquired for each block, with a frame rate of 24fps, a visible light resolution of 4096×2160 pixels, and infrared spectral data of 8-14μm. The resulting raw baseline image set is stored as a time-series file set, organized by block.

[0075] Multi-scale feature decomposition processing is divided into two parallel computation streams. The spatial domain texture feature extraction stream processes the visible light image sequence, using a four-level Gaussian pyramid downsampling to construct a multi-scale space. At each scale level, the gray-level co-occurrence matrix algorithm is applied to calculate the contrast, correlation, and entropy values ​​in the 0°, 45°, 90°, and 135° directions, generating a 144-dimensional texture feature vector. The frequency domain reflectance feature processing stream focuses on the infrared image sequence. First, a fast Fourier transform is performed on each frame to extract the frequency domain coefficients, and then the energy values ​​of eight characteristic frequency bands are calculated. Simultaneously, pixel-level reflectance is analyzed to generate a statistical histogram, quantifying the probability distribution of reflectance intensity in the [0,1] interval.

[0076] Feature storage employs a distributed database architecture. The baseline texture feature set is organized in a tree structure: the root node is the block coordinate ID, and child nodes contain feature vector sets with different timestamps. The optical reflectance feature set is stored as a third-order tensor structure: the first order corresponds to the block grid position, the second order represents the time series index, and the third order contains frequency band energy values ​​and reflectance histogram binning data. The feature library implements a version control mechanism; each baseline update generates a new feature snapshot, and historical versions are retained for state backtracking. All feature data is encrypted using AES-256 and stored on a solid-state storage array, and the indexing system supports millisecond-level feature retrieval.

[0077] The unit performs periodic feature calibration. Every 24 hours, a baseline feature verification is automatically performed. The verification process includes: acquiring the latest baseline image set, extracting feature vectors, and comparing them with the stored baseline using cosine similarity. When the similarity falls below a preset tolerance value, an automatic feature library update process is initiated, where the new feature set overwrites the old data after multiple rounds of noise filtering. The update log records the coordinates of changed blocks and the magnitude of feature differences, and the feature drift trend is visualized through the system monitoring interface.

[0078] The baseline feature library provides fundamental data support for the entire system. The optical parameters of the multispectral imaging equipment precisely correspond to the block division positions, and the equipment spatial resolution is set to correspond to an actual 3.2mm × 3.2mm ground area per pixel. The stored baseline features include time dimension information, with the daily baseline data volume reaching 17TB. Columnar compression storage technology is used to reduce the effective storage volume to 35% of the original data. The unit has a built-in fault recovery mechanism; when hardware malfunctions and data acquisition is interrupted, it automatically switches to redundant equipment to continue the task and marks the feature data of the abnormal time period as pending calibration.

[0079] Example 2: See Figure 3 The real-time image analysis unit processes the continuous data stream transmitted from the multispectral imaging device. The image preprocessing subunit receives the raw image sequence, which consists of interleaved RGB frames from the visible light channel and thermal radiation frames from the infrared channel. Visible light frame processing employs an adaptive histogram equalization algorithm to analyze the regional distribution characteristics of image brightness, segmenting highlight and shadow regions and adjusting their gamma values ​​accordingly. Infrared frame processing focuses on thermal radiation data, eliminating pixel value drift caused by environmental temperature disturbances. Motion blur correction addresses device jitter by capturing attitude angular velocity using a gyroscope sensor, calculating the point spread function to reconstruct a clear image, and dynamically selecting the correction kernel size based on wind speed data. The processed dual-channel images are temporarily stored in a circular buffer, retaining the most recent five frames for temporal domain analysis.

[0080] The feature mapping subunit is equipped with a dedicated hardware accelerator. This unit loads a pre-built deep convolutional network model, and the network input layer receives a pre-processed visible light image. The model architecture consists of five feature extraction stages: the first stage uses a 7×7 convolutional kernel to output a 64-channel feature map; the second stage applies a 3×3 convolution with a stride of 2 to achieve downsampling; the third to fifth stages use residual connection structures to extract multi-level features. The output of each stage is uniformly scaled to the original value using bilinear interpolation. Figure 1 A 4x4 dimension is used to form a spatially aligned, multi-level feature map stack. The bottom layer of the stack preserves edge details, while the higher layers abstract the outline patterns of foreign objects. The map data format is a floating-point tensor, with a 16-dimensional feature descriptor appended to each pixel location.

[0081] The reflectance analysis subunit independently processes the infrared spectral data stream. The unit parses the 8-14μm band information in the infrared image, separating background radiation from target reflectance components. Reflectance intensity is calculated based on Planck's radiation law, converting thermal radiation into reflectance, with a range calibrated between 0.2 and 0.95 reflectance. The distribution histogram is constructed using a dynamic binning mechanism; the bin width is automatically adjusted based on the current scene's reflectance range, with the smallest binning unit being 0.005 reflectance values. The histogram output is a 256-dimensional normalized vector, with each dimension representing the percentage of pixels within the corresponding reflectance range within the block. The processing includes outlier filtering to remove outliers caused by lens contamination.

[0082] The coordinate association subunit establishes spatial correspondences between multi-source data. This subunit receives multi-level texture feature tensors output from the feature mapping subunit and reflection intensity vectors generated by the reflection analysis subunit. The texture feature tensors are converted to pixel coordinates using a coordinate mapping table, which records the projection relationship between the network's output feature points and the original image coordinates. The reflection vectors are spatially resampled to match the visible light image resolution, and bicubic interpolation is used to maintain data continuity. Alignment is performed based on the runway coordinate system, registering texture features and reflection data according to a geographic coordinate grid. Each spatial grid point generates a composite feature vector containing 16-dimensional texture attributes, 1-dimensional reflection values, and 3-dimensional location encoding. The final real-time feature vector set is organized according to a time series, with each feature point accompanied by a millisecond-level timestamp and block ID index.

[0083] The feature deviation analysis unit performs dynamic difference detection. The unit loads reference data from the baseline feature library. The baseline texture feature set is stored as a sparse matrix structure, containing standard texture templates under different lighting conditions; the optical reflectance feature set contains reflectance baselines caused by seasonal changes. When texture deviation calculation starts, the unit retrieves the real-time texture vector and corresponding baseline data by block ID. The Euclidean distance algorithm calculates the feature space distance pixel-by-pixel, and the distance values ​​are smoothed using a Gaussian window to reduce local abrupt changes. The output texture deviation matrix contains the difference value for each grid point, and the data is stored as a single-precision floating-point two-dimensional array.

[0084] Reflectance deviation analysis employs a spectral feature comparison mode. The unit analyzes the real-time reflection vector and the reference reflection set, treating them as vector elements in a high-dimensional spectral space. A spectral angle mapping algorithm calculates the cosine of the angle between the vectors, and outputs the angle difference through an inverse cosine transform. The difference value is normalized and mapped to the 0-1 interval, with 0 representing a perfect match. A reflection deviation matrix is ​​generated to label spectral mismatch areas, maintaining the same matrix dimension as the texture deviation matrix. During the analysis, a partitioned adaptive threshold is applied, with a more lenient discrimination tolerance set for high-reflectivity material blocks.

[0085] The dual-bias matrix transmission employs differential coding technology. The texture bias matrix and reflection bias matrix are transmitted as a single packet, with the data packet including matrix header information: runway partition version number, acquisition timestamp, and device serial number. Matrix data undergoes lossless compression to reduce bandwidth consumption, achieving a compression rate of over 65% using the LZW algorithm. The transmission interval is fixed at 250 milliseconds, with a retransmission mechanism triggered in case of abnormalities to ensure data integrity. The receiving end buffers the bias matrices from the most recent three periods for input preparation for the trend analysis module. All matrices are appended with checksums to ensure data reliability; if verification fails, data from the previous period is used for temporary processing.

[0086] The unit implements a fault monitoring mechanism. The self-diagnostic module continuously monitors for anomalies in deviation calculations: when the abrupt change rate of consecutive grid points in the texture deviation matrix exceeds 25%, a sensor cleaning alarm is activated; when the reflection deviation remains below the detection threshold, the infrared sensor gain setting is automatically adjusted. Processing latency is maintained within 70 milliseconds, and timeouts trigger algorithm acceleration strategies—reducing the feature vector dimension or lowering the spatial sampling rate. Historical deviation data is retained for a 24-hour period to trace the state change trends during the detection process.

[0087] Example 3: See Figure 4 The multi-level detection model configuration unit loads a pre-trained foreign object detection neural network from a persistent repository during the initialization phase. This network adopts a U-Net architecture topology. The encoder contains an eight-level downsampling module, with each level containing two convolutional operations and a max-pooling layer. The decoder achieves upsampling through transposed convolutions and concatenates them with the corresponding encoding layer features. The initial model parameters are derived from training results of tens of thousands of airport scene samples, and the model file format is the Open Neural Network Exchange Format. The parameter adaptation sub-unit dynamically adjusts the model parameters to adapt to the current feature environment. This sub-unit receives the block optical reflectance feature set transmitted by the baseline feature library construction unit and extracts the temporal statistics of the feature set as the basis for adjustment. The adjustment process is executed step-by-step: first, the parameters of the initial four convolutional kernels are frozen to maintain general feature extraction capabilities; then, a weight update algorithm is applied to the first layer of the decoder, scaling the convolutional kernel gain coefficients based on the variance data of the block texture features; finally, an embedding vector encoding the material features is added to the output layer and projected onto the feature space through a fully connected layer.

[0088] The feature fusion subunit processes dual-source data from the feature deviation analysis unit. Input data includes a texture deviation matrix and a reflection deviation matrix, both single-precision floating-point two-dimensional arrays. The fusion operation unfolds in four-dimensional space: the first dimension indexes the spatial positions of different blocks, the second dimension represents the image row coordinates, the third dimension represents the column coordinates, and the fourth dimension represents the deviation feature channels. The texture deviation matrix is ​​assigned to channels 0-5 to store six sets of texture statistics; the reflection deviation matrix occupies channels 6-11 to record spectral difference indices. The stitching operation follows tensor alignment rules; when the input matrix size is M×N, the output tensor size is M×N×12. The output tensor includes device ID and acquisition timestamp metadata.

[0089] The model inference subunit performs deep processing of tensor data. The adjusted U-Net network receives feature fusion tensors as input, and the first layer of the encoding path uses dilated convolutional kernels to expand the receptive field. The kernel size is 5×5 and the dilation rate is set to 2. Each convolutional layer is followed by a batch normalization layer and a parameterized modified linear unit activation function. Skip connection operations are implemented in the third, fifth, and seventh levels, concatenating the encoder intermediate features with the upsampled features along the channel dimension. The output layer applies a spatial pyramid pooling module, capturing multi-scale contextual information through multiple parallel dilated convolutions. The final output foreign object probability heatmap is a 32-bit floating-point matrix with the same size as the original input image. The value range of each pixel [0,1] represents the probability density of foreign object presence.

[0090] The model integration control subunit manages multi-block joint inference. This module collects variance distribution data of the feature sets of each block in the benchmark feature library construction unit and establishes a quantitative index for the stability of block features. The system maintains the variance distribution matrix S, and the matrix elements... Indicates the first Block number The variance values ​​of each feature channel are calculated over a 24-hour statistical window. The weight allocation sub-unit calculates the block confidence level based on the variance matrix.

[0091]

[0092] Here Indicates the first Block confidence weights This represents the total number of feature channels. The variance is represented by the characteristic variance. The calculation results are normalized so that the sum of the weights of all blocks is 1, and blocks with high variance are assigned lower weights to suppress the influence of unstable regions.

[0093] The decision fusion subunit receives foreign object probability heatmaps from all blocks. The input heatmap set undergoes spatial calibration and is stitched together according to the block division coordinates to form a complete runway coverage map. The fusion process employs a weighted average strategy: the global probability value of each pixel is equal to the heatmap value of its corresponding block multiplied by the confidence weight of that block. The fusion result generates a global detection probability map, with the data format being a floating-point two-dimensional matrix. The spatial resolution maintains a 1:100 mapping relationship with the actual runway size. The fusion output includes block boundary marker data for subsequent precise positioning processing.

[0094] The data flow management module optimizes processing efficiency. The hardware accelerator employs a mixed-precision computing mode: forward inference uses half-precision floating-point numbers to improve throughput, while weight updates retain single-precision precision. Memory allocation implements a dynamic scheduling strategy, allocating independent GPU memory space when multiple blocks are processed concurrently; and reusing computing resources when processing a single block. Inference latency is controlled within a 50-millisecond cycle, and a model simplification mechanism is activated in timeout scenarios, skipping the pyramid pooling module to reduce computational complexity. All processing results are marked with version numbers and time identifiers and transmitted to the alarm decision unit via a high-speed serial bus.

[0095] Example 4: See Figure 5 The alarm decision unit receives a global detection result data packet transmitted by the multi-level detection model configuration unit. The data packet structure includes three core fields: a timestamp field recording the precise time of data acquisition in UTC milliseconds; a probability heatmap field storing a two-dimensional floating-point array, where each element represents the probability of an object being present at the corresponding runway coordinate point; and a block identifier field recording the block number to which each pixel belongs in the heatmap. Simultaneously, the unit acquires the texture deviation matrix and reflection deviation matrix sent by the feature deviation analysis unit. The matrix data includes spatial calibration parameters to ensure strict alignment with the heatmap coordinate system. Upon data reception, format verification is first performed, checking timestamp continuity, matrix dimension matching, and numerical validity. If an anomaly is detected, the most recent three valid frames are used for interpolation compensation.

[0096] Foreign object (FOO) localization is implemented using gradient analysis. The unit first binarizes regions exceeding a threshold of 0.85 in the probability heatmap to generate potential FAO masks. For each connected component, the corresponding region data of the texture deviation matrix and reflection deviation matrix are extracted. Spatial gradient calculation uses the central difference method, calculating the rate of change of deviation values ​​along both the longitudinal and transverse directions of the runway, and identifying gradient abrupt change points as candidate FAO boundary points. The boundary point set is fitted into a polygonal region using a convex hull algorithm, and the centroid coordinates of the polygon are calculated as the FAO center position. The localization result includes the following attributes: geographic coordinates are represented using the airport's local coordinate system with centimeter-level accuracy; size features record the length and width of the polygon's circumscribed rectangle; and orientation angles describe the angle between the FAO's main extension direction and the runway centerline.

[0097] The threat level assessment module analyzes multi-dimensional indicators of foreign object characteristics. The assessment indicator system includes two types of data: static attributes and dynamic trends. Static attributes are directly extracted from the location results, including the projected area of ​​the foreign object, its distance from the runway centerline, and material reflectivity. Dynamic trends are obtained by analyzing data changes over five consecutive detection cycles, including the area expansion rate, position movement vector, and probability value growth slope. The assessment rule base adopts a hierarchical decision tree structure. The first layer determines whether the foreign object is in a critical area (e.g., 300 meters before the runway entrance), the second layer analyzes whether the foreign object size exceeds a safety threshold, and the final layer adjusts the level based on the overall dynamic trend. Threat levels are divided into three levels: Level 1 represents high-risk foreign objects that require immediate removal, Level 2 represents medium-risk foreign objects that require monitoring, and Level 3 represents low-risk foreign objects that can be temporarily deferred.

[0098] The time-series verification subunit maintains sliding time window data. The default window span is set to 30 seconds, storing a historical record of all detection results within the period. The subunit implements a state machine model to manage the foreign object's lifecycle, defining three basic states: initial detection, continuous tracking, and disappearance confirmation. When a foreign object first appears, a tracking instance is created, a unique identifier is assigned, and a feature vector is initialized. The state continuity judgment subunit monitors the threat level change curve. When the threat level increases for three consecutive periods, and the last level reaches level two or above, an emergency alarm protocol is triggered. A false alarm filtering mechanism is designed for transient anomalies. If a foreign object appears within a single period but the probability value on the heatmap decays to below 0.2 in subsequent periods, it is marked as environmental noise and not reported.

[0099] During the alarm protocol execution phase, a response strategy is selected based on the threat level. Level 1 alarms trigger a system-wide emergency response, including sending a red alert signal to the air traffic control tower, activating the runway lighting warning system, and automatically generating a NOTAM (Not Attention Notice). Level 2 alarms initiate the inspection plan, sending a work order containing the foreign object's coordinates to the airfield operations department, requiring on-site verification within 30 minutes. Level 3 alarms are only recorded in the runway maintenance log for periodic analysis. All alarm events generate structured reports containing the following key fields: event number using a combination of year, month, day, hour, minute, second + block number; location data recording the foreign object's center coordinates and boundary vertex set; time series data storing the probability values ​​and characteristic deviation statistics for the most recent six periods; and a processing status field tracking the alarm response progress.

[0100] The table below shows the data records of the alarm decision-making process for a typical foreign object incident:

[0101]

[0102] The system implements multiple safeguards to ensure decision reliability. A data consistency check module compares the spatiotemporal characteristics of the heatmap and the deviation matrix; when significant contradictions are found, a manual review process is initiated. The clock synchronization system maintains a time deviation of less than 5 milliseconds across all units to avoid timing analysis errors. A historical case library stores processing records of typical foreign object incidents from the past three years, providing similarity references for current decisions. A real-time monitoring interface for unit operation status displays the load of each stage of the processing pipeline, including performance indicators such as data reception rate, computation latency, and storage throughput; abnormal situations automatically trigger a degradation processing mode. All output data is written to the airport's central database and simultaneously backed up to an off-site disaster recovery center, with a backup cycle of 15-minute incremental updates.

[0103] Example 5: The data management unit continuously receives the spatiotemporal distribution dataset output by the feature deviation analysis unit. This dataset contains temporal change records of the texture deviation matrix and reflection deviation matrix, with each record appended with a collection timestamp and block location identifier. The spatiotemporal distribution data is organized in a three-dimensional structure: the first dimension represents the time axis, with new data points generated at 250-millisecond intervals; the second dimension stores the runway's horizontal coordinate index; and the third dimension records the vertical coordinate index. Matrix data values ​​are stored in single-precision floating-point format, with each element representing the deviation intensity at the corresponding spatiotemporal location. The system generates a unique hash checksum for each frame of data. The checksum is calculated based on the value range characteristics of the matrix elements and is used for subsequent data integrity verification.

[0104] The storage optimization subunit processes the raw input data stream. This subunit parses the three-dimensional structure of the spatiotemporal distribution data and extracts the spatial distribution pattern of each time slice. The unit generates a two-level storage index: a time index partitioned by minute to establish a directory structure, storing all time slice data for each minute; and a spatial index to establish a mapping table between block coordinates and physical storage locations. The index file uses a tree structure to accelerate queries, with leaf nodes recording the physical sector addresses of the data files in the storage array. The storage location selection algorithm is optimized based on access frequency: recent data is stored in the high-speed area of ​​the solid-state drive, while historical data is migrated to a large-capacity mechanical hard drive array. Index updates and data write operations maintain atomicity, and the index structure can be rebuilt through transaction logs after power failure recovery.

[0105] The data compression subunit monitors the statistical characteristics of the spatiotemporal distribution data. The system maintains a global variance analyzer, calculating the variance sequences of the texture deviation matrix and reflection deviation matrix every five minutes. The variance calculation range includes all data points from the most recent three detection cycles, and the rolling variance value is output for compression decisions. When the rolling variance of the texture deviation matrix is ​​continuously below the compression threshold of 0.015 and the rolling variance of the reflection deviation matrix is ​​below 0.02, the unit activates the lossy compression process. The compression process is executed on a dedicated coprocessor, using an improved JPEG2000 algorithm: first, the floating-point matrix is ​​normalized to the integer range of 0-255, then a discrete wavelet transform is performed to generate multi-level sub-band coefficients; all information of the low-frequency sub-band is retained, and quantization processing based on human visual characteristics is implemented for the high-frequency sub-band; finally, a compressed bitstream is generated through an MQ encoder. The compression ratio is controlled between 5:1 and 8:1, and a marker header is added to the compressed data to record the original matrix range and quantization parameters.

[0106] The storage space allocation subunit dynamically adjusts the data retention period. This unit obtains alarm event logs generated by the alarm decision unit through the system bus. The logs contain alarm level and trigger time information. The allocation algorithm maintains an alarm frequency counter: with a statistical window of 6 hours, when the number of triggers for a level 1 alarm reaches 5, the storage period of the corresponding block's historical data is automatically extended to 72 hours; areas without alarms retain a default storage period of 24 hours; and areas with level 2 alarms adopt an intermediate value of 48 hours. The data cleanup task is executed daily at 4:00 AM, marking expired data according to the storage period configuration. The cleanup process consists of two phases: the first phase moves expired data to a buffer pool and marks it as pending deletion; the second phase performs physical erasure when the system load is below 30%. The cleanup task generates an operation report, recording the amount of storage space released and the time range of data deletion.

[0107] The storage resource management module coordinates the hardware infrastructure. The system manages multiple storage volumes: a high-performance storage volume using an NVMe solid-state drive array with a capacity of 8TB, responsible for storing nearly 24 hours of raw data; and an archive storage volume using a RAID6 hard disk drive group with a total capacity of 192TB, storing compressed historical data. The volume management program monitors the utilization rate of each storage volume, and when the primary volume's space occupancy exceeds 85%, it initiates automatic data migration: selecting the earliest inactive data from that period, performing compression processing, and then transferring it to the archive volume. Archive data index files are stored independently, supporting fast retrieval by time range and block coordinates. All storage devices are configured with dual power supply redundancy, and in the event of an abnormal power outage, the backup power supply supports 10 minutes of data buffering and transfer.

[0108] Fault tolerance mechanisms ensure data security. The unit implements a real-time data backup channel: during normal operation, each frame of data is synchronously written to the main storage array and the backup server. Backups use a differential transmission mode, sending only incrementally changed portions. The data recovery function is automatically activated upon detecting a verification failure: it locates the corrupted data by comparing hash codes and retrieves a healthy copy from the backup server to overwrite and repair it. If the backup data is also unavailable, a data reconstruction process is initiated: linear interpolation reconstruction is performed based on data from preceding and following time frames. The unit maintains an operational status dashboard, continuously displaying key indicators such as storage space utilization, data compression ratio, and backup synchronization latency. The timestamp calibration system automatically synchronizes daily with the airport's base clock, ensuring that the time deviation of all records is less than 10 milliseconds.

[0109] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0110] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A runway foreign object detection system based on image recognition, characterized in that, Includes the following execution units: The runway block division unit is used to divide the runway surface into several independent detection blocks based on predefined runway partitioning rules. The benchmark feature library construction unit is used to store the benchmark texture feature set and optical reflection feature set of each detection block in the absence of foreign objects; The real-time image analysis unit is used to acquire real-time image sequences of the current detection block through multispectral imaging equipment deployed at the runway perimeter, and extract its real-time texture feature vector and reflection feature vector; The feature deviation analysis unit is used to dynamically compare the real-time texture feature vector with the reference texture feature set to generate a texture deviation matrix, and at the same time compare the real-time reflection feature vector with the optical reflection feature set to generate a reflection deviation matrix. The alarm decision unit is used to generate foreign object location coordinates and alarm level instructions based on the joint analysis results of the texture deviation matrix and the reflection deviation matrix; The multi-level detection model configuration unit includes: The model initialization subunit is used to load the pre-trained foreign object detection neural network model; The parameter adaptation subunit is used to dynamically adjust the model convolution kernel weights based on the benchmark feature set from the benchmark feature library construction unit. The feature fusion subunit is used to perform tensor concatenation of the texture deviation matrix and the reflection deviation matrix from the feature deviation analysis unit; The model inference subunit is used to generate a foreign object probability heatmap by processing the spliced ​​tensor through the adjusted foreign object detection neural network. The alarm decision unit performs the following operations: Receive global detection results from the multi-level detection model configuration unit; Obtain the texture deviation matrix and reflection deviation matrix transmitted by the feature deviation analysis unit; When the probability of foreign objects in the global detection results exceeds a predefined threshold, the coordinates of the center of the foreign object are located based on the spatial distribution gradient of the texture deviation matrix and the reflection deviation matrix. The foreign object threat level is calculated based on the deviation amplitude of pixels surrounding the center coordinates of the foreign object. Trigger the corresponding alert protocol based on the threat level; The alarm decision unit also includes: The time-series verification subunit is used to continuously acquire global detection results for multiple detection cycles; The status continuous judgment subunit will initiate an emergency alarm command when the foreign object threat level continues to rise within a predefined duration. The false alarm filtering subunit ignores alarm triggering when the foreign object threat level exceeds the threshold in a single cycle but disappears in the next cycle.

2. The runway foreign object detection system based on image recognition according to claim 1, characterized in that, The runway block division unit performs the following operations: Obtain the actual physical dimensions and surface material distribution data of the runway; Using the runway centerline as the reference axis, the runway is divided into equal-area grid blocks according to a predefined block side length threshold. Each grid block is assigned a corresponding material feature code based on its surface material type. The material feature code is bound to the block coordinates and then transmitted to the benchmark feature library construction unit.

3. The runway foreign object detection system based on image recognition according to claim 2, characterized in that, The benchmark feature library construction unit performs the following operations: Receive material feature codes and block coordinates from the runway block division unit; During a predefined baseline acquisition period, a set of baseline images of each block under interference-free conditions is continuously acquired using a multispectral imaging device. Multi-scale feature decomposition is performed on the baseline image set of each block to extract its spatial domain texture features and frequency domain reflectance features respectively; The spatial domain texture features are stored as a reference texture feature set, and the frequency domain reflection features are stored as an optical reflection feature set.

4. The runway foreign object detection system based on image recognition according to claim 3, characterized in that, The real-time image parsing unit includes: The image preprocessing subunit is used to perform illumination equalization and motion blur correction on the acquired real-time image sequence; The feature mapping subunit is used to extract multi-level texture feature maps from the corrected image through a deep convolutional network; The reflection analysis subunit is used to analyze infrared spectral data in an image to generate a histogram of reflection intensity distribution. The coordinate association subunit is used to align the multi-level texture feature map with the reflection intensity distribution histogram according to pixel coordinates to generate a real-time feature vector set.

5. The runway foreign object detection system based on image recognition according to claim 4, characterized in that, The feature deviation analysis unit performs the following operations: Receive the baseline texture feature set and optical reflection feature set from the baseline feature library construction unit; Obtain the real-time feature vector set generated by the real-time image parsing unit; The Euclidean distance matrix between the real-time texture feature vector and the corresponding block reference texture feature set is calculated as the texture deviation matrix; The spectral angle difference between the real-time reflection feature vector and the corresponding block optical reflection feature set is calculated as the reflection deviation matrix; The texture deviation matrix and reflection deviation matrix are transmitted to the alarm decision unit.

6. The runway foreign object detection system based on image recognition according to claim 5, characterized in that, The multi-level detection model configuration unit also includes: The model integration control subunit is used to acquire the variance distribution data of the feature sets of each block in the benchmark feature library construction unit; The weight allocation subunit is used to calculate the confidence weight of each block detection model based on the variance distribution data. The decision fusion subunit is used to perform weighted fusion of foreign object probability heatmaps from different blocks to generate global detection results.

7. The runway foreign object detection system based on image recognition according to claim 1, characterized in that, It also includes a data management unit, which comprises: The storage optimization subunit is used to acquire the spatiotemporal distribution data of the texture deviation matrix and reflection deviation matrix generated by the feature deviation analysis unit; The data compression subunit initiates lossy compression of the feature matrix when the variance of the spatiotemporal distribution data is continuously lower than a predefined compression threshold. The storage space allocation subunit dynamically adjusts the historical data storage cycle based on the alarm frequency generated by the alarm decision unit.

Images