An image recognition monitoring system for an airport runway intrusion target

Abstract

The application provides an image recognition monitoring system for runway intrusion targets, comprising a video acquisition module, an image preprocessing module, a target detection module, a target tracking module and a cross-camera consistency recognition module; wherein the video acquisition module is arranged on multiple cameras in a target area and is used for acquiring original video streams; the image preprocessing module is used for preprocessing the original video streams, including image local contrast enhancement, non-local mean denoising and super-resolution reconstruction; the target detection module is used for detecting targets in the preprocessed video images, obtaining target categories, confidence scores and corresponding target bounding boxes; the target tracking module is used for tracking targets according to the target detection results, obtaining tracking trajectories; and the cross-camera consistency recognition module is used for calculating and recognizing according to the target detection results and the tracking trajectories, completing cross-camera target association.

Description

Technical Field

[0001] This invention relates to a target recognition and monitoring system, and more particularly to an image recognition and monitoring system for targets intruding into an airport runway. Background Technology

[0002] With the rapid development of the air transport industry, the operational efficiency and safety of civil aviation airports are facing unprecedented challenges. Runways, as one of the most critical infrastructures of an airport, are the essential passage for aircraft takeoff and landing, and their safety status directly affects flight safety. However, in recent years, "runway incursions" caused by unauthorized entry into runway areas by non-cooperative targets such as personnel, vehicles, equipment, or birds have occurred frequently, becoming a prominent issue restricting aviation safety. Statistics show that hundreds of runway incursions occur globally each year, a significant portion of which result in serious aviation accidents or near misses. Therefore, how to achieve real-time monitoring of runway incursions by non-cooperative targets has become a key scientific problem and engineering challenge in the field of intelligent airport operation management.

[0003] Currently, existing technologies for monitoring runway incursions mainly rely on two types of methods: First, traditional manual monitoring, where airport tower or ground monitoring personnel observe the runway situation in real time via video monitors. However, due to the large monitoring range, complex weather conditions, and limited attention span of monitoring personnel, manual monitoring often suffers from fatigue, missed detections, and delays. Second, automated detection methods based on traditional image processing algorithms, such as background modeling, inter-frame differencing, and optical flow methods. These methods can achieve certain detection results in static scenes or controlled environments, but in actual airport operating environments, they are often severely affected by factors such as changes in lighting, shadow interference, low visibility, and the diversity of target appearances, resulting in insufficient detection accuracy and stability, making it difficult to meet the high standards required for runway safety.

[0004] In recent years, with the widespread application of deep learning technology in object detection and recognition, detection methods based on convolutional neural networks (such as Faster R-CNN, YOLO series, SSD, etc.) have achieved significant breakthroughs in both accuracy and speed, and have gradually been applied to traffic monitoring and public safety. However, directly applying these methods to airport runway environments still faces several challenges:

[0005] First, the types of non-cooperative targets in the runway scenario are complex, including people, vehicles, birds and other foreign objects. The targets vary greatly in size and appearance, which makes the detection model perform unstably on small targets and targets with special aspect ratios.

[0006] Secondly, airport environments change drastically, with varying lighting and weather conditions during the day, night, and rainy days, which makes traditional models insufficient in terms of cross-scene generalization capabilities. Thirdly, the switching between multiple cameras and overlapping coverage areas often lead to the same target being repeatedly identified or losing its association under different cameras, affecting the continuity and reliability of overall surveillance.

[0007] Furthermore, existing technologies have significant shortcomings in target tracking and cross-camera recognition. Traditional tracking methods such as SORT or KCF are prone to target loss under conditions of rapid movement or partial occlusion. While appearance-based re-identification (ReID) methods have made progress in pedestrian re-identification, their direct application to airport scenarios often fails to achieve ideal recognition results due to the diverse target categories, significant pose differences, and substantial changes in camera angles. Especially under cross-camera conditions, differences in lighting, resolution, and imaging angles between different cameras significantly reduce the consistency of target features, leading to misidentification and missed identification.

[0008] Therefore, how to achieve high-precision detection, stable tracking, and consistent recognition across cameras for non-cooperative targets on the runway in complex airport environments is a technical challenge that urgently needs to be solved. Summary of the Invention

[0009] Purpose of the invention: The technical problem to be solved by the present invention is to provide an image recognition and monitoring system for airport runway intrusion targets, which addresses the shortcomings of the existing technology.

[0010] To address the aforementioned technical problems, this invention discloses an image recognition and monitoring system for airport runway intrusion targets, comprising:

[0011] The system includes a video acquisition module, an image preprocessing module, a target detection module, a target tracking module, and a cross-camera consistency recognition module; among which,

[0012] The video acquisition module is installed on multiple cameras in the target area to capture raw video streams;

[0013] The image preprocessing module preprocesses the original video stream, including local image contrast enhancement, non-local mean denoising, and super-resolution reconstruction.

[0014] The target detection module detects targets in the preprocessed video image and obtains the target category, confidence score and corresponding target bounding box;

[0015] The target tracking module performs target tracking based on the target detection results and obtains the tracking trajectory;

[0016] The cross-camera consistency recognition module performs calculations and recognition based on the target detection results and tracking trajectory to complete the cross-camera target association.

[0017] Furthermore, the image local contrast enhancement includes:

[0018] The input video frame is divided into pixel blocks of a preset size and the contrast gain limit within the block is set;

[0019] Let the input video frame be I, and after converting it to the LAB color space, we get the luminance channel L;

[0020] For the A pixel block, its grayscale histogram is grayscale level is , , This represents the number of gray levels; let the cropping threshold be... The histogram after cropping It is expressed as follows:

[0021]

[0022] Calculate the corresponding cumulative distribution function , means as follows:

[0023]

[0024] Calculate the local grayscale mapping function , means as follows:

[0025]

[0026] Bilinear interpolation is performed on the mapping results of adjacent pixel blocks to obtain the enhanced luminance channel. And synthesize an enhanced image with the original color channels. As output.

[0027] Furthermore, the nonlocal mean denoising includes:

[0028] Let the input image to be denoised be pixel The denoising results are shown below:

[0029]

[0030] in, Represented by pixels The search window is centered on the search. Represents pixels For pixels The weighted contribution, or weight, satisfies:

[0031]

[0032] Weight The calculation method is as follows:

[0033]

[0034] in, and ) represent the neighborhood blocks centered at pixels p and q, respectively. This represents the Gaussian-weighted Euclidean distance. For smoothing parameters, These are the normalization coefficients;

[0035] The denoising result of the current pixel is the weighted average of all pixels in the search window, calculated according to their respective weights.

[0036] Furthermore, the super-resolution reconstruction includes:

[0037] A super-resolution reconstruction model is constructed, whose network structure includes a shallow feature extraction layer, a deep feature extraction module, and an upsampling reconstruction module. The shallow feature extraction layer consists of a 3×3 convolutional layer and a ReLU activation function, mapping the input low-resolution image to a high-dimensional feature space. The deep feature extraction module consists of 16 stacked residual blocks, each containing two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function, with skip connections introduced to element-wise add the input and output of the residual block. The upsampling reconstruction module uses sub-pixel convolution to achieve upsampling. The super-resolution reconstruction model is trained using a weighted combination of the L1 loss function and a feature map-aware loss extracted based on a pre-trained VGG-19 network.

[0038] Let the low-resolution image be The corresponding image resolution is The super-resolution reconstruction model is represented as Then reconstruct the image It is expressed as follows:

[0039]

[0040] Training loss of super-resolution reconstruction model , means as follows:

[0041] in, It is an L1 norm. It is the square L2 norm. This represents the features extracted by the pre-trained VGG network. and These are the weighting coefficients.

[0042] Furthermore, the detection of targets in the preprocessed video image includes:

[0043] A YOLOv8n detection model was constructed, comprising a backbone network, a neck network, and a detection head. The backbone network uses an improved CSPDarkNet, consisting of stacked CBS convolutional, C2f, and SPPF modules. The CBS convolutional module is composed of Conv2d, BatchNorm2d, and SiLU activation functions in sequence. The C2f module is an improved cross-stage partial connection module that splits the input feature map channels. The SPPF module is located at the end and aggregates multi-scale contextual information through three cascaded 5×5 max-pooling layers. The backbone network contains five downsampling stages, and incorporates 8x downsampling and 16x downsampling... The 1x and 32x downsampling results are fed into the neck network. The neck network adopts a PAN-FPN bidirectional feature fusion structure, adding a bottom-up enhancement branch to the top-down path of the FPN network. It completes the top-down propagation through upsampling concatenation and C2f fusion, and then completes the bottom-up enhancement through downsampling concatenation, finally outputting detection feature maps at three scales. The detection head separates the classification and regression tasks into two independent branches, each containing two layers of 3×3 convolutions and one layer of 1×1 prediction convolution, finally predicting the target center point offset and bounding box width and height. The loss function of the YOLOv8n detection model adopts the binary cross-entropy classification loss and the regression loss of the CIoU and DFL weighted combination. The YOLOv8n detection model is trained and optimized using the airport dataset.

[0044] The preprocessed video image is input into the optimized YOLOv8n detection model to obtain the target category, confidence score, and corresponding target bounding box.

[0045] Furthermore, the target tracking based on the target detection results includes:

[0046] Using the target bounding box and confidence score of the current frame detection results provided by the target detection module, the motion state of each tracked target is predicted by Kalman filtering to obtain the predicted position of the target in the current frame;

[0047] Calculate the matching cost matrix between the detection result and the predicted location, including Mahalanobis distance and cosine distance;

[0048] The Mahalanobis distance and appearance distance are linearly fused using weighted coefficients to obtain the comprehensive matching cost;

[0049] The augmented path method is used to find the matching scheme with the minimum total cost based on the comprehensive matching cost method, and the target identity association and trajectory update of the current frame are completed.

[0050] For targets whose identities are successfully associated, Kalman filtering is used to update their motion state. For unmatched detected targets, a new tracking trajectory is initialized or they are marked as lost.

[0051] Furthermore, the appearance distance is calculated based on the target detection features obtained by the appearance feature extraction network of the self-trained data.

[0052] Furthermore, the appearance feature extraction network for the self-trained data includes:

[0053] The input layer receives RGB three-channel color images with a size of 64×128;

[0054] After the input layer are two standard convolutional modules, each consisting of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function;

[0055] The feature map spatial resolution was then reduced to 32×64 using a 2×2 max pooling layer.

[0056] Then there are 6 stacked residual modules. Each residual module contains two branches. The main branch consists of two 3×3 convolutional layers in sequence. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The residual branch is a 1×1 convolutional layer. The outputs of the main branch and the residual branch are added element by element and then output through the ReLU activation function.

[0057] Then, a 2×2 max pooling layer is used to reduce the feature map to 16×32, and then 6 residual modules are stacked.

[0058] Subsequently, the spatial dimension is compressed to 1×1 by a global average pooling layer, resulting in a 128-dimensional feature vector.

[0059] Finally, after passing through a fully connected layer, a batch normalization layer, and an L2 normalization layer, the final 128-dimensional normalized appearance embedding vector is output, which is the feature of the detected target.

[0060] During the training phase, airport video data was used as training samples, and joint supervision was carried out using the cross-entropy loss function and the triplet loss function.

[0061] Furthermore, the target detection results and tracking trajectory are used to calculate and identify the target. A ResNet-50-based re-identification model is used to extract the target's depth appearance features, and cross-camera matching is performed by combining the weighted similarity of Euclidean distance and cosine distance.

[0062] Furthermore, the calculation and recognition based on the target detection results and tracking trajectory includes:

[0063] Let the first The camera at the The video images captured in frames are The first output of the target detection module Each detection box is ,in, , , and Let x, y, width, and height be the origin, x, y, width, and height of the detection box, respectively. Then the corresponding target region image is: , means as follows:

[0064]

[0065] in, This indicates a cropping operation on the original video image based on the object detection bounding box;

[0066] The target region image is input into the ResNet-50 feature extraction network. Then, normalization is performed to obtain the normalized depth appearance feature vector. , means as follows:

[0067]

[0068]

[0069] The ResNet-50 feature extraction network includes:

[0070] The initial convolutional layer Conv1 and a 3×3 max-pooling layer, connected sequentially, reduce the input image resolution from 256×128 to 64×32. This is followed by four feature extraction stages consisting of stacked residual modules. Each stage uses convolutions with a stride of 2 to progressively downsample the image, ultimately outputting an 8×4 feature map with 2048 channels. ;

[0071] The feature map is subjected to global average pooling to obtain a 2048-dimensional feature vector, which is then reduced to 512 dimensions by a fully connected layer. After batch normalization and L2 normalization, the final 512-dimensional normalized depth appearance feature vector is obtained. .

[0072] When matching across cameras, 512-dimensional normalized depth appearance feature vectors are extracted from the target images acquired by different cameras. Euclidean distance, which reflects the difference in absolute magnitude, and cosine distance, which reflects the difference in direction, are calculated. A weighted fusion strategy is used to obtain a comprehensive similarity score. When the comprehensive score exceeds the threshold, the target is determined to be the same person, thus completing the cross-camera target association.

[0073] Beneficial effects:

[0074] (1) The present invention effectively improves the detection effect under low light and low resolution conditions through image preprocessing technology, and enhances the adaptability of the system.

[0075] (2) The lightweight detection model based on YOLOv8n in this invention balances accuracy and real-time performance, and can meet the requirements of large-scale airport monitoring.

[0076] (3) This invention significantly improves the stability and robustness of target tracking by using an improved DeepSORT algorithm through feature input dimension adjustment and self-trained appearance network;

[0077] (4) This invention combines ResNet-50 feature extraction and similarity weighted fusion method to achieve high-precision consistency recognition in cross-camera scenarios and solves the problem of target association in multi-camera environments; Attached Figure Description

[0078] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0079] Figure 1 This is a schematic diagram of the overall architecture of the system of the present invention.

[0080] Figure 2 This is a flowchart of the target detection process based on YOLOv8n.

[0081] Figure 3 Flowchart for the improved DeepSORT tracing method.

[0082] Figure 4 This is a flowchart of cross-camera consistency recognition and similarity fusion determination based on ResNet-50. Detailed Implementation

[0083] This invention proposes an image recognition and monitoring system for airport runway intrusion targets. By constructing an end-to-end video analysis system, it can stably detect and track non-cooperative targets in the runway area under complex lighting and weather conditions. Combined with cross-camera consistency recognition, it achieves real-time monitoring of intrusion behavior, thereby effectively improving the safety and intelligence level of airport operations. Figure 1 As shown, the present invention adopts the following overall architecture:

[0084] The overall network architecture of this system adopts a cascaded design. Besides the video acquisition modules mounted on multiple cameras, it consists of four core modules connected serially: an image preprocessing module, an object detection module, an object tracking module, and a cross-camera re-identification module. The image preprocessing module, as the front-end processing unit, is responsible for enhancing the quality of the input video frames. It internally includes a CLAHE contrast enhancement submodule, a non-local mean denoising submodule, and a super-resolution reconstruction submodule, which work sequentially. The object detection module receives the enhanced image and uses a YOLOv8n network to output the bounding box and category information of the object. The object tracking module takes the detection results and appearance features as input and uses an improved DeepSORT algorithm to associate the same object in the temporal dimension. It internally includes a Kalman filter motion prediction submodule, an appearance feature extraction submodule, and a Hungarian matching submodule. The cross-camera re-identification module extracts deep appearance features from different camera perspectives using a ResNet-50 backbone network and performs cross-camera consistency matching of the same object using a weighted similarity metric. The modules exchange information through standardized data interfaces. The preprocessing module outputs enhanced video frames (RGB images with a size of 640×640), the detection module outputs a list containing bounding box coordinates, confidence scores, and category labels, the tracking module outputs tracking results with unique identifiers and motion trajectories, and the re-identification module outputs global identifiers across cameras. Finally, the alarm decision unit determines whether to trigger an intrusion alarm based on the duration and confidence of the target's intrusion into the runway area.

[0085] Part 1: Image Preprocessing Module.

[0086] An image preprocessing module is introduced at the input end to address the issue of inconsistent video quality under varying lighting and weather conditions at airports. Specifically, this includes:

[0087] 1. The CLAHE algorithm is used to enhance the local contrast of an image (Reference: Reza A M. Realization of the contrast limited adaptive histogram equalization (CLAHE) for real-time image enhancement[J]. Journal of VLSI signal processing systems for signal, image and video technology, 2004, 38(1): 35-44.). By limiting the gain of overly bright areas within a block, overexposure of the target in backlit scenes is avoided, while the visibility of details of the target in dark areas is enhanced; the details are as follows:

[0088] By dividing the image into 8×8 pixel blocks and limiting the contrast gain within each block (maximum gain set to 40), adaptive adjustment of local brightness is achieved, ensuring that the target remains clearly visible under low light, backlight, or uneven lighting conditions. Let the input video frame be I, which is converted to the LAB color space to obtain the luminance channel L. For the k-th local image block Ω... k Its grayscale histogram is h k (i), where the gray levels are i = 0, 1, ..., G-1. Let the cropping threshold be T. c The histogram after cropping for:

[0089]

[0090] Corresponding cumulative distribution function for:

[0091]

[0092] Sentence grayscale mapping function for:

[0093]

[0094] Then, bilinear interpolation is performed on the mapping results of adjacent image blocks to obtain the enhanced luminance channel. And synthesize an enhanced image with the original color channels. Compared to global histogram equalization, this method improves the problems of overexposure in the target area and loss of detail in the dark area in backlit scenes.

[0095] 2. A non-local means denoising method is used to suppress noise interference and improve image detail quality (Reference: Buades A, Coll B, Morel J M. A non-local algorithm for image denoising[C] / / 2005 IEEE Computer Society conference on computer vision and pattern recognition (CVPR'05). IEEE, 2005, 2: 60-65.); details are as follows:

[0096] Noise suppression is achieved based on non-local similarity between image pixels. This method removes random noise generated under high ISO or severe weather conditions while preserving image edge and texture details, thereby improving detection accuracy. The specific computational mechanism of the non-local mean denoising method is as follows:

[0097] Let the image to be denoised be pixel The denoising result is as follows:

[0098]

[0099] in, Represented in pixels The search window is centered on the search. Represents pixels For pixels The weighted contribution, or weight, satisfies:

[0100]

[0101] Weights are defined as:

[0102]

[0103] in, and ) represent the domain blocks centered at p and q, respectively. This represents the Gaussian-weighted Euclidean distance. For smoothing parameters, This is the normalization coefficient.

[0104] Ultimately, the denoising result for the current pixel is the weighted average of all pixels within the search window, calculated according to their respective weights. By utilizing redundant information from all similar structures across the entire image, rather than relying solely on local neighborhoods, this method effectively removes Gaussian noise and random noise caused by severe weather while preserving the edge contours and texture details of the target, providing high-quality input for subsequent detection stages.

[0105] 3. Improve the clarity and detail resolution of low-resolution videos through deep learning-based super-resolution reconstruction models, thereby providing better input data for subsequent object detection and recognition; specifically as follows:

[0106] A deep learning-based super-resolution reconstruction model is employed to upscale and reconstruct surveillance video frames, enhancing detail representation under low-resolution input. This method improves the detection accuracy by at least 5% for 360p video input. The super-resolution reconstruction model's network structure comprises three parts: a shallow feature extraction layer, a deep feature extraction module, and an upsampling reconstruction module. The shallow feature extraction layer consists of a 3×3 convolutional layer (3 input channels, 64 output channels) and a ReLU activation function, mapping the low-resolution input image to a high-dimensional feature space. The deep feature extraction module consists of 16 stacked residual blocks. Each residual block contains two 3×3 convolutional layers (64 channels), a batch normalization layer, and a ReLU activation function. Skip connections are introduced to element-wise add the input and output of the residual block, mitigating the gradient vanishing problem in deep networks and enabling the network to learn richer high-frequency detail information. The upsampling reconstruction module uses sub-pixel convolution for upsampling, first expanding the feature channel count through a 3×3 convolutional layer. (in This is the upsampling factor. (for the number of output channels), then... The feature map pixels of each channel are rearranged to obtain spatial resolution magnification. The output image is multiplied by a factor of 1, and finally reconstructed into a high-resolution RGB image through a 3×3 convolutional layer. The network is trained using a weighted combination of the L1 loss function and the feature map-aware loss extracted from the pre-trained VGG-19 network.

[0107] Let the low-resolution image be I. LR The corresponding image resolution is I. HR Super-resolution networks are The reconstructed image is:

[0108]

[0109] The network training loss can be written as:

[0110] in, The L1 norm is used to measure the pixel-level absolute error between the reconstructed image and the true high-resolution image. Here, is the squared L2 norm, used to measure the squared Euclidean distance between features extracted by the pre-trained VGG network. The first term is the pixel-level L1 reconstruction loss, and the second term is the perceptual loss. Let λ1 and λ2 represent the features extracted by the pre-trained VGG network. p These are the weighting coefficients.

[0111] Compared to the traditional bicubic interpolation upsampling method, the super-resolution reconstruction model of this invention can effectively recover the lost texture details and edge information in low-resolution surveillance video frames.

[0112] Part Two: Target Detection Module.

[0113] In the target detection stage, this invention employs the YOLOv8n model as the basic detection network (reference: JOCHER G, CHAURASIA A, QIU J. Ultralytics YOLO[M / OL]. 2023.). This model is lightweight and has strong real-time performance, maintaining high detection accuracy under limited computing resources. Through retraining and optimization on the airport dataset, YOLOv8n can achieve high-precision detection of people, vehicles, aircraft, and other non-cooperative targets. In experiments, the detection model used in this invention achieves an mAP@0.5 score exceeding 88% on the standard airport dataset, with an inference speed of 25 frames per second, meeting the requirements of real-time monitoring. Compared with traditional methods such as Faster R-CNN, this invention significantly improves both detection accuracy and real-time performance.

[0114] The YOLOv8n detection model consists of three parts: a backbone network, a neck network, and a detection head. The backbone network uses a modified CSPDarkNet, composed of stacked CBS convolutional, C2f, and SPPF modules. The CBS convolutional module is composed of Conv2d, BatchNorm2d, and SiLU activation functions in sequence. C2f is an improved cross-stage partial connection module that splits the input feature map channels. SPPF is located at the end, aggregating multi-scale contextual information through three cascaded 5×5 max-pooling layers. The backbone network contains five downsampling stages, outputting feature maps P1-P5, where P3 (8x downsampling), P4 (16x), and P5 (32x) are fed into the neck network. The neck network employs a PAN-FPN bidirectional feature fusion structure, adding a bottom-up enhancement branch to the top-down path of FPN, enabling more efficient transmission of low-level localization information to high-level layers. The system employs a top-down propagation approach using upsampling and C2f fusion, followed by bottom-up enhancement using downsampling, ultimately outputting detection feature maps at three scales: N3, N4, and N5. This enhances the feature representation capability for multi-scale targets. The detection head utilizes a decoupled design, separating classification and regression tasks into two independent branches (each containing two 3×3 convolutional layers and one 1×1 prediction convolutional layer), avoiding feature conflicts between tasks. Anchor-free detection directly predicts the target center point offset and bounding box width and height, eliminating the need for manual adjustment of anchor box hyperparameters. The loss function uses a binary cross-entropy classification loss and a regression loss weighted by CIoU and DFL.

[0115] Part Three: Target Tracking Module.

[0116] In the target tracking stage, this invention introduces the DeepSORT multi-target tracking algorithm as the basic tracking framework (reference: [4] Wojke N, Bewley A, Paulus D. Simple online and realtime tracking with adeep association metric[C] / / 2017 IEEE international conference on image processing (ICIP). IEEE, 2017: 3645-3649.), and makes improvements to address the significant differences in aspect ratio between vehicles and aircraft in airport scenarios. The improvements of this invention compared to the traditional DeepSORT mainly include two points: First, the size of the appearance feature input is adjusted from 128×64 to 64×128 to adapt to the lateral aspect ratio features of airport vehicles and aircraft; second, an appearance feature extraction network is self-trained using airport scene data, making the appearance embedding more suitable for non-cooperative targets such as vehicles, aircraft, and airport personnel, rather than directly using a general feature network trained based on pedestrian data.

[0117] This invention adjusts the input size of the appearance features to 64×128 to accommodate the aspect ratio characteristics of airport vehicles and aircraft, while utilizing Kalman filtering and Hungarian matching mechanisms to achieve stable tracking of multiple targets. The processing flow of the improved DeepSORT algorithm consists of the following steps.

[0118] The first step is to use the current frame detection results (bounding box coordinates and confidence scores) provided by the target detection module to predict the motion state of each tracked target using Kalman filtering, thus obtaining the predicted position of the target in the current frame. Let the state vector of the j-th trajectory in the t-th frame be:

[0119]

[0120] Where u, v are the coordinates of the bounding box center, a is the aspect ratio, and h is the height of the bounding box. These represent the corresponding rates of change.

[0121] The Kalman prediction process is as follows:

[0122]

[0123]

[0124] Where F is the state transition matrix, P is the covariance matrix, and Q is the process noise covariance matrix.

[0125] The second step is to calculate the matching cost matrix between the detection results and the prediction results. The matching cost is a weighted fusion of two parts: one is the motion matching cost based on Mahalanobis distance, which measures the consistency between the detection position and the prediction position in the motion state space, for the detection box d. i The motion consistency with trajectory Tj is represented by Mahalanobis distance:

[0126]

[0127] Among them, z i Let H be the observation vector of the detection box, and S be the observation matrix. j To predict the observation covariance matrix.

[0128] By introducing the prediction covariance matrix, Mahalanobis distance can adaptively consider the uncertainty of state estimation. Secondly, it uses appearance matching cost based on cosine distance to measure the similarity of the detected target and the tracked target in appearance features. This cost is calculated based on their respective L2-normalized appearance feature vectors (128 dimensions), with smaller values ​​indicating greater similarity between the two appearance features. Let r be the detected target features extracted by the self-trained appearance network. i The trajectory's historical appearance features are r j Both are L2 normalized, so the appearance distance is:

[0129]

[0130] The third step involves linearly fusing the Mahalanobis distance and cosine distance using weighted coefficients to obtain the comprehensive matching cost. The comprehensive matching cost is defined as follows:

[0131]

[0132] Where λϵ[0,1] are the weighting coefficients of motion features and appearance features.

[0133] The problem of matching the detected target with historical trajectories can be expressed as:

[0134]

[0135] The constraints are:

[0136]

[0137] Where aij = 1 indicates that the i-th detection box matches the j-th trajectory successfully, otherwise it is 0.

[0138] The fourth step is to input the comprehensive matching cost matrix into the Hungarian algorithm to solve for the optimal bipartite graph matching between detection and tracking. The Hungarian algorithm uses the augmented path method to find the matching scheme with the minimum total cost, thus completing the target identity association and trajectory update for the current frame.

[0139] The fifth step involves updating the motion state of successfully matched targets using Kalman filtering, initializing unmatched detected targets with new tracking trajectories (formally created after confirmation by 3 consecutive frames of detection), marking unmatched tracking targets as lost (deleting the trajectory after 30 consecutive frames of unmatched tracking), and using a cascaded matching strategy to prioritize matching tracking trajectories that have appeared for a shorter time (i.e., have lower uncertainty).

[0140] This invention also introduces an appearance feature extraction network based on self-trained data. This network uses airport video data as training samples and can extract embedding vectors that better match the characteristics of airport scenes, thus maintaining tracking continuity even when the target is occluded or moving rapidly, effectively improving the adaptability and stability of the tracking. The complete structure of the appearance feature extraction network from input to output is as follows: The input layer receives a 64×128 RGB three-channel color image, followed by two standard convolutional modules. Each module consists of a 3×3 convolutional layer (stride 1, padding 1, output channels 32 and 64 respectively), a batch normalization layer, and a ReLU activation function. Then, a 2×2 max-pooling layer (stride 2) reduces the feature map spatial resolution to 32×64. Following this are six stacked residual modules. Each residual module contains two branches. The main branch consists of two 3×3 convolutional layers (stride 1, padding 1, output channels 64) sequentially. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The branch is a 1×1 convolutional layer (stride 1, output channels 64, used to match the number of channels output by the main branch). The outputs of the main branch and the residual branch are added element-wise and then output through the ReLU activation function. The output feature map size of each residual module is the same as the input, maintaining a spatial resolution of 32×64 and 64 channels. Then, a 2×2 max pooling layer reduces the feature map to 16×32. Six residual modules with the same structure as above are stacked (output channels 128). Then, a global average pooling layer compresses the spatial dimension to 1×1, resulting in a 128-dimensional feature vector. Finally, a fully connected layer, a batch normalization layer, and an L2 normalization layer are added to output the final 128-dimensional normalized appearance embedding vector. During the training phase, the network employs two loss functions for joint supervision: a cross-entropy loss function, which calculates the cross-entropy loss between the predicted identity distribution and the real identity label by inputting a 128-dimensional embedding vector into a fully connected classification layer (the number of output nodes equals the total number of target identities labeled in the training set); and a triplet loss function, which constructs anchor sample, positive sample, and negative sample triplets in the embedding space. The loss function is based on the difference between the Euclidean distance between the anchor sample and the positive sample, and the difference between the anchor sample and the negative sample, and introduces an interval threshold as a constraint. A hard sample mining strategy is used during training, selecting the most difficult-to-distinguish positive and negative sample pairs in each training batch to construct triplets, thereby accelerating network convergence and improving the discriminative ability of the embedded features. Through end-to-end training on airport surveillance video data, the network learns more discriminative representations of airport target appearance features, significantly outperforming traditional general appearance feature extractors trained on pedestrian datasets in feature extraction for non-pedestrian targets such as vehicles and aircraft.

[0141] Experiments show that, in airport scenarios, the improved DeepSORT of this invention significantly outperforms traditional methods in multi-target tracking accuracy (MOTA) under daytime, nighttime, and rainy conditions, and can still maintain high tracking continuity under complex conditions such as occlusion and rapid target movement.

[0142] Part Four: Cross-Camera Re-identification Module.

[0143] In the cross-camera consistency recognition stage, this invention employs a ResNet-50-based re-identification model to extract the target's depth features and combines weighted similarity using Euclidean and cosine distances for cross-camera matching. This method enables accurate identification and association of the same target across different cameras, resolving the issue of misjudgment common in traditional methods in cross-camera scenarios. In practical experiments, the re-identification module of this invention achieves an accuracy rate exceeding 93% in cross-camera consistency recognition tasks, significantly improving the robustness of the surveillance system.

[0144] Let the video image captured by the m-th camera in the t-th frame be... The i-th detection box output by the target detection module is The corresponding target region image is:

[0145]

[0146] in, This indicates the operation of cropping the original video image based on the object detection bounding box; the target region is input into the ResNet-50 feature extraction network. As the backbone network for feature extraction in the cross-camera re-identification module, its architecture sequentially includes an initial convolutional layer Conv1 (7×7 convolution, batch normalization, and ReLU), a 3×3 max pooling layer to reduce the input image resolution from 256×128 to 64×32, followed by four feature extraction stages consisting of stacked residual modules. Each stage progressively downsamples through convolutions with a stride of 2, ultimately outputting an 8×4 feature map with 2048 channels. Each residual module employs a three-layer residual structure of compression, convolution, and dilation. 1×1 convolutions are used to compress and restore the number of channels, reducing computational cost. Residual connections use 1×1 convolutions for alignment when dimensions are mismatched; otherwise, identity mapping is used, effectively mitigating the degradation problem of deep networks. The output feature map is then subjected to global average pooling to obtain a 2048-dimensional feature vector, which is further reduced to 512 dimensions by a fully connected layer. After batch normalization and L2 normalization, the final 512-dimensional normalized depth appearance feature vector is obtained. .

[0147]

[0148]

[0149] During the training phase, a dataset was constructed for multiple target categories, including airport personnel, vehicles, and aircraft. Each target identity was labeled with at least 10 image samples from different cameras at different times. A joint supervision strategy of cross-entropy loss and triplet loss was adopted. The former uses a fully connected classification layer to supervise identity classification, ensuring linear separability of different identity features. The latter constructs anchor sample-positive sample-negative sample triplets to constrain features of the same target to be close together and features of different targets to be far apart. The total loss is the sum of the two. Data enhancements such as random erasure, random cropping, and color jitter were also used to improve model robustness. For cross-camera matching, 512-dimensional normalized features were extracted from target images from different cameras. Euclidean distance (reflecting absolute amplitude differences) and cosine distance (reflecting directional differences) were calculated. A weighted fusion strategy was used to obtain a comprehensive similarity score, with cosine distance having a larger weight to prioritize directional consistency. When the comprehensive score exceeds a threshold, the target is determined to be of the same identity, completing the cross-camera target association.

[0150] Example 1:

[0151] This invention proposes a practical image recognition and monitoring system for airport runway intrusion targets. The system can run in a GPU-accelerated server environment. Hardware configuration includes an Intel i7 or higher CPU, 16GB of RAM, and an NVIDIA RTX 3060 graphics card. The software environment is developed using Python, with the PyTorch deep learning framework. The system supports running on Linux or Windows and can connect to airport video surveillance systems via LAN, enabling integration with existing airport management platforms. The overall system architecture is as follows: Figure 1 As shown, the system mainly includes a video acquisition module, an image preprocessing module, a target detection module, a target tracking module, and a cross-camera consistency recognition module.

[0152] The video acquisition module is used to acquire surveillance video data of the airport runway and surrounding areas. The video resolution can be set to 720p or 1080p, with a frame rate of 25 frames per second, and supports simultaneous input from multiple cameras. The acquired video is first input into the image preprocessing module. This module uses the CLAHE algorithm for illumination normalization to ensure clear visibility of targets in uneven lighting and nighttime environments; it uses a non-local means method to denoise video frames, suppressing noise caused by high ISO and weather conditions; and it improves video clarity and resolution through a deep learning-based super-resolution reconstruction method so that the subsequent detection module can operate on higher-quality images.

[0153] The object detection module is implemented based on the YOLOv8n model, which balances lightweight design with accuracy. By retraining with a specific airport scene dataset, it is able to detect non-cooperative targets such as people, vehicles, and aircraft. The model employs a feature extraction method combining convolution and attention mechanisms, outputting detection results including object category and bounding box.

[0154] The target tracking module employs an improved DeepSORT algorithm. Unlike the traditional DeepSORT which uses 128×64 feature input, this invention adjusts the input to 64×128 to adapt to the aspect ratios of vehicles and aircraft. Simultaneously, this invention self-trains an appearance feature extraction network based on airport video data. The extracted features are used for Kalman filtering and Hungarian matching mechanisms, thereby maintaining tracking continuity and robustness even when the target is moving rapidly or partially occluded.

[0155] The cross-camera consistency recognition module is built on a ResNet-50 deep convolutional neural network. It extracts feature vectors from targets under different cameras and performs cross-camera matching by combining a weighted similarity formula of Euclidean distance and cosine distance. This method can effectively handle problems such as differences in lighting and imaging angles, and achieve consistent confirmation of the same target under different cameras.

[0156] The specific implementation scheme of the image preprocessing module is as follows:

[0157] In complex airport environments, lighting conditions change drastically, and surveillance videos often suffer from noise and low resolution. To address these issues, this invention designs a multi-step image preprocessing workflow.

[0158] First, the CLAHE algorithm is used to process the input video frames. This algorithm separates the reflection and illumination components, enabling the image to maintain stable brightness even under uneven lighting conditions. For example, in nighttime or backlit environments, CLAHE can significantly enhance the contrast of the target area, thereby improving the input quality of the detection model.

[0159] Secondly, a nonlocal means denoising method is used to smooth the video frames. This method removes noise while preserving edge details by calculating the nonlocal similarity between pixels. Experiments show that the image after nonlocal means processing improves both PSNR and SSIM metrics, providing clearer input for subsequent detection.

[0160] Finally, low-resolution videos are upscaled and reconstructed using deep learning-based super-resolution reconstruction models (such as EDSR or ESRGAN) to improve detail. In the experiments, the average accuracy of the detection model improved by approximately 5% as the resolution was increased from 360p to 720p.

[0161] like Figure 2 As shown, the object detection module uses the YOLOv8n network. By retraining on an airport scene dataset, the model can identify multiple non-cooperative targets, including people, vehicles, and aircraft. In experiments, the YOLOv8n model achieved an mAP@0.5 of 88.3% and a recall of 85.9% in the airport scene, with an inference frame rate of approximately 25 fps, meeting real-time requirements. In contrast, the traditional Faster R-CNN model only achieved an mAP@0.5 of 82.7% in the same scene, with an inference speed of less than 2 fps, failing to meet the needs of real-time monitoring.

[0162] like Figure 3 As shown, the target tracking module is based on an improved DeepSORT algorithm. By adjusting the feature input size to 64×128 and introducing a self-trained appearance network, the system significantly outperforms the original algorithm in terms of tracking stability. Experiments show that in daytime scenes, the improved DeepSORT of this invention achieves 83.9% on the MOTA metric, 78.8% in nighttime scenes, and 64.8% in rainy conditions, all of which are higher than the performance of the original SORT algorithm. Especially in nighttime and low visibility conditions, the improved algorithm exhibits stronger robustness.

[0163] like Figure 4 As shown, the cross-camera recognition module uses ResNet-50 to extract target features. For airport scenarios, this invention constructs a dataset with cross-camera labels and achieves feature alignment across cameras through supervised training. For the same target under different cameras, the system calculates its weighted similarity using Euclidean distance and cosine distance. When the similarity exceeds a set threshold, the system determines that the target is the same object. Experiments show that this method achieves an accuracy of 93% in cross-camera consistency recognition tasks, outperforming traditional methods that only use a single distance metric.

[0164] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding an image recognition and monitoring system for airport runway intrusion targets, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0165] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MCU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0166] This invention provides a concept and method for an image recognition monitoring system for airport runway intrusion targets. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. An image recognition and monitoring system for airport runway intrusion targets, characterized in that, include: The system includes a video acquisition module, an image preprocessing module, a target detection module, a target tracking module, and a cross-camera consistency recognition module; among which, The video acquisition module is installed on multiple cameras in the target area to capture raw video streams; The image preprocessing module preprocesses the original video stream, including local image contrast enhancement, non-local mean denoising, and super-resolution reconstruction. The target detection module detects targets in the preprocessed video image and obtains the target category, confidence score and corresponding target bounding box; The target tracking module performs target tracking based on the target detection results and obtains the tracking trajectory; The cross-camera consistency recognition module performs calculations and recognition based on the target detection results and tracking trajectory to complete the cross-camera target association.

2. The image recognition and monitoring system for airport runway intrusion targets according to claim 1, characterized in that, The image local contrast enhancement includes: The input video frame is divided into pixel blocks of a preset size and the contrast gain limit within the block is set; Let the input video frame be I, and after converting it to the LAB color space, we get the luminance channel L; For the A pixel block, its grayscale histogram is grayscale level is , , This represents the number of gray levels; let the cropping threshold be... The histogram after cropping It is expressed as follows: ； Calculate the corresponding cumulative distribution function , means as follows: ； Calculate the local gray-scale mapping function , means as follows: ； Bilinear interpolation is performed on the mapping results of adjacent pixel blocks to obtain the enhanced luminance channel. And synthesize an enhanced image with the original color channels. As output.

3. The image recognition and monitoring system for airport runway intrusion targets according to claim 2, characterized in that, The aforementioned nonlocal mean denoising includes: Let the input image to be denoised be pixel The denoising results are shown below: ； in, Represented by pixels The search window is centered on the search. Represents pixels For pixels The weighted contribution, or weight, satisfies: ； Weight The calculation method is as follows: ； in, and ) represent the neighborhood blocks centered at pixels p and q, respectively. This represents the Gaussian-weighted Euclidean distance. For smoothing parameters, These are the normalization coefficients; The denoising result of the current pixel is the weighted average of all pixels in the search window, calculated according to their weights.

4. The image recognition and monitoring system for airport runway intrusion targets according to claim 3, characterized in that, The super-resolution reconstruction includes: A super-resolution reconstruction model is constructed, whose network structure includes a shallow feature extraction layer, a deep feature extraction module, and an upsampling reconstruction module. The shallow feature extraction layer consists of a 3×3 convolutional layer and a ReLU activation function, mapping the input low-resolution image to a high-dimensional feature space. The deep feature extraction module consists of 16 stacked residual blocks, each containing two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function, with skip connections introduced to element-wise add the input and output of the residual block. The upsampling reconstruction module uses sub-pixel convolution to achieve upsampling. The super-resolution reconstruction model is trained using a weighted combination of the L1 loss function and a feature map-aware loss extracted based on a pre-trained VGG-19 network. Let the low-resolution image be The corresponding image resolution is The super-resolution reconstruction model is represented as Then reconstruct the image It is expressed as follows: ； Training loss of super-resolution reconstruction model , means as follows: ; in, It is an L1 norm. It is the square L2 norm. This represents the features extracted by the pre-trained VGG network. and These are the weighting coefficients.

5. The image recognition and monitoring system for airport runway intrusion targets according to claim 4, characterized in that, The detection of targets in the preprocessed video image includes: A YOLOv8n detection model was constructed, comprising a backbone network, a neck network, and a detection head. The backbone network uses an improved CSPDarkNet, consisting of stacked CBS convolutional, C2f, and SPPF modules. The CBS convolutional module is composed of Conv2d, BatchNorm2d, and SiLU activation functions in sequence. The C2f module is an improved cross-stage partial connection module that splits the input feature map channels. The SPPF module is located at the end and aggregates multi-scale contextual information through three cascaded 5×5 max-pooling layers. The backbone network contains five downsampling stages, and incorporates 8x downsampling and 16x downsampling... The 1x and 32x downsampling results are fed into the neck network. The neck network adopts a PAN-FPN bidirectional feature fusion structure, adding a bottom-up enhancement branch to the top-down path of the FPN network. It completes the top-down propagation through upsampling concatenation and C2f fusion, and then completes the bottom-up enhancement through downsampling concatenation, finally outputting detection feature maps at three scales. The detection head separates the classification and regression tasks into two independent branches, each containing two layers of 3×3 convolutions and one layer of 1×1 prediction convolution, finally predicting the target center point offset and bounding box width and height. The loss function of the YOLOv8n detection model adopts the binary cross-entropy classification loss and the regression loss of the CIoU and DFL weighted combination. The YOLOv8n detection model is trained and optimized using the airport dataset. The preprocessed video image is input into the optimized YOLOv8n detection model to obtain the target category, confidence score, and corresponding target bounding box.

6. The image recognition and monitoring system for airport runway intrusion targets according to claim 5, characterized in that, The target tracking based on the target detection results includes: Using the target bounding box and confidence score of the current frame detection results provided by the target detection module, the motion state of each tracked target is predicted by Kalman filtering to obtain the predicted position of the target in the current frame; Calculate the matching cost matrix between the detection result and the predicted location, including Mahalanobis distance and cosine distance; The Mahalanobis distance and appearance distance are linearly fused using weighted coefficients to obtain the comprehensive matching cost; The augmented path method is used to find the matching scheme with the minimum total cost based on the comprehensive matching cost method, and the target identity association and trajectory update of the current frame are completed. For targets whose identities are successfully associated, Kalman filtering is used to update their motion state. For unmatched detected targets, a new tracking trajectory is initialized or they are marked as lost.

7. The image recognition and monitoring system for airport runway intrusion targets according to claim 6, characterized in that, The appearance distance is calculated based on the target features obtained by the appearance feature extraction network of the self-trained data.

8. The image recognition and monitoring system for airport runway intrusion targets according to claim 7, characterized in that, The self-trained data appearance feature extraction network includes: The input layer receives RGB three-channel color images with a size of 64×128; After the input layer are two standard convolutional modules, each consisting of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function; The feature map spatial resolution was then reduced to 32×64 using a 2×2 max pooling layer. Then there are 6 stacked residual modules. Each residual module contains two branches. The main branch consists of two 3×3 convolutional layers in sequence. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The residual branch is a 1×1 convolutional layer. The outputs of the main branch and the residual branch are added element by element and then output through the ReLU activation function. Then, a 2×2 max pooling layer is used to reduce the feature map to 16×32, and then 6 residual modules are stacked. Subsequently, the spatial dimension is compressed to 1×1 by a global average pooling layer, resulting in a 128-dimensional feature vector. Finally, after passing through a fully connected layer, a batch normalization layer, and an L2 normalization layer, the final 128-dimensional normalized appearance embedding vector is output, which is the feature of the detected target. During the training phase, airport video data was used as training samples, and joint supervision was carried out using the cross-entropy loss function and the triplet loss function.

9. The image recognition and monitoring system for airport runway intrusion targets according to claim 8, characterized in that, The method involves calculating and recognizing targets based on target detection results and tracking trajectories, using a ResNet-50-based re-identification model to extract the target's depth and appearance features, and combining weighted similarity based on Euclidean distance and cosine distance for cross-camera matching.

10. The image recognition and monitoring system for airport runway intrusion targets according to claim 9, characterized in that, The calculation and recognition based on the target detection results and tracking trajectory includes: Let the first The camera at the The video images captured in frames are The first output of the target detection module Each detection box is ,in, , , and Let x, y, width, and height be the origin, x, y, width, and height of the detection box, respectively. Then the corresponding target region image is: , means as follows: ； in, This indicates a cropping operation on the original video image based on the object detection bounding box; The target region image is input into the ResNet-50 feature extraction network. Then, normalization is performed to obtain the normalized depth appearance feature vector. , means as follows: ； ； The ResNet-50 feature extraction network includes: The initial convolutional layer Conv1 and a 3×3 max-pooling layer, connected sequentially, reduce the input image resolution from 256×128 to 64×32. This is followed by four feature extraction stages consisting of stacked residual modules. Each stage uses convolutions with a stride of 2 to progressively downsample the image, ultimately outputting an 8×4 feature map with 2048 channels. ; The feature map is subjected to global average pooling to obtain a 2048-dimensional feature vector, which is then reduced to 512 dimensions by a fully connected layer. After batch normalization and L2 normalization, the final 512-dimensional normalized depth appearance feature vector is obtained. ; When matching across cameras, 512-dimensional normalized depth appearance feature vectors are extracted from the target images acquired by different cameras. Euclidean distance, which reflects the difference in absolute magnitude, and cosine distance, which reflects the difference in direction, are calculated. A weighted fusion strategy is used to obtain a comprehensive similarity score. When the comprehensive score exceeds the threshold, the target is determined to be the same person, thus completing the cross-camera target association.

Images