A method for airport runway line detection based on multi-task learning

Abstract

The present invention belongs to the field of computer vision technology and specifically discloses a method for detecting airport runway lines based on multi-task learning. The method comprises the following steps: acquiring runway image data; inputting the runway image data into a YOLOv8 target detection module, performing runway line detection on the runway image data using the YOLOv8 target detection module, and extracting the area to be detected from the runway image data; inputting the area to be detected into a coarse positioning region tracking module, which outputs the target runway area within the area to be detected; and inputting the target runway area into a runway feature detection module, which outputs runway information corresponding to the runway image data. The present invention can improve the accuracy of runway recognition in complex scenes, resolving the problem that existing technologies are not robust enough in complex scenes.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to an airport runway line detection method based on multi-task learning. Background Art

[0002] In airport landing scenarios, accurate and rapid runway line detection technology is essential for ensuring a safe landing. With the development of computer vision and image processing technologies, visual navigation has gradually been incorporated into navigation systems. Compared to traditional navigation systems, visual navigation offers advantages such as strong autonomy, strong anti-interference capabilities, low cost, and increased accuracy with decreasing distance. The autonomous drone landing algorithm based on visual information uses real-time runway environment images as input, and the onboard computer rapidly identifies and detects runway information.

[0003] Current runway line detection algorithms fall into two categories: those based on traditional algorithms and those based on deep learning. Airport runway line detection based on traditional algorithms relies on manually designed features and the use of traditional machine learning algorithms for target detection. This process primarily extracts local features from images for image feature matching or image classification. The process can be divided into three steps: feature extraction, feature description, and feature matching. Although traditional target detection algorithms perform well in certain scenarios, they are often less robust to challenges such as scale changes, perspective changes, occlusion, and complex backgrounds. With the rise of deep learning and the improvement of hardware parallel computing capabilities, target detection methods based on deep learning are able to better learn abstract image features and achieve better detection performance in practical applications. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the existing technology, the present invention provides an airport runway line detection method based on multi-task learning, which solves the problem that the existing technology is not robust enough in complex scenarios.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a method for detecting airport runway lines based on multi-task learning, comprising the following steps:

[0006] S1. Acquire runway image data;

[0007] The runway image data is continuous video frame images;

[0008] S2. Input the runway image data into the YOLOv8 target detection module, perform runway line detection on the runway image data through the YOLOv8 target detection module, and extract the area to be detected in the runway image data;

[0009] The YOLOv8 target detection module includes a first feature extraction module, a first feature fusion module and a detection head;

[0010] The first feature extraction module is used to extract features from the runway image data and fuse the extracted feature maps to obtain an initial fused feature map;

[0011] The first feature fusion module is used to further fuse the fused feature map to obtain a fused feature map;

[0012] The detection head is used to detect the fused feature map and output the area to be detected;

[0013] S3, inputting the area to be detected into a coarse positioning area tracking module, determining the target to be tracked in the area to be detected in the initial frame through the coarse positioning area tracking module, performing runway target tracking on the area to be detected after the initial frame based on the target to be tracked, and outputting the target runway area in the area to be detected;

[0014] The algorithm of the coarse positioning area tracking module is the kernel correlation filter target tracking algorithm;

[0015] S4. Input the target runway area into the runway feature detection module, and output the runway information corresponding to the runway image data through the runway feature detection module;

[0016] The runway feature detection module includes a second feature extraction module, a second feature fusion module, a semantic segmentation branch, and a target detection branch;

[0017] The second feature extraction module is used to extract feature maps of different resolutions of the target runway area;

[0018] The second feature fusion module is used to fuse feature maps of different resolutions to obtain fused features of different scales;

[0019] The semantic segmentation branch is used to segment and identify fused features of different scales to obtain runway line recognition results;

[0020] The target detection branch is used to downsample the output of fused features of different scales and perform further fusion and recognition to obtain the runway number recognition results and runway entrance sign recognition results.

[0021] The beneficial effects of the above scheme are as follows: the present invention focuses on the runway range through the YOLOv8 target detection module to achieve coarse runway positioning, thereby improving the efficiency and accuracy of subsequent runway line and runway marking detection. The coarse positioning area tracking module is based on continuous frames of the video stream, and on the basis of the target pixel position of the previous frame, the motion model is used to mine the pixel movement relationship between the front and back, thereby achieving target tracking, solving the problems of missed detection and false detection while ensuring globality, suppressing pixel position jumps, and combining with the YOLOv8 target detection module to achieve continuous, stable and fast runway coarse positioning effect. The runway feature detection module considers the learning process of multiple related tasks, utilizes useful information between multiple related learning tasks and shares features between related tasks, and improves the performance of each subtask.

[0022] Furthermore, in S2, the first feature extraction module includes a C2f module, a CBS module, and an SPPF module;

[0023] The first feature extraction module extracts features from the runway image data and fuses the extracted feature maps to obtain an initial fused feature map as follows:

[0024] S21, extracting a bottom-level feature map and a high-level feature map of the runway image data through the C2f module, and fusing the bottom-level feature map and the high-level feature map to obtain a first feature map;

[0025] S22. Convolve the first feature map through a CBS module to obtain a second feature map;

[0026] S23. Superimpose the second feature map through the SPPF module to obtain an initial fused feature map.

[0027] The beneficial effects of this further solution are as follows: the C2f module combines feature maps at different levels through multi-scale feature fusion, improving the model's detection capabilities for objects of different scales. The CBS module obtains richer gradient flow information while ensuring lightweightness. Its activation function is a sigmoid weighted linear combination, which is continuous and differentiable, making the network nonlinear and thus improving its expressive power. The SPPF module fuses the multi-scale features of an image by superimposing feature maps at different scales, thereby increasing the network's detection accuracy.

[0028] Furthermore, in S2, the structure of the first feature fusion module is a combination of FPN and PANI9;

[0029] The detection head is a decoupling detection head.

[0030] The beneficial effect of this further solution is that the structure of the first feature fusion module upsamples deep features and concatenates them with upper-layer features, then downsamples and further fuses them with the deep features. This achieves a repeated fusion of deep semantic information and upper-layer texture information, improving the network's detection of objects of different scales. By using two branches to predict the bounding box position and object type respectively, the structure that does not share parameters between different detection heads can accelerate convergence and improve the algorithm's prediction accuracy.

[0031] Furthermore, in S2, the loss function of the YOLOv8 object detection module includes classification loss and bounding box prediction loss;

[0032] The classification loss is binary cross entropy loss;

[0033] Bounding box prediction losses include CIoU loss and Distribution Focal loss;

[0034] Among them, CIoU loss L CIOU for:

[0035]

[0036] Among them, i represents a constant, which is generally 1 according to the definition of the loss function, IoU represents the intersection-over-union ratio of the bounding box and the real box, b represents the center point of the bounding box, and b gt represents the center point of the true box, ρ represents the Euclidean distance between the center point of the bounding box and the center point of the true box, c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box, α is the penalty factor for the aspect ratio, and v represents the similarity of the aspect ratio. w gt and w represent the width of the real box and the predicted box respectively, h gt and h represent the heights of the true box and the predicted box respectively, β represents the penalty factor of the center point distance, and d represents the feature length of the predicted box and the true box.

[0037] The beneficial effect of the above further solution is that the loss function of the YOLOv8 object detection module can make the network more flexible in predicting the position of the bounding box, and improve the accuracy of the network's prediction of objects with unclear boundaries.

[0038] Furthermore, in S3, the algorithm used by the coarse positioning area tracking module is a kernel correlation filtering algorithm.

[0039] The beneficial effect of the above further solution is that the combination of the kernel correlation filter algorithm and the YOLOv8 target detection module can achieve continuous, stable and fast runway coarse positioning effect.

[0040] Furthermore, in S4, the second feature extraction module includes a Focus module, a C2f module, a CBS module, and an SPPF module;

[0041] The second feature fusion module uses the SPPF module and FPN network.

[0042] Furthermore, in S4, the loss function of the runway feature detection module includes bounding box prediction loss, classification loss, confidence loss, pixel segmentation category loss, and pixel segmentation CloU loss;

[0043] Bounding box prediction loss L box for:

[0044]

[0045] Among them, N represents the number of detected targets in a frame image, CIoU represents the intersection-over-union ratio of the predicted target box and the real target box, and Pbox i Indicates the position of the i-th predicted target box, Tbox i Represents the true value of the position of the i-th predicted target box, L BCE represents the binary cross entropy loss;

[0046] Classification loss L cls for:

[0047]

[0048] Among them, Pcls i Indicates the category of the i-th predicted target box, Tcls i Represents the true value of the category of the i-th predicted target box;

[0049] Confidence loss L obj for:

[0050]

[0051] Among them, Pobj i Represents the confidence of the i-th predicted target box, Tobj i represents the true value of the confidence of the i-th predicted target box, balance represents the balance coefficient, which is used to balance the contribution of positive and negative samples, and balance t Indicates the balance coefficient when the training step is t, balance0 indicates the initial balance coefficient, N pos and N neg They represent the number of positive and negative samples when the training step is t, t represents the number of training steps, and T represents the total number of training steps;

[0052] Pixel segmentation category loss L seg for:

[0053]

[0054] Among them, L CE (Pseg, Tseg) represents the cross entropy loss, Pseg represents the semantic segmentation prediction result, Tseg represents the semantic segmentation annotation truth value, and η represents the balance factor between the cross entropy loss and the Dice loss;

[0055] Pixel segmentation CloU loss L iou_l for:

[0056]

[0057] Among them, CIoU seg Represents the intersection-over-union ratio of the semantic segmentation prediction result and the semantic segmentation annotation truth value, ρ 2 (S,S gt ) represents the shape difference metric between the semantic segmentation prediction area and the semantic segmentation annotation area, A represents the reference area, and ρ 2 (E,E gt ) represents the edge difference measure between the semantic segmentation prediction area and the semantic segmentation annotation area, P represents the reference perimeter, and γ represents the weight coefficient.

[0058] The beneficial effect of this further solution is that the Focus module uses a slicing operation to split the high-resolution feature map into multiple low-resolution feature maps, stacking the feature information across channels to ensure that data information is not lost during downsampling. The SPPF module is used to generate and fuse features at different scales, while the FPN network fuses features at different semantic levels, ensuring that the generated features contain information at multiple scales and semantic levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 The figure is a flowchart of an airport runway line detection method based on multi-task learning.

[0060] Figure 2 Schematic diagram of the kernel correlation filtering algorithm.

[0061] Figure 3 This is a structural diagram of the runway feature detection module.

[0062] Figure 4 Schematic diagram of the process of extracting runway line parameters.

[0063] Figure 5 Schematic diagram of the process for runway threshold marking identification. DETAILED DESCRIPTION

[0064] The exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the accompanying drawings are merely exemplary and are intended to illustrate the principles and spirit of the present invention, rather than to limit the scope of the present invention.

[0065] like Figure 1 As shown in FIG, a method for detecting airport runway lines based on multi-task learning includes the following steps:

[0066] S1. Acquire runway image data; the runway image data is continuous video frame images.

[0067] For example, the runway image data may be image information obtained by a forward-looking monocular camera fixedly mounted in front of the aircraft in the flight direction. The image acquisition frame rate may be 30 frames per second, the image type may be an RGB color image, and the image resolution may be 640*640 pixels.

[0068] S2. Input the runway image data into the YOLOv8 target detection module, perform runway line detection on the runway image data through the YOLOv8 target detection module, and extract the area to be detected in the runway image data;

[0069] The YOLOv8 target detection module includes a first feature extraction module, a first feature fusion module and a detection head;

[0070] The first feature extraction module is used to extract features from the runway image data and fuse the extracted feature maps to obtain an initial fused feature map;

[0071] The first feature fusion module is used to further fuse the fused feature map to obtain a fused feature map;

[0072] The detection head is used to detect the fused feature map and output the area to be detected.

[0073] In this embodiment, in S2, the first feature extraction module includes a C2f module, a CBS module, and an SPPF module;

[0074] The first feature extraction module extracts features from the runway image data and fuses the extracted feature maps to obtain an initial fused feature map as follows:

[0075] S21, extracting a bottom-level feature map and a high-level feature map of the runway image data through the C2f module, and fusing the bottom-level feature map and the high-level feature map to obtain a first feature map;

[0076] S22. Convolve the first feature map through a CBS module to obtain a second feature map;

[0077] S23. Superimpose the second feature map through the SPPF module to obtain an initial fused feature map.

[0078] In this embodiment, in S2, the structure of the first feature fusion module is a combined structure of FPN and PANI9; the detection head is a decoupled detection head.

[0079] In this embodiment, in S2, the loss function of the YOLOv8 object detection module includes classification loss and bounding box prediction loss;

[0080] The classification loss is binary cross entropy loss;

[0081] Bounding box prediction losses include CIoU loss and Distribution Focal loss;

[0082] Among them, CIoU loss L CIOU for:

[0083]

[0084] Among them, i represents a constant, which is generally 1 according to the definition of the loss function, IoU represents the intersection-over-union ratio of the bounding box and the real box, b represents the center point of the bounding box, and b gt represents the center point of the true box, ρ represents the Euclidean distance between the center point of the bounding box and the center point of the true box, c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box, α is the penalty factor for the aspect ratio, and v represents the similarity of the aspect ratio. w gt and w represent the width of the real box and the predicted box respectively, h gt and h represent the heights of the true box and the predicted box respectively, β represents the penalty factor of the center point distance, and d represents the feature length of the predicted box and the true box.

[0085] For example, when capturing runway image data, the scale and perspective of the runway image captured by an aircraft's onboard camera will change significantly, and various interferences may occur in the surrounding environment. Therefore, directly detecting runway lines and runway markings in the original image will result in a high number of false detections and low detection accuracy. Using coarse runway line positioning offers the advantages of greater scale adaptability, improved real-time performance and improved accuracy, and enhanced scene adaptability.

[0086] YOLOv8 is an advanced object detection algorithm and the latest version of the YOLO (You Only Look Once) series. Based on the YOLOv8 object detection algorithm, the YOLOv8 object detection module selects the pixel locations of the runway from the original image captured by the camera and extracts the runway image from the original image through cropping and scaling.

[0087] The YOLOv8 algorithm supports a variety of visual tasks, such as object detection, segmentation, and pose estimation. The YOLOv8 object detection module includes the first feature extraction module (Backbone), the first feature fusion module (Neck), and the detection head (Head). The first feature extraction module consists of the C2f module, the CBS module, and the SPPF module. The C2f module uses a split operation to split the input features into two parts based on the number of channels. One part is passed through n layers of Bottleneck to extract high-level features, and the other part is directly superimposed with the final high-level features to fuse the low-level feature map with the high-level feature map. In the YOLOv8 network, the C2f module (Convolutional to Feature module) is an important component and plays a key role in improving model performance and efficiency. The CBS module (Convolution BatchNorm SiLU) is a commonly used convolutional block consisting of a convolution layer, a batch normalization layer, and the Sigmoid Linear Unit (SiLU) activation function. Compared to the ReLU function, the SiLU function has a smoother curve when approaching zero. The SPPF module (Spatial Pyramid Pooling-Fast) is improved on the basis of spatial pyramid pooling (SPP). It fuses the multi-scale features of the image by superimposing feature maps at different scales, reducing the amount of calculation and improving the computational efficiency.

[0088] The Feature Pyramid Network (FPN) is a multi-scale feature fusion method primarily used for object detection. It builds a feature pyramid to fuse feature maps of different scales, thereby improving the model's detection capabilities for objects of varying sizes. The Path Aggregation Network with Improved 9-layer (PANI9) is an improved version of the feature fusion network that further enhances the feature fusion effect by adding more paths and layers.

[0089] The loss function of the YOLOv8 target detection module consists of two parts: classification loss and bounding box prediction loss. cls Binary cross entropy loss (BCE) is used, and the bounding box prediction loss is calculated by CIoU loss L CIOU (CloULoss) and Distribution Focal Loss L DFL(Distribution Focal Loss). Distribution Focal loss regresses the bounding box position in a probabilistic way. By setting the size of the hyperparameter reg_max, the position of the bounding box is mapped to a fixed value of 0 to reg, and then discretized by the softmax function to obtain the probability relative to each fixed value. Finally, L is calculated by cross loss. DFL . L CIOU The aspect ratio of the bounding box is taken into account when calculating the intersection-over-union ratio.

[0090] The YOLOv8 target detection module introduces the Task-Aligned Assigner positive and negative sample assignment strategy during training. This strategy weights the category scores s and CIoU scores u predicted by all pixels to obtain weighted scores, and regards the first k samples after the score sorting as positive samples. CIOU The higher the value of t, the closer it is to 1. At this time, the predicted bounding box matches the annotated box more closely, that is, it meets the criteria of a positive sample. Through training, the network can be guided to dynamically focus on high-quality positive samples. The weighted score calculation formula is as follows:

[0091] t=s α ×u β

[0092] Where t represents the weighted score, s represents the category score, u represents the CIoU score, and α and β are hyperparameters used to balance the importance of classification and localization, respectively.

[0093] S3. Input the area to be detected into the coarse positioning area tracking module, determine the target to be tracked in the area to be detected in the initial frame through the coarse positioning area tracking module, perform runway target tracking on the area to be detected after the initial frame based on the target to be tracked, and output the target runway area in the area to be detected; the algorithm of the coarse positioning area tracking module is the kernel correlation filter target tracking algorithm.

[0094] In this embodiment, in S3, the algorithm used by the coarse positioning area tracking module is the kernel correlation filtering algorithm.

[0095] For example, the kernelized correlation filter algorithm (KCF) first uses a circulant matrix to generate training samples in the initial frame, where the original base samples are regarded as positive samples and the other samples generated by the circulant matrix are negative samples. These samples can be easily calculated using the properties of fast Fourier transform and Fourier diagonalization, and the trained tracker can generate the desired distribution form when acting on these samples. During the tracking process, the kernel correlation filter algorithm uses the trained tracker to perform filtering calculations on the window area to obtain an output distribution map, and the maximum response position is used as the center position of the predicted target. For the tracker, the new tracking result is used as the base sample to generate a circulant matrix to obtain a new sample set, and the tracker is updated through online training.

[0096] like Figure 2 As shown, Figure 2 The figure below is a flowchart of the kernel correlation filtering algorithm. The kernel correlation filtering algorithm first starts the target detection thread and determines the target to be tracked in the initial frame based on the detected runway. The target tracking thread then completes initialization using the target to be tracked. After the tracking thread is initialized, the tracking thread continuously tracks the runway target in the image. To suppress tracking drift, the target to be tracked in the tracking algorithm needs to be updated. After the target to be tracked is selected in the initial frame, the updated target is treated as a template image and its eigenvalues ​​(HOG features) are calculated. After the tracking thread completes tracking for each frame, the eigenvalues ​​of the tracking result are calculated and compared with the template eigenvalues. If the similarity is greater than the update threshold, the target tracking thread continues to run. Otherwise, the target detection thread runs to update the tracked target.

[0097] S4. Input the target runway area into the runway feature detection module, and output the runway information corresponding to the runway image data through the runway feature detection module;

[0098] The runway feature detection module includes a second feature extraction module, a second feature fusion module, a semantic segmentation branch, and a target detection branch;

[0099] The second feature extraction module is used to extract feature maps of different resolutions of the target runway area;

[0100] The second feature fusion module is used to fuse feature maps of different resolutions to obtain fused features of different scales;

[0101] The semantic segmentation branch is used to segment and identify fused features of different scales to obtain runway line recognition results;

[0102] The target detection branch is used to downsample the output of fused features of different scales and perform further fusion and recognition to obtain the runway number recognition results and runway entrance sign recognition results.

[0103] In this embodiment, in S4, the second feature extraction module includes a Focus module, a C2f module, a CBS module, and an SPPF module; and the second feature fusion module uses the SPPF module and the FPN network.

[0104] In this embodiment, in S4, the loss function of the runway feature detection module includes bounding box prediction loss, classification loss, confidence loss, pixel segmentation category loss, and pixel segmentation CloU loss;

[0105] Bounding box prediction loss L box for:

[0106]

[0107] Among them, N represents the number of detected targets in a frame image, CIoU represents the intersection-over-union ratio of the predicted target box and the real target box, and Pbox i Indicates the position of the i-th predicted target box, Tbox i Represents the true value of the position of the i-th predicted target box, L BCE represents the binary cross entropy loss;

[0108] Classification loss L cls for:

[0109]

[0110] Among them, Pcls i Indicates the category of the i-th predicted target box, Tcls i Represents the true value of the category of the i-th predicted target box;

[0111] Confidence loss L obj for:

[0112]

[0113] Among them, Pobj i Represents the confidence of the i-th predicted target box, Tobj i represents the true value of the confidence of the i-th predicted target box, balance represents the balance coefficient, which is used to balance the contribution of positive and negative samples, and balance t Indicates the balance coefficient when the training step is t, balance0 indicates the initial balance coefficient, N pos and N neg They represent the number of positive and negative samples when the training step is t, t represents the number of training steps, and T represents the total number of training steps;

[0114] Pixel segmentation category loss L seg for:

[0115]

[0116] Among them, L CE (Pseg, Tseg) represents the cross entropy loss, Pseg represents the semantic segmentation prediction result, Tseg represents the semantic segmentation annotation truth value, and η represents the balance factor between the cross entropy loss and the Dice loss;

[0117] Pixel segmentation CloU loss L iou_l for:

[0118]

[0119] Among them, CIoU seg Represents the intersection-over-union ratio of the semantic segmentation prediction result and the semantic segmentation annotation truth value, ρ 2 (S,S gt ) represents the shape difference metric between the semantic segmentation prediction area and the semantic segmentation annotation area, A represents the reference area, and ρ 2 (E,E gt ) represents the edge difference measure between the semantic segmentation prediction area and the semantic segmentation annotation area, P represents the reference perimeter, and γ represents the weight coefficient.

[0120] In the confidence loss, an adaptive balancing coefficient strategy is employed to account for the varying distribution and importance of positive and negative samples at different stages of the runway line detection process. Initially, the model's judgment of positive and negative samples is inaccurate, so the weight of positive samples is increased to prioritize learning their features. As training progresses and model performance improves, the weight of positive samples is appropriately reduced to balance training between positive and negative samples. This approach allows the model to dynamically adjust the balancing coefficient based on sample distribution at different stages of training, improving the accuracy of confidence predictions.

[0121] In the pixel segmentation category loss, the model pays attention to the segmentation of boundary pixels while paying attention to the accuracy of pixel classification, reducing the ambiguity and inaccuracy of the runway line segmentation results.

[0122] In the pixel segmentation CloU loss, the shape difference measure between the semantic segmentation prediction area and the semantic segmentation annotated area can be calculated by the contour matching algorithm. A represents the reference area, such as the average area of ​​the predicted area and the true area. The edge difference measure between the semantic segmentation prediction area and the semantic segmentation annotated area can be calculated by the edge detection algorithm. P represents the reference perimeter, such as the average perimeter of the predicted area and the true area. Such a loss function can more comprehensively evaluate the segmentation results and improve the accuracy of runway line semantic segmentation.

[0123] For example, the second feature extraction module in the runway feature detection module consists of a Focus module, a C2f module, a CBS module, and an SPPF module. The Focus module is typically combined with a Feature Pyramid Network (FPN) to achieve multi-scale feature fusion. For details on the C2f module, CBS module, SPPF module, and FPN network, refer to S2.

[0124] The second feature fusion module in the runway feature detection module adopts the combination of SPPF module and FPN network.

[0125] In the semantic segmentation branch of the runway feature detection module, since the three runway lines have a fixed positional relationship, all three lines are segmented using the same branch, rather than using separate branches for each line. This effectively reduces the possibility of cross-linking between the three runway line segmentation results. The output of the semantic segmentation branch is a matrix of size (640, 640, 4), with the same height and width as the original image. The four channels represent the probability of a pixel being the left runway line, the right runway line, the runway start line, and the background, respectively.

[0126] The object detection branch within the runway feature detection module is responsible for detecting runway numbers and runway entrance markings. The PAN architecture is first used to downsample the output of the second feature fusion module and further fuse it with deep features. This achieves a repeated fusion of deep semantic information and upper-layer texture information, improving the network's detection of objects of varying scales.

[0127] like Figure 3 As shown, Figure 3 This is a schematic diagram of the runway feature detection module. Figure 3 In the feature extraction (i.e., the second feature extraction module), feature fusion (i.e., the second feature extraction module), semantic segmentation branch and target detection branch, the parameters and connection relationships are as follows: Figure 3 As shown in .

[0128] In the pixel-wise segmentation CloU loss, Pseg represents the semantic segmentation prediction result. Its length and width match the original image size, and the value of each position ranges from 0 to 3, indicating whether the pixel is the left edge, right edge, starting edge, or background. Tseg represents the semantic segmentation annotation truth value, which has the same format as the prediction structure.

[0129] Optionally, the output of the semantic segmentation branch in the runway feature detection module can be used to extract runway line parameters. Figure 4 As shown, Figure 4The following is a flow chart for extracting runway parameters. First, the output of the runway feature detection module is channel-separated to obtain segmentation results for the left runway, right runway, and runway starting line. Because the runway segmentation results may contain mis-segments, these mis-segmented areas need to be removed. Dilation and erosion operations are used to fill holes in the segmentation results, and the maximum perimeter contour is extracted to remove mis-segmented areas of other small areas. Coarse positioning image restoration restores the segmentation results to the original image, ensuring that the runway parameters match the original image resolution. Finally, the least squares method is used to fit the segmentation results of each runway line to obtain the slope and intercept of the runway line.

[0130] Optionally, the output of the target detection branch in the runway feature detection module can be used for runway number recognition, identifying the runway number numeric information and thus obtaining the runway azimuth information. Number recognition can be performed using a simple, lightweight convolutional network consisting of two convolutional activation layers, two maximum pooling layers, and two fully connected layers. The convolutional network input is a resized grayscale image of the runway number region with a length and width of (28,28). The output vector is a vector of size (1,36), representing the probability of runway numbers 1 to 36. The index corresponding to the maximum output value is the recognized runway number.

[0131] Training a convolutional network requires a large number of images of different runway numbers captured during landing. Existing public datasets mostly consist of bird's-eye views of runway numbers, and the runway numbers do not fully cover all runway types from 1 to 36. To address this, we collected information on runway number dimensions from airport runway design standards and constructed images of numbers 0 to 9. These images were then permuted and combined to generate bird's-eye views of numbers 1 to 36. Finally, an affine transformation was used to generate runway number images from different angles, thus forming a runway number dataset.

[0132] Optionally, the output of the target detection branch in the runway feature detection module can be used to identify the runway entrance mark, identify the number of runway entrance segments, and thus obtain the prior information of the runway width. Figure 5 As shown, Figure 5This is a flowchart for runway threshold marking recognition. First, the runway threshold area image undergoes grayscale conversion and size normalization preprocessing to improve the applicability of the subsequent detection algorithm to different scenes and image sizes. Adaptive binarization uses a 20% grayscale value to remove the runway environment while retaining the white threshold line segments. After removing black noise through dilation and erosion, the runway edge is corrected to eliminate its influence on threshold line segment detection. Finally, the image is detected using line scanning to determine the number of white threshold line segments. The line scanning algorithm samples pixels at equal intervals along the y-axis of the image. For each pixel on the sampled line, the algorithm compares the pixel values ​​of the preceding and following pixels to determine whether it represents the left edge of a line segment transitioning from black to white. Finally, the number of left edges on the sampled line is counted to determine the number of threshold line segments in the image.

[0133] Optionally, the identified runway azimuth, runway width and runway geometric parameter information may be printed on the final output image.

[0134] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A method for detecting airport runway lines based on multi-task learning, characterized in that: The method comprises: S1. Acquire runway image data; The runway image data is continuous video frame images; S2. Inputting the runway image data into a YOLOv8 target detection module, performing runway line detection on the runway image data by the YOLOv8 target detection module, and extracting a to-be-detected area from the runway image data; The YOLOv8 target detection module includes a first feature extraction module, a first feature fusion module and a detection head; The first feature extraction module is used to extract features from the runway image data and fuse the extracted feature maps to obtain an initial fused feature map; The first feature fusion module is used to further fuse the fused feature map to obtain a fused feature map; The detection head is used to detect the fused feature map and output the area to be detected; S3, inputting the area to be detected into a coarse positioning area tracking module, determining a target to be tracked in the area to be detected in the initial frame by the coarse positioning area tracking module, performing runway target tracking on the area to be detected after the initial frame based on the target to be tracked, and outputting a target runway area in the area to be detected; The algorithm of the coarse positioning area tracking module is a kernel correlation filter target tracking algorithm; S4. Inputting the target runway area into a runway feature detection module, and outputting runway information corresponding to the runway image data through the runway feature detection module; The runway feature detection module includes a second feature extraction module, a second feature fusion module, a semantic segmentation branch and a target detection branch; The second feature extraction module is used to extract feature maps of different resolutions of the target runway area; The second feature fusion module is used to fuse feature maps of different resolutions to obtain fused features of different scales; The semantic segmentation branch is used to segment and identify fused features of different scales to obtain runway line recognition results; The target detection branch is used to output downsampling of fused features of different scales, and further fusion and recognition are performed to obtain runway number recognition results and runway entrance mark recognition results; The second feature extraction module includes a Focus module, a C2f module, a CBS module and an SPPF module; The second feature fusion module adopts the SPPF module and the FPN network; The loss function of the runway feature detection module includes bounding box prediction loss, classification loss, confidence loss, pixel segmentation category loss and pixel segmentation CloU loss; The bounding box prediction loss for: in, Indicates the number of detected targets in a frame of image, represents the intersection-over-union ratio of the predicted target box and the true target box, Indicates the The position of the predicted target box, Indicates the The true value of the position of the predicted target box, represents the binary cross entropy loss; The classification loss for: in, Indicates the The category of the predicted target box, Indicates the The true value of the category of the predicted target box; The confidence loss for: in, Indicates the The confidence of the predicted target box, Indicates the The true value of the confidence of the predicted target box, represents the balance coefficient, which is used to balance the contribution of positive and negative samples, and , Indicates the number of training steps is The balance coefficient when represents the initial equilibrium coefficient, and Respectively represent the number of training steps The number of positive and negative samples when represents the number of training steps, Indicates the total number of training steps; The pixel segmentation class loss for: in, represents the cross entropy loss, Represents the semantic segmentation prediction result, represents the semantic segmentation annotation truth value, Represents the balance factor between cross entropy loss and Dice loss; The pixel segmentation CloU loss for: in, It represents the intersection-over-union ratio of the semantic segmentation prediction result and the semantic segmentation annotation true value. Represents the shape difference metric between the semantic segmentation prediction area and the semantic segmentation annotation area, represents the reference area, Represents the edge difference metric between the semantic segmentation prediction area and the semantic segmentation annotation area, represents the reference circumference, Represents the weight coefficient.

2. The method according to claim 1, characterized in that In S2, the first feature extraction module includes a C2f module, a CBS module and an SPPF module; The first feature extraction module extracts features from the runway image data and fuses the extracted feature maps to obtain an initial fused feature map. The method is as follows: S21, extracting a bottom-level feature map and a high-level feature map of the runway image data through the C2f module, and fusing the bottom-level feature map and the high-level feature map to obtain a first feature map; S22. Convolve the first feature map using the CBS module to obtain a second feature map; S23. Superimpose the second feature map through the SPPF module to obtain the initial fusion feature map.

3. The method according to claim 1, characterized in that In S2, the structure of the first feature fusion module is a combination of FPN and PANI9; The detection head is a decoupling detection head.

4. The method according to claim 1, wherein In S2, the loss function of the YOLOv8 target detection module includes classification loss and bounding box prediction loss; The classification loss is a binary cross entropy loss; The bounding box prediction loss includes CIoU loss and Distribution Focal loss; Among them, CIoU loss for: in, Represents a constant, which is generally 1 according to the loss function definition. represents the intersection-over-union ratio of the bounding box and the ground-truth box, represents the center point of the bounding box, represents the center point of the ground-truth box, Represents the Euclidean distance between the center point of the bounding box and the center point of the ground truth box, Represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box. is the penalty factor for aspect ratio, represents the similarity of aspect ratios, , and Represent the width of the real box and the predicted box respectively, and Represent the height of the real box and the predicted box respectively, represents the penalty factor of the center point distance, Indicates the feature length of the predicted box and the true box.

Images