Airport runway line detection method based on multi-task learning

Abstract

The invention belongs to the technical field of computer vision, and particularly discloses an airport runway line detection method based on multi-task learning, which comprises the following steps: acquiring runway image data; inputting the runway image data into a YOLOv8 target detection module, performing runway line detection on the runway image data through the YOLOv8 target detection module, and extracting a to-be-detected area in the runway image data; inputting the to-be-detected area into a coarse positioning area tracking module, and outputting a target runway area in the to-be-detected area; and inputting the target runway area into a runway feature detection module, and outputting runway information corresponding to the runway image data through the runway feature detection module. According to the invention, the identification accuracy of the runway in a complex scene can be improved. The problem that in the prior art, robustness is not enough in a complex scene is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to an airport runway line detection method based on multi-task learning. Background Art

[0002] In airport landing scenarios, accurate and fast runway line detection technology is the basis for achieving safe landing of aircraft. With the development of computer vision technology and image processing technology, visual navigation technology has gradually been incorporated into the scope of navigation systems. Compared with traditional navigation systems, visual navigation has the advantages of strong autonomy, strong anti-interference, low cost, and increased accuracy as the distance decreases. The autonomous landing algorithm of UAVs based on visual information takes real-time runway environment images as input, and the onboard computer quickly identifies and detects runway information.

[0003] The current runway line detection algorithms include two categories: those based on traditional algorithms and those based on deep learning. Airport runway line detection based on traditional algorithms relies on manually designed features and the use of traditional machine learning algorithms for target detection. It mainly extracts local features from images to perform image feature matching or image classification. The process can be divided into three steps: feature extraction, feature description, and feature matching. Although traditional target detection algorithms perform well in some specific scenarios, they are usually not robust enough under challenges such as scale changes, perspective changes, occlusions, and complex backgrounds. With the rise of deep learning and the improvement of hardware parallel computing capabilities, target detection methods based on deep learning can better learn the abstract features of images and achieve better detection performance in practical applications. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides an airport runway line detection method based on multi-task learning, which solves the problem that the prior art is not robust enough in complex scenarios.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a method for detecting airport runway lines based on multi-task learning, comprising the following steps:

[0006] S1, obtaining runway image data;

[0007] The runway image data is continuous video frame images;

[0008] S2. Input the runway image data into the YOLOv8 target detection module, perform runway line detection on the runway image data through the YOLOv8 target detection module, and extract the area to be detected in the runway image data;

[0009] The YOLOv8 target detection module includes a first feature extraction module, a first feature fusion module and a detection head;

[0010] The first feature extraction module is used to extract features from the runway image data and fuse the extracted feature maps to obtain an initial fused feature map;

[0011] The first feature fusion module is used to further fuse the fused feature map to obtain a fused feature map;

[0012] The detection head is used to detect the fused feature map and output the area to be detected;

[0013] S3, inputting the area to be detected into a coarse positioning area tracking module, determining the target to be tracked in the area to be detected in the initial frame through the coarse positioning area tracking module, performing runway target tracking on the area to be detected after the initial frame based on the target to be tracked, and outputting the target runway area in the area to be detected;

[0014] The algorithm of the coarse positioning area tracking module is the kernel correlation filter target tracking algorithm;

[0015] S4, inputting the target runway area into a runway feature detection module, and outputting the runway information corresponding to the runway image data through the runway feature detection module;

[0016] The runway feature detection module includes a second feature extraction module, a second feature fusion module, a semantic segmentation branch and a target detection branch;

[0017] The second feature extraction module is used to extract feature maps of different resolutions of the target runway area;

[0018] The second feature fusion module is used to fuse feature maps of different resolutions to obtain fused features of different scales;

[0019] The semantic segmentation branch is used to segment and identify fused features of different scales to obtain runway line recognition results;

[0020] The object detection branch is used to downsample the output of fused features of different scales, and perform further fusion and recognition to obtain the runway number recognition results and runway entrance sign recognition results.

[0021] The beneficial effect of the above scheme is that the present invention focuses on the runway range through the YOLOv8 target detection module to achieve runway coarse positioning, thereby improving the efficiency and accuracy of subsequent runway line and runway mark detection. The coarse positioning area tracking module is based on the continuous frames of the video stream. On the basis of the target pixel position of the previous frame, the motion model is used to mine the pixel movement relationship between the front and back, thereby achieving target tracking, solving the problems of missed detection and false detection while ensuring globality, suppressing pixel position jumps, and combining with the YOLOv8 target detection module to achieve continuous, stable and fast runway coarse positioning effect. The runway feature detection module considers the learning process of multiple related tasks, utilizes the useful information between multiple related learning tasks and shares the features between related tasks, and improves the performance of each subtask.

[0022] Further, in S2, the first feature extraction module includes a C2f module, a CBS module and an SPPF module;

[0023] The first feature extraction module extracts features from the runway image data and fuses the extracted feature maps to obtain an initial fused feature map:

[0024] S21, extracting a bottom-level feature map and a high-level feature map of the runway image data through a C2f module, and fusing the bottom-level feature map with the high-level feature map to obtain a first feature map;

[0025] S22, convolving the first feature map through a CBS module to obtain a second feature map;

[0026] S23. Superimpose the second feature map through the SPPF module to obtain an initial fused feature map.

[0027] The beneficial effects of the above further scheme are: the C2f module combines feature maps of different levels through multi-scale feature fusion, thereby improving the model's detection ability for targets of different scales. The CBS module obtains richer gradient flow information while ensuring lightweight, and the function activation function of the CBS module is a Sigmoid weighted linear combination. The activation function is characterized by continuous differentiability, which can make the network nonlinear, thereby improving the network's expression ability. The SPPF module fuses the multi-scale features of the image by superimposing feature maps at different scales together, thereby increasing the network's detection accuracy.

[0028] Furthermore, in S2, the structure of the first feature fusion module is a combined structure of FPN and PANI9;

[0029] The detection head is a decoupled detection head.

[0030] The beneficial effect of the above further scheme is that the structure of the first feature fusion module upsamples the deep features and splices them with the upper features, and then downsamples them and further fuses them with the deep features, realizing the repeated fusion of deep semantic information and upper texture information, and improving the network's detection effect on objects of different scales. By using two branches to predict the bounding box position and target type respectively, the structure in which parameters are not shared between different detection heads can accelerate convergence and improve the prediction accuracy of the algorithm.

[0031] Furthermore, in S2, the loss function of the YOLOv8 object detection module includes classification loss and bounding box prediction loss;

[0032] The classification loss is binary cross entropy loss;

[0033] Bounding box prediction losses include CIoU loss and Distribution Focal loss;

[0034] Among them, CIoU loss L CIOU for:

[0035]

[0036] Among them, i represents a constant, which is generally 1 according to the definition of the loss function, IoU represents the intersection over union ratio of the bounding box and the ground-truth box, b represents the center point of the bounding box, and b gt represents the center point of the true box, ρ represents the Euclidean distance between the center point of the bounding box and the center point of the true box, c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box, α is the penalty factor for the aspect ratio, and v represents the similarity of the aspect ratio. w gt and w represent the width of the real box and the predicted box respectively, h gt and h represent the heights of the true box and the predicted box, respectively, β represents the penalty factor of the center point distance, and d represents the feature length of the predicted box and the true box.

[0037] The beneficial effect of the above further scheme is that the loss function of the YOLOv8 object detection module can make the network more flexible in predicting the position of the bounding box and improve the accuracy of the network's prediction of objects with unclear boundaries.

[0038] Furthermore, in S3, the algorithm used by the coarse positioning area tracking module is a kernel correlation filtering algorithm.

[0039] The beneficial effect of the above further scheme is that the combination of the kernel correlation filter algorithm and the YOLOv8 target detection module can achieve a continuous, stable and fast runway coarse positioning effect.

[0040] Further, in S4, the second feature extraction module includes a Focus module, a C2f module, a CBS module and a SPPF module;

[0041] The second feature fusion module uses the SPPF module and FPN network.

[0042] Furthermore, in S4, the loss function of the runway feature detection module includes bounding box prediction loss, classification loss, confidence loss, pixel segmentation category loss, and pixel segmentation CloU loss;

[0043] Bounding box prediction loss L box for:

[0044]

[0045] Among them, N represents the number of detected targets in a frame image, CIoU represents the intersection-over-union ratio between the predicted target box and the true target box, and Pbox i Indicates the position of the i-th predicted target box, Tbox i represents the true value of the position of the i-th predicted target box, L BCE represents the binary cross entropy loss;

[0046] Classification loss L cls for:

[0047]

[0048] Among them, Pcls i Indicates the category of the i-th predicted target box, Tcls i Represents the true value of the category of the i-th predicted target box;

[0049] Confidence loss L obj for:

[0050]

[0051] Among them, Pobj i Represents the confidence of the i-th predicted target box, Tobj i represents the true value of the confidence of the i-th predicted target box, balance represents the balance coefficient, which is used to balance the contribution of positive and negative samples, and balance t Indicates the balance coefficient when the training step is t, balance 0 represents the initial equilibrium coefficient, N pos and N neg They represent the number of positive and negative samples when the training step is t, t represents the training step, and T represents the total training step;

[0052] Pixel segmentation category loss Lseg for:

[0053]

[0054] Among them, L CE (Pseg, Tseg) represents the cross entropy loss, Pseg represents the semantic segmentation prediction result, Tseg represents the semantic segmentation annotation truth value, and η represents the balance factor between the cross entropy loss and the Dice loss;

[0055] Pixel segmentation CloU loss L iou_l for:

[0056]

[0057] Among them, CIoU seg represents the intersection-over-union ratio of the semantic segmentation prediction result and the semantic segmentation annotation truth value, ρ 2 (S,S gt ) represents the shape difference measure between the semantic segmentation prediction area and the semantic segmentation annotation area, A represents the reference area, ρ 2 (E,E gt ) represents the edge difference metric between the semantic segmentation prediction area and the semantic segmentation annotation area, P represents the reference perimeter, and γ represents the weight coefficient.

[0058] The beneficial effect of the above further scheme is that the Focus module uses slicing operation to split the high-resolution feature map into multiple low-resolution feature maps, and stacks the feature information on the channel, which can ensure that no data information is lost during downsampling. The SPPF module is used to generate and fuse features of different scales, and the FPN network fuses features at different semantic levels, so that the generated features contain multi-scale and multi-semantic level information. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 The figure is a flowchart of an airport runway line detection method based on multi-task learning.

[0060] Figure 2 Schematic diagram of the process of kernel correlation filtering algorithm.

[0061] Figure 3 It is a structural diagram of the runway feature detection module.

[0062] Figure 4 Schematic diagram of the process of extracting runway line parameters.

[0063] Figure 5 Schematic diagram of the process for runway threshold marking identification. DETAILED DESCRIPTION

[0064] Now, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the accompanying drawings are only exemplary and are intended to explain the principles and spirit of the present invention, rather than to limit the scope of the present invention.

[0065] like Figure 1 As shown, a method for detecting airport runway lines based on multi-task learning includes the following steps:

[0066] S1. Acquire runway image data; the runway image data is continuous video frame images.

[0067] Exemplarily, the runway image data may be image information obtained by a forward-looking monocular camera fixedly installed in front of the aircraft's flight direction, the image acquisition frame rate may be 30 frames per second, the image type may be an RGB color image, and the image resolution may be 640*640 pixels.

[0068] S2. Input the runway image data into the YOLOv8 target detection module, perform runway line detection on the runway image data through the YOLOv8 target detection module, and extract the area to be detected in the runway image data;

[0069] The YOLOv8 target detection module includes a first feature extraction module, a first feature fusion module and a detection head;

[0070] The first feature extraction module is used to extract features from the runway image data and fuse the extracted feature maps to obtain an initial fused feature map;

[0071] The first feature fusion module is used to further fuse the fused feature map to obtain a fused feature map;

[0072] The detection head is used to detect the fused feature map and output the area to be detected.

[0073] In this embodiment, in S2, the first feature extraction module includes a C2f module, a CBS module and an SPPF module;

[0074] The first feature extraction module extracts features from the runway image data and fuses the extracted feature maps to obtain an initial fused feature map:

[0075] S21, extracting a bottom-level feature map and a high-level feature map of the runway image data through a C2f module, and fusing the bottom-level feature map with the high-level feature map to obtain a first feature map;

[0076] S22, convolving the first feature map through a CBS module to obtain a second feature map;

[0077] S23. Superimpose the second feature map through the SPPF module to obtain an initial fused feature map.

[0078] In this embodiment, in S2, the structure of the first feature fusion module is a structure combining FPN and PANI9; and the detection head is a decoupled detection head.

[0079] In this embodiment, in S2, the loss function of the YOLOv8 target detection module includes classification loss and bounding box prediction loss;

[0080] The classification loss is binary cross entropy loss;

[0081] Bounding box prediction losses include CIoU loss and Distribution Focal loss;

[0082] Among them, CIoU loss L CIOU for:

[0083]

[0084] Among them, i represents a constant, which is generally 1 according to the definition of the loss function, IoU represents the intersection over union ratio of the bounding box and the ground-truth box, b represents the center point of the bounding box, and b gt represents the center point of the true box, ρ represents the Euclidean distance between the center point of the bounding box and the center point of the true box, c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box, α is the penalty factor for the aspect ratio, and v represents the similarity of the aspect ratio. w gt and w represent the width of the real box and the predicted box respectively, h gt and h represent the heights of the true box and the predicted box, respectively, β represents the penalty factor of the center point distance, and d represents the feature length of the predicted box and the true box.

[0085] For example, when collecting runway image data, the scale and viewing angle of the runway image taken by the aircraft's camera will change significantly, and various interferences may occur in the environment around the runway. Therefore, directly detecting the runway line and runway markings in the original image will result in more false detections and low detection accuracy. Using runway line coarse positioning has the advantages of stronger scale adaptability, improved contradiction between real-time and accuracy, and enhanced scene adaptability.

[0086] Among them, YOLOv8 is an advanced target detection algorithm and the latest version of the YOLO (You Only Look Once) series. Based on the YOLOv8 target detection algorithm, the YOLOv8 target detection module selects the pixel position of the runway from the original image taken by the camera, and extracts the runway image from the original image through cropping and scaling.

[0087] The YOLOv8 algorithm supports a variety of visual tasks, such as object detection, segmentation, and pose estimation. The YOLOv8 object detection module includes the first feature extraction module (Backbone), the first feature fusion module (Neck), and the detection head (Head). The first feature extraction module consists of the C2f module, the CBS module, and the SPPF module. The C2f module divides the input features into two parts according to the number of channels through the Split operation. One part proposes high-level features through n layers of Bottleneck, and the other part is directly superimposed with the last high-level features to fuse the underlying feature map with the high-level feature map. In the YOLOv8 network, the C2f module (Convolutional to Feature module) is an important component, which plays a key role in improving the performance and efficiency of the model. The CBS module (Convolution BatchNorm SiLU) is a commonly used convolution block, which consists of a convolution layer (Convolution), a batch normalization layer (Batch Normalization), and a SiLU activation function (Sigmoid LinearUnit). Compared with the ReLU function, the SiLU function has a smoother curve when it is close to zero. The SPPF module (Spatial Pyramid Pooling-Fast) is improved on the basis of spatial pyramid pooling (SPP). It fuses the multi-scale features of the image by superimposing feature maps at different scales, reducing the amount of calculation and improving the calculation efficiency.

[0088] FPN (Feature Pyramid Network) is a multi-scale feature fusion method, mainly used for target detection tasks. It builds a feature pyramid to fuse feature maps of different scales, thereby improving the model's ability to detect targets of different sizes. PANI9 (Path Aggregation Network with Improved 9-layer) is an improved version of the feature fusion network, which further enhances the effect of feature fusion by adding more paths and layers.

[0089] The loss function of the YOLOv8 target detection module consists of two parts: classification loss and bounding box prediction loss. cls Binary cross entropy loss (BCE) is used, and the bounding box prediction loss is composed of CIoU loss L CIOU (CloULoss) and Distribution Focal Loss L DFL(Distribution Focal Loss). Distribution Focal loss regresses the bounding box position in a probabilistic way. By setting the size of the hyperparameter reg_max, the position of the bounding box is mapped to a fixed value of 0 to reg, and then discretized by the softmax function to obtain the probability relative to each fixed value. Finally, L is calculated by cross loss. DFL .L CIOU The aspect ratio of the bounding box is taken into account in the calculation based on the intersection and union ratio.

[0090] The YOLOv8 target detection module introduces the Task-Aligned Assigner positive and negative sample assignment strategy during training. This strategy weights the category scores s and CIoU scores u predicted by all pixels to obtain weighted scores, and regards the first k samples after the scores are sorted as positive samples. CIOU The higher the value, the closer the value of t is to 1. At this time, the predicted bounding box matches the labeled box more closely, that is, it meets the standard of positive samples more closely. Through training, the network can be guided to dynamically focus on high-quality positive samples. The weighted score calculation formula is as follows:

[0091] t=s α ×u β

[0092] Among them, t represents the weighted score, s represents the category score, u represents the CIoU score, and α and β are hyperparameters used to balance the importance of classification and localization, respectively.

[0093] S3. Input the area to be detected into the coarse positioning area tracking module, determine the target to be tracked in the area to be detected in the initial frame through the coarse positioning area tracking module, perform runway target tracking on the area to be detected after the initial frame based on the target to be tracked, and output the target runway area in the area to be detected; the algorithm of the coarse positioning area tracking module is the kernel correlation filter target tracking algorithm.

[0094] In this embodiment, in S3, the algorithm used by the coarse positioning area tracking module is a kernel correlation filtering algorithm.

[0095] Exemplarily, the kernelized correlation filter algorithm (KCF) first generates training samples using a circulant matrix in the initial frame, where the original base sample is regarded as a positive sample and the other samples generated by the circulant matrix are negative samples. These samples can be easily calculated using the properties of fast Fourier transform and Fourier diagonalization, and the trained tracker can generate the desired distribution form when acting on these samples. During the tracking process, the kernelized correlation filter algorithm uses the trained tracker to filter the window area to obtain the output distribution map, and the maximum response position is used as the center position of the predicted target. For the tracker, the new tracking result is used as the base sample to generate a circulant matrix to obtain a new sample set, and the tracker is updated through online training.

[0096] like Figure 2 As shown, Figure 2 The figure is a flowchart of the kernel correlation filter algorithm. The kernel correlation filter algorithm first starts the target detection thread, determines the target to be tracked in the initial frame based on the detected runway, and the target tracking thread completes initialization using the target to be tracked. After the tracking thread is initialized, the tracking thread continuously tracks the runway target in the image. In order to suppress the tracking drift phenomenon, the target to be tracked in the tracking algorithm needs to be updated. After the target to be tracked is selected in the initial frame, the updated target is regarded as a template image and its feature value (HOG feature) is calculated. After the tracking thread completes tracking of each frame of the image, the feature value of the tracking result is calculated and compared with the template feature value. If the similarity is greater than the update threshold, the target tracking thread continues to run, otherwise the target detection thread runs to update the tracked target.

[0097] S4, inputting the target runway area into a runway feature detection module, and outputting the runway information corresponding to the runway image data through the runway feature detection module;

[0098] The runway feature detection module includes a second feature extraction module, a second feature fusion module, a semantic segmentation branch and a target detection branch;

[0099] The second feature extraction module is used to extract feature maps of different resolutions of the target runway area;

[0100] The second feature fusion module is used to fuse feature maps of different resolutions to obtain fused features of different scales;

[0101] The semantic segmentation branch is used to segment and identify fused features of different scales to obtain runway line recognition results;

[0102] The object detection branch is used to downsample the output of fused features of different scales, and perform further fusion and recognition to obtain the runway number recognition results and runway entrance sign recognition results.

[0103] In this embodiment, in S4, the second feature extraction module includes a Focus module, a C2f module, a CBS module and an SPPF module; the second feature fusion module adopts an SPPF module and an FPN network.

[0104] In this embodiment, in S4, the loss function of the runway feature detection module includes bounding box prediction loss, classification loss, confidence loss, pixel segmentation category loss and pixel segmentation CloU loss;

[0105] Bounding box prediction loss L box for:

[0106]

[0107] Among them, N represents the number of detected targets in a frame image, CIoU represents the intersection-over-union ratio between the predicted target box and the true target box, and Pbox i Indicates the position of the i-th predicted target box, Tbox i represents the true value of the position of the i-th predicted target box, L BCE represents the binary cross entropy loss;

[0108] Classification loss L cls for:

[0109]

[0110] Among them, Pcls i Indicates the category of the i-th predicted target box, Tcls i Represents the true value of the category of the i-th predicted target box;

[0111] Confidence loss L obj for:

[0112]

[0113] Among them, Pobj i Represents the confidence of the i-th predicted target box, Tobj i represents the true value of the confidence of the i-th predicted target box, balance represents the balance coefficient, which is used to balance the contribution of positive and negative samples, and balance t Indicates the balance coefficient when the training step is t, balance 0 represents the initial equilibrium coefficient, N pos and N neg They represent the number of positive and negative samples when the training step is t, t represents the training step, and T represents the total training step;

[0114] Pixel segmentation category loss Lseg for:

[0115]

[0116] Among them, L CE (Pseg, Tseg) represents the cross entropy loss, Pseg represents the semantic segmentation prediction result, Tseg represents the semantic segmentation annotation truth value, and η represents the balance factor between the cross entropy loss and the Dice loss;

[0117] Pixel segmentation CloU loss L iou_l for:

[0118]

[0119] Among them, CIoU seg represents the intersection-over-union ratio of the semantic segmentation prediction result and the semantic segmentation annotation truth value, ρ 2 (S,S gt ) represents the shape difference measure between the semantic segmentation prediction area and the semantic segmentation annotation area, A represents the reference area, ρ 2 (E,E gt ) represents the edge difference metric between the semantic segmentation prediction area and the semantic segmentation annotation area, P represents the reference perimeter, and γ represents the weight coefficient.

[0120] In the confidence loss, considering that the distribution and importance of positive and negative samples are different at different stages in the runway line detection process, an adaptive balance coefficient strategy is adopted. In the early stage of training, the model makes inaccurate judgments on positive and negative samples, and the weight of positive samples is increased to make the model pay more attention to the learning of positive sample features; as the training progresses, the model performance improves, and the weight of positive samples is appropriately reduced to balance the training of positive and negative samples. This method allows the model to dynamically adjust the balance coefficient according to the sample distribution at different stages of training, thereby improving the accuracy of confidence prediction.

[0121] In the pixel segmentation category loss, the model pays more attention to the segmentation of boundary pixels while paying attention to the accuracy of pixel classification, reducing the ambiguity and inaccuracy of the runway line segmentation results.

[0122] In the pixel segmentation CloU loss, the shape difference measure between the semantic segmentation prediction area and the semantic segmentation annotation area can be calculated by the contour matching algorithm. A represents the reference area, such as the average area of ​​the predicted area and the real area. The edge difference measure between the semantic segmentation prediction area and the semantic segmentation annotation area can be calculated by the edge detection algorithm. P represents the reference circumference, such as the average circumference of the predicted area and the real area. Such a loss function can more comprehensively evaluate the segmentation results and improve the accuracy of runway line semantic segmentation.

[0123] Exemplarily, the second feature extraction module in the runway feature detection module is composed of a Focus module, a C2f module, a CBS module, and an SPPF module. The Focus module is usually combined with a feature pyramid network (FPN) to achieve multi-scale feature fusion. The C2f module, the CBS module, the SPPF module, and the FPN network can refer to the explanation in S2.

[0124] The second feature fusion module in the runway feature detection module adopts the combination of SPPF module and FPN network.

[0125] In the semantic segmentation branch of the runway feature detection module, since the three runway lines have a fixed positional relationship, the same branch is used to segment the three lines instead of using a separate branch for each line, which can effectively reduce the cross-linking of the segmentation results of the three runway lines. The output of the semantic segmentation branch is a matrix of size (640,640,4), whose length and width are consistent with the original image, and the four channels represent the probability that the pixel is the left runway line, the right runway line, the runway start line, and the background.

[0126] The target detection branch in the runway feature detection module is responsible for detecting the runway number and runway entrance sign on the runway. First, the PAN structure is used to downsample the output of the second feature fusion module, and then further fused with the deep features to achieve repeated fusion of deep semantic information and upper texture information, thereby improving the network's detection effect on targets of different scales.

[0127] like Figure 3 As shown, Figure 3 Figure 1 is a schematic diagram of the runway feature detection module. Figure 3 In the example, the parameters and connection relationships in feature extraction (i.e., the second feature extraction module), feature fusion (i.e., the second feature extraction module), semantic segmentation branch, and target detection branch are as follows: Figure 3 as shown in .

[0128] In the pixel segmentation CloU loss, Pseg is the semantic segmentation prediction result, whose length and width are consistent with the original image size, and the value of each position is composed of 0 to 3, indicating that the pixel is the left line, right line, starting line or background. Tseg represents the semantic segmentation annotation true value, and its form is consistent with the prediction structure.

[0129] Optionally, the output result of the semantic segmentation branch in the runway feature detection module can be used to extract runway line parameters. Figure 4 As shown, Figure 4The figure is a flow chart of the runway line parameter extraction. First, the output results of the runway feature detection module are channel-separated to obtain the segmentation results of the left runway, the right runway and the runway start line. Since the runway line segmentation results may have mis-segmentation, the mis-segmented areas need to be removed. The expansion and corrosion operation is used to fill the holes in the segmentation results, and the maximum perimeter contour is extracted to remove the mis-segmentation of other small areas. Coarse positioning image restoration is to restore the segmentation results to the original image to ensure that the runway line parameters are consistent with the resolution of the original image. Finally, the least squares method is used to fit the segmentation results of each runway line to obtain the slope and intercept of the runway line.

[0130] Optionally, the output result of the target detection branch in the runway feature detection module can be used for runway number recognition, and the digital information of the runway number can be identified to obtain the azimuth information of the runway. Number recognition can be performed through a simple and lightweight convolutional network, which consists of two convolutional activation layers, two maximum pooling layers, and two fully connected layers. The convolutional network inputs a grayscale image of the runway number area with a resized length and width of (28,28), and outputs a vector of size (1,36), which represents the probability of runway numbers 1 to 36. The index corresponding to the maximum output value is the recognized runway number.

[0131] The training of the convolutional network requires a large number of different runway number images taken during the landing process. Most of the existing public data sets are runway bird's-eye views, and the runway numbers cannot completely cover all runway types from 1 to 36. To this end, we can collect information about the runway number size in the airport runway design standards, construct number images from 0 to 9, and then arrange and combine them to generate a bird's-eye view of numbers from 1 to 36. Finally, we use affine transformation to generate runway number images at different angles to form a runway number data set.

[0132] Optionally, the output result of the target detection branch in the runway feature detection module can be used to perform runway entrance mark recognition to identify the number of runway entrance line segments, thereby obtaining prior information on the runway width. Figure 5 As shown, Figure 5The figure is a flowchart of runway entrance mark recognition. First, the runway entrance area image is preprocessed by grayscale and size normalization to improve the applicability of the subsequent detection algorithm to different scenes and images of different sizes. Adaptive value binarization uses a grayscale value of 20% to remove the runway environment while retaining the white entrance line segment. After removing black noise through expansion corrosion, the runway edge is corrected to remove the impact of the runway edge on the entrance line segment detection. Finally, the image is detected by line scanning to obtain the number of white entrance line segments. The line scanning algorithm samples the pixels on the y-axis of the image at equal intervals, traverses each pixel on the sampling line, and judges whether it is the left edge of the line segment that turns from black to white by comparing the pixel values ​​of the two pixels before and after. Finally, the number of left edges on the sampling line is counted to determine the number of entrance line segments of the image.

[0133] Optionally, the identified runway azimuth, runway width and runway geometric parameter information may be printed on the final output image.

[0134] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for detecting airport runway lines based on multi-task learning, characterized in that: The method comprises: S1, obtaining runway image data; The runway image data is continuous video frame images; S2, inputting the runway image data into a YOLOv8 target detection module, performing runway line detection on the runway image data through the YOLOv8 target detection module, and extracting the area to be detected in the runway image data; The YOLOv8 target detection module includes a first feature extraction module, a first feature fusion module and a detection head; The first feature extraction module is used to extract features from the runway image data and fuse the extracted feature maps to obtain an initial fused feature map; The first feature fusion module is used to further fuse the fused feature map to obtain a fused feature map; The detection head is used to detect the fused feature map and output the area to be detected; S3, inputting the area to be detected into a coarse positioning area tracking module, determining the target to be tracked in the area to be detected in the initial frame through the coarse positioning area tracking module, performing runway target tracking on the area to be detected after the initial frame based on the target to be tracked, and outputting the target runway area in the area to be detected; The algorithm of the coarse positioning area tracking module is a kernel correlation filter target tracking algorithm; S4, inputting the target runway area into a runway feature detection module, and outputting the runway information corresponding to the runway image data through the runway feature detection module; The runway feature detection module includes a second feature extraction module, a second feature fusion module, a semantic segmentation branch and a target detection branch; The second feature extraction module is used to extract feature maps of different resolutions of the target runway area; The second feature fusion module is used to fuse feature maps of different resolutions to obtain fused features of different scales; The semantic segmentation branch is used to segment and identify fused features of different scales to obtain runway line recognition results; The target detection branch is used to output down-sampling of fused features of different scales, and to perform further fusion and recognition to obtain runway number recognition results and runway entrance sign recognition results.

2. The method according to claim 1, characterized in that In S2, the first feature extraction module includes a C2f module, a CBS module and an SPPF module; The first feature extraction module extracts features from the runway image data and fuses the extracted feature maps to obtain an initial fused feature map: S21, extracting a bottom-level feature map and a high-level feature map of the runway image data through the C2f module, and fusing the bottom-level feature map and the high-level feature map to obtain a first feature map; S22, convolving the first feature map through the CBS module to obtain a second feature map; S23. Superimposing the second feature map through the SPPF module to obtain the initial fused feature map.

3. The method according to claim 1, characterized in that In S2, the structure of the first feature fusion module is a combined structure of FPN and PANI9; The detection head is a decoupling detection head.

4. The method according to claim 1, characterized in that In S2, the loss function of the YOLOv8 target detection module includes classification loss and bounding box prediction loss; The classification loss is a binary cross entropy loss; The bounding box prediction loss includes CIoU loss and Distribution Focal loss; Among them, CIoU loss L CIOU for: Among them, i represents a constant, which is generally 1 according to the definition of the loss function, IoU represents the intersection over union ratio of the bounding box and the ground-truth box, b represents the center point of the bounding box, and b gt represents the center point of the true box, ρ represents the Euclidean distance between the center point of the bounding box and the center point of the true box, c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box, α is the penalty factor for the aspect ratio, and v represents the similarity of the aspect ratio. w gt and w represent the width of the real box and the predicted box respectively, h gt and h represent the heights of the true box and the predicted box, respectively, β represents the penalty factor of the center point distance, and d represents the feature length of the predicted box and the true box.

5. The method according to claim 1, characterized in that: In S3, the algorithm used by the coarse positioning area tracking module is a kernel correlation filtering algorithm.

6. The method according to claim 1, characterized in that In S4, the second feature extraction module includes a Focus module, a C2f module, a CBS module and an SPPF module; The second feature fusion module adopts the SPPF module and the FPN network.

7. The method according to claim 1, characterized in that In S4, the loss function of the runway feature detection module includes bounding box prediction loss, classification loss, confidence loss, pixel segmentation category loss and pixel segmentation CloU loss; The bounding box prediction loss L box for: Among them, N represents the number of detected targets in a frame image, CIoU represents the intersection-over-union ratio between the predicted target box and the true target box, and Pbox i Indicates the position of the i-th predicted target box, Tbox i represents the true value of the position of the i-th predicted target box, L BCE represents the binary cross entropy loss; The classification loss L cls for: Among them, Pcls i Indicates the category of the i-th predicted target box, Tcls i Represents the true value of the category of the i-th predicted target box; The confidence loss L obj for: Among them, Pobj i Represents the confidence of the i-th predicted target box, Tobj i represents the true value of the confidence of the i-th predicted target box, balance represents the balance coefficient, which is used to balance the contribution of positive and negative samples, and balance t represents the balance coefficient when the training step is t, balance0 represents the initial balance coefficient, N pos and N neg They represent the number of positive and negative samples when the training step is t, t represents the training step, and T represents the total training step; The pixel segmentation class loss L seg for: Among them, L CE (Pseg, Tseg) represents the cross entropy loss, Pseg represents the semantic segmentation prediction result, Tseg represents the semantic segmentation annotation truth value, and η represents the balance factor between the cross entropy loss and the Dice loss; The pixel segmentation CloU loss L iou_l for: Among them, CIoU seg represents the intersection-over-union ratio of the semantic segmentation prediction result and the semantic segmentation annotation truth value, ρ 2 (S,S gt ) represents the shape difference measure between the semantic segmentation prediction area and the semantic segmentation annotation area, A represents the reference area, ρ 2 (E,E gt ) represents the edge difference metric between the semantic segmentation prediction area and the semantic segmentation annotation area, P represents the reference perimeter, and γ represents the weight coefficient.

Images