Semi-supervised learning-based airport runway foreign object detection method, device, equipment and medium

Abstract

This application provides a method, apparatus, device, and medium for foreign object detection on airport runways based on semi-supervised learning, relating to the field of foreign object detection on airport runways. The method includes: collecting multiple images of the airport runway; manually annotating some images from the multiple images to obtain multiple partially labeled images with real labels; randomly selecting multiple bounding box images of foreign objects from a database; pasting the bounding box images of multiple foreign objects onto the runway region of each partially labeled image to obtain multiple labeled images; pre-training based on unlabeled images from the multiple images; transferring the weights obtained from the pre-training to a foreign object detection model; fine-tuning the foreign object detection model based on the multiple labeled images to obtain a final trained foreign object detection model; inputting a real-time target runway image into the final trained foreign object detection model; and outputting the detection result of whether the runway contains foreign objects, thereby improving the detection accuracy of small foreign objects on airport runways.

Description

Technical Field

[0001] This application relates to the field of foreign object detection on airport runways, and provides a method, apparatus, equipment and medium for foreign object detection on airport runways based on semi-supervised learning. Background Technology

[0002] Foreign Object Debris (FOD) detection on airport runways is a critical technology in the field of aviation safety. Foreign objects on the runway (such as stones, tools, and metal fragments) can pose a serious threat to aircraft takeoff and landing, and may even cause major accidents. Therefore, accurately detecting and promptly removing FOD from the runway is essential for ensuring aviation safety.

[0003] In recent years, with the development of deep learning technology, deep learning-based FOD detection methods have gradually become a research hotspot. These methods utilize deep learning models such as convolutional neural networks (CNNs) to automatically extract features from runway images and perform foreign object detection.

[0004] Existing deep learning methods typically rely on traditional supervised learning, requiring a large amount of precisely labeled data for model training. However, some foreign objects on runways are small in size and have low contrast with the background. When manually labeling small foreign objects, it is easy to miss them or label them inaccurately, resulting in low detection accuracy of small foreign objects on airport runways. Summary of the Invention

[0005] This application provides a method, apparatus, equipment, and medium for detecting foreign objects on airport runways based on semi-supervised learning, which addresses the problem of low detection accuracy for small foreign objects on airport runways.

[0006] Firstly, a semi-supervised learning-based method for detecting foreign objects on airport runways is provided, including:

[0007] Multiple images of an airport runway are collected, and some of these images are manually labeled to obtain multiple partially labeled images with real labels; the real labels are the bounding boxes of foreign objects.

[0008] Multiple bounding box images of foreign objects are randomly selected from the database, and these images are pasted onto the runway area of ​​each partially marked image to obtain multiple marked images; the database contains bounding box images of foreign objects smaller than a preset size.

[0009] Pre-training is performed based on the unlabeled images in the multiple images and the MoCo v2 framework, and the weights obtained from the pre-training are transferred to the foreign object detection model.

[0010] The foreign object detection model is fine-tuned and trained based on the multiple labeled images to obtain the final trained foreign object detection model.

[0011] The real-time target runway image is input into the finally trained foreign object detection model, which outputs the detection result of whether the runway contains foreign objects.

[0012] Optionally, before randomly selecting multiple bounding box images of foreign objects from the database, pasting the multiple bounding box images of foreign objects onto the runway region of each partially labeled image to obtain multiple labeled images, the method further includes:

[0013] Generate multiple cue points based on each partially labeled image;

[0014] Each partially labeled image and the multiple cue points are input into SAM v2, and multiple segmentation masks are output.

[0015] The multiple segmentation masks are sorted in descending order of area, and the top K segmentation masks are selected; K is a positive integer greater than 0.

[0016] The region corresponding to the segmentation mask with the highest confidence among the K segmentation masks is determined as the runway region of each partially labeled image.

[0017] Optionally, the step of generating multiple cue points based on each partially labeled image includes:

[0018] Based on the bounding box of each partially labeled image, multiple prompt points are randomly generated that meet preset conditions; the preset conditions include that the prompt point is not inside the bounding box and the distance between the prompt point and the bounding box is greater than a preset distance.

[0019] Optionally, the step of randomly selecting multiple bounding box images of foreign objects from the database and pasting these bounding box images onto the runway region of each partially labeled image to obtain multiple labeled images includes:

[0020] Randomly select multiple bounding box images of foreign objects from the database, and perform geometric transformations on the bounding box images of the multiple foreign objects respectively to obtain multiple transformed bounding box images;

[0021] The transformed bounding box images are pasted onto the runway region of each partially labeled image to obtain multiple labeled images.

[0022] Optionally, the step of fine-tuning the foreign object detection model based on the multiple labeled images to obtain the final trained foreign object detection model includes:

[0023] In the first stage, the foreign object detection model is trained based on the multiple labeled images to obtain the initially trained foreign object detection model;

[0024] In the second stage, each labeled image is input into the initially trained foreign object detection model, and pseudo-labels for each labeled image are output.

[0025] The real and pseudo labels of each labeled image are fused to obtain a labeled image with fused labels;

[0026] The initially trained foreign object detection model is trained based on the labeled images fused with the labels to obtain the final trained foreign object detection model.

[0027] Optionally, each labeled image may have multiple pseudo-labels; the process of fusing the real and pseudo-labels of each labeled image to obtain a labeled image with fused labels includes:

[0028] Delete pseudo-labels that are lower than the target confidence threshold from multiple pseudo-labels to obtain the remaining pseudo-labels for each labeled image;

[0029] The real labels and remaining pseudo labels of each labeled image are fused to obtain a labeled image with fused labels.

[0030] Optionally, fusing the real and pseudo labels of each labeled image to obtain a labeled image with fused labels includes:

[0031] Calculate the intersection-union ratio (IoU) of the bounding boxes of the ground truth labels and the remaining pseudo labels for each labeled image;

[0032] If the cross-union ratio of the bounding boxes is less than the cross-union ratio threshold, then the real labels and the remaining pseudo labels of each labeled image are fused to obtain a labeled image with fused labels.

[0033] Optionally, before deleting pseudo-labels below the target confidence threshold from multiple pseudo-labels to obtain the remaining pseudo-labels for each labeled image, the method further includes:

[0034] Based on the current training round and the total number of training rounds, the initial confidence threshold is adjusted to obtain the target confidence threshold; the target confidence threshold is negatively correlated with the current training round.

[0035] Optionally, the formula for calculating the target confidence threshold is as follows:

[0036]

[0037] Where, θ new θ is the target confidence threshold. initial N is the initial confidence threshold. epochs The total number of training rounds, This refers to the current training round.

[0038] Secondly, a semi-supervised learning-based foreign object detection device for airport runways is provided, comprising:

[0039] The annotation module is used to collect multiple images of the airport runway, manually annotate some of the images, and obtain multiple partially labeled images with real labels; the real labels are the bounding boxes of foreign objects.

[0040] The pasting module is used to randomly select multiple bounding box images of foreign objects from the database, and paste the bounding box images of the multiple foreign objects onto the runway area of ​​each partially marked image to obtain multiple marked images; the database contains bounding box images of foreign objects smaller than a preset size.

[0041] The pre-training module is used to pre-train based on the unlabeled images in the multiple images and the MoCo v2 framework, and transfer the weights obtained from the pre-training to the foreign object detection model;

[0042] The fine-tuning training module is used to fine-tune the foreign object detection model based on the multiple labeled images to obtain the final trained foreign object detection model.

[0043] The detection module is used to input the real-time target runway image into the finally trained foreign object detection model and output the detection result of whether the runway contains foreign objects.

[0044] Thirdly, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the airport runway foreign object detection method based on semi-supervised learning described in the first aspect.

[0045] Fourthly, this application provides a computer-readable storage medium storing a computer program, on which a processor executes the computer program to implement the airport runway foreign object detection method based on semi-supervised learning described in the first aspect.

[0046] Compared with the prior art, the beneficial effects of the embodiments of this application are as follows:

[0047] This application provides a semi-supervised learning-based method for foreign object detection on airport runways. The method includes: collecting multiple images of the airport runway; manually annotating some images to obtain multiple partially labeled images with real-world labels; bounding boxes labeled as foreign objects; randomly selecting multiple bounding box images of foreign objects from a database and pasting these bounding box images onto the runway region of each partially labeled image to obtain multiple labeled images; pre-storing bounding box images of foreign objects smaller than a preset size in the database; pre-training based on unlabeled images from the multiple images and the MoCo v2 framework, and transferring the pre-trained weights to the foreign object detection model; fine-tuning the foreign object detection model based on the multiple labeled images to obtain the final trained foreign object detection model; inputting a real-time target runway image into the final trained foreign object detection model and outputting the detection result of whether the runway contains foreign objects.

[0048] As can be seen, this application uses the MoCo v2 framework for self-supervised pre-training, which helps the model learn useful features from unlabeled images, reducing reliance on manually annotated data. MoCo v2 pre-training provides the model with a more general feature extraction capability. When transferred to foreign object detection models, it avoids training from scratch, thereby reducing model training time and improving convergence speed. Through weight transfer, it helps the model achieve good training results even with limited labeled data. Furthermore, pasting the bounding box images of small foreign objects into the runway area increases data diversity, allowing the model to learn more features about small foreign objects, enhancing its ability to detect small objects in complex environments, and thus improving the detection accuracy of small foreign objects on airport runways. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0051] Figure 2 A flowchart illustrating the semi-supervised learning-based foreign object detection method for airport runways provided in this application embodiment;

[0052] Figure 3 This is a schematic diagram of the structure of the airport runway foreign object detection device based on semi-supervised learning provided in the embodiments of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0054] Traditional FOD detection methods primarily rely on sensor systems such as millimeter-wave radar and lidar. These systems detect foreign objects on the runway by emitting electromagnetic waves or laser beams and identify them based on reflected signals. However, these methods are costly and complex to maintain. Traditional image processing techniques preprocess runway images (e.g., edge detection, filtering) and combine them with traditional machine vision and machine learning algorithms for FOD identification. While these methods can detect FODs to some extent, their performance is limited by the inherent limitations of image processing techniques. Deep learning-based FOD detection methods utilize deep learning models such as convolutional neural networks (CNNs) to automatically extract features from runway images and perform FOD detection. Compared to traditional image processing methods, deep learning methods have significant advantages in feature extraction and classification capabilities, and can better handle complex runway environments.

[0055] Traditional image processing methods and existing deep learning models are difficult to effectively identify small foreign objects in airport runways. Therefore, this application provides a semi-supervised learning-based method for detecting foreign objects on airport runways, which can be executed by a central server.

[0056] Before introducing the specific technical solutions of this application, let's first describe the application scenarios involved in the embodiments of this application. Please refer to... Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. The application scenario includes multiple cameras, edge computing nodes, and a central server.

[0057] Multiple cameras, all high-resolution optical cameras (supporting visible light and infrared imaging), capture images of the airport runway and send them to edge computing nodes deployed around the runway. These edge computing nodes preprocess the received images, performing tasks such as illumination correction, deblurring, and applying basic enhancement operations like random cropping and rotation, before transmitting them to the central server. The central server can then perform foreign object detection based on the received images; the specific detection methods will be described in detail below.

[0058] It should be noted that, Figure 1 This example uses two cameras, but there is actually no limit to the number of cameras. Multiple cameras are deployed on both sides of the airport runway to ensure that multiple cameras can capture the entire airport runway.

[0059] Based on the application scenarios of the foregoing embodiments, the following will combine... Figure 2 This application introduces a semi-supervised learning-based method for detecting foreign objects on airport runways, with the following steps:

[0060] S201. Collect multiple images of the airport runway, manually annotate some of the images, and obtain multiple partially labeled images with real labels.

[0061] In practice, the central server can acquire multiple images of the airport runway via cameras or download existing airport runway image datasets from the internet, thus obtaining numerous images of the runway covering different scenarios (such as different weather conditions, lighting conditions, and time periods). Then, a certain percentage (e.g., 30%) of the images are randomly selected for manual annotation. The location and size of any foreign objects are marked using bounding boxes, resulting in multiple partially labeled images. If there are multiple foreign objects on the runway in an image, the image will have multiple ground truth labels, each containing the bounding box of one foreign object.

[0062] Foreign objects (FOOs) are objects or materials found on airport runways that may pose a threat to aircraft operational safety. FEOs include natural FEOs, man-made FEOs, and environmental FEOs. Natural FEOs include items such as gravel, branches, animal carcasses, and sand. Man-made FEOs include tools left behind by maintenance personnel (such as wrenches and screwdrivers), parts detached from aircraft or equipment (such as bolts and nuts), plastic products, metal fragments, paper, and packaging materials. Environmental FEOs include snow or ice accumulation on runways in winter, rainwater or melted snow, and oil spills from aircraft or vehicles.

[0063] S202. Randomly select multiple bounding box images of foreign objects from the database, paste the multiple bounding box images of foreign objects into the runway area of ​​each partially marked image, and obtain multiple marked images.

[0064] In the specific implementation process, firstly, the bounding boxes of the target foreign object can be extracted from images with real labels. The images with real labels can come from internal and external datasets. The internal dataset refers to multiple partially labeled images manually labeled in S201, while external datasets include publicly available datasets such as COCO and Pascal VOC. The target foreign object can refer to a foreign object with an actual size smaller than a preset size (e.g., small foreign objects such as stones or fragments), or it can refer to foreign objects of all sizes (including large and small foreign objects).

[0065] Secondly, the image and the bounding box of the target foreign object can be input into SAM v2. SAM v2 is a general segmentation model based on deep learning that can generate a high-quality segmentation mask based on cues (such as bounding boxes, points, or text). The segmentation mask Mi is a binary image, where... They represent different objects.

[0066] Then, based on the segmentation mask output by SAM v2, the bounding box images of the target foreign object can be segmented from the image and stored in the database, thus pre-storing bounding box images of foreign objects smaller than a preset size in the database. Finally, m bounding box images of foreign objects (0 ≤ m ≤ 5) can be randomly selected from the database and pasted into the runway area. To ensure that the bounding box images of the entire foreign object are located within the runway area, the pasting position must meet the following conditions:

[0067]

[0068]

[0069] in, Let the coordinates be the top-left corner of the bounding box image of the foreign object. The coordinates of the bottom right corner of the bounding box image of the foreign object are given. This indicates the runway area.

[0070] Considering that if there are already many real labels (e.g., multiple foreign objects are already labeled in the image), adding too many foreign objects may obscure or interfere with the accuracy of these existing labels. Furthermore, for images with a large number of labels, adding too many foreign objects may make the image appear unnatural and reduce the realism of the data. Therefore, in one possible embodiment, the number of foreign objects randomly selected from the database can be determined based on the number of real labels in each partially labeled image. The number of real labels is negatively correlated with the number of foreign objects; in other words, the more real labels in each partially labeled image, the fewer foreign objects will be randomly selected from the database.

[0071] For example, a partially labeled image has 0 real labels and 5 foreign objects randomly selected from the database; another partially labeled image has 1 real label and 4 foreign objects randomly selected from the database.

[0072] In this embodiment, when the number of true labels for an image is large, selecting fewer foreign objects can help maintain the accuracy of the annotation and the realism of the image, avoiding confusion or computational burden caused by overly complex scenarios during model training. This approach improves model training efficiency, enhances its generalization ability, avoids data bias, and ultimately improves the model's performance in practical applications.

[0073] S203. Pre-training is performed based on unlabeled images from multiple images and the MoCo v2 framework, and the weights obtained from the pre-training are transferred to the foreign object detection model.

[0074] In practical implementation, the foreign object detection model can adopt the YOLOv8 model. Since the YOLOv8 model's backbone network is an improved version based on CSPDarknet, its structure is highly compatible with CSPDarknet, and pre-trained CSPDarknet weights can be directly loaded. Therefore, based on unlabeled images and a MoCo v2 pre-trained CSPDarknet backbone network, MoCo v2 (Momentum Contrast v2) is a self-supervised learning framework based on contrastive learning, used to learn high-quality visual representations from unlabeled data.

[0075] The MoCo v2 framework uses a momentum encoder to generate feature representations for query samples and key samples. Query samples refer to the input samples in the current batch, while key samples are the feature representations corresponding to the query samples generated by the momentum encoder, typically from samples in historical batches or queues. A contrastive loss function is used to optimize the model, maximizing the similarity between query samples and positive samples (i.e., key samples corresponding to the query samples), while minimizing the similarity between query samples and negative samples (i.e., key samples from different samples). The contrastive loss function is defined as:

[0076]

[0077] in, This represents the similarity between the query sample q and the key sample k. Indicates a negative sample. Indicates a negative sample. τ The temperature coefficient, typically set to 0.07, is used to adjust the smoothness of the similarity distribution. Lower temperatures amplify similarity differences, thus enhancing the contrast effect. Negative samples are dynamically updated through a queue mechanism to maintain their diversity and timeliness.

[0078] After pre-training, the weights of the CSPDarknet backbone network are transferred to the YOLOv8 model backbone network as initial weights. The pre-trained model has learned rich feature representations on a large-scale dataset, which can provide better initialization for the YOLOv8 model, thereby accelerating the training process and improving the model's performance in subsequent foreign object detection tasks.

[0079] S204. Fine-tune the foreign object detection model based on multiple labeled images to obtain the final trained foreign object detection model.

[0080] In the specific implementation process, after obtaining the initial weights of the foreign object detection model, each labeled image can be input into the foreign object detection model, and the predicted label can be output. Based on the error between the true label and the predicted label of each labeled image, the initial weights of the foreign object detection model are adjusted until the preset conditions are met (such as reaching the maximum number of iterations, meeting the preset accuracy, etc.), and the final trained foreign object detection model is obtained.

[0081] S205. Input the real-time image of the target runway into the finally trained foreign object detection model, and output the detection result of whether the target runway contains foreign objects.

[0082] In practice, the real-time target runway image captured by any camera is preprocessed, typically including image resizing, normalization, and color space conversion to ensure it matches the input format used during training. The processed image is then input into the final trained foreign object detection model. If the model detects a foreign object, it outputs the object's bounding box and confidence score.

[0083] The central server can also issue graded alerts for detected foreign objects:

[0084] 1. High priority: Foreign objects with a confidence level ≥ 0.9 (such as metal tools, large pieces of gravel).

[0085] 2. Medium priority: Foreign objects with a confidence level of 0.7 ≤ confidence level < 0.9 (such as small fragments, glass shards).

[0086] 3. Low priority: Foreign objects with a confidence level of <0.7.

[0087] The central server can trigger an alarm in the control tower, attracting the attention of staff through sound and light alarms, and sending the location and priority of foreign objects to ground staff so that they can prepare accordingly (e.g., bring the appropriate cleaning equipment) and quickly locate and remove the foreign objects based on their location.

[0088] In one possible embodiment, before performing S202, the method further includes the following steps:

[0089] For each partially labeled image, generate multiple cue points; input each partially labeled image and multiple cue points into SAM v2, and output multiple segmentation masks; sort the multiple segmentation masks in descending order of area, and select the top K segmentation masks; K is a positive integer greater than 0; determine the region corresponding to the segmentation mask with the highest confidence among the K segmentation masks as the runway region of each partially labeled image.

[0090] In the implementation process, each partially labeled image and the generated cue points are input into SAM v2. SAM v2 generates segmentation masks near each cue point based on the input cue points, thus producing multiple segmentation masks. Since the runway is usually the largest continuous region in the image, the top K segmentation masks with the largest areas are selected from all segmentation masks, prioritizing the retention of regions with larger coverage areas. Among the K candidate segmentation masks, the mask with the highest confidence is selected as the final result. If there is more than one mask with the highest confidence, the segmentation mask with the largest area is selected first.

[0091] For example, if the mask with the largest area has a low confidence level (e.g., 0.6), while the mask with the second largest area has a high confidence level (e.g., 0.9), the above method will still prioritize the mask with the second largest area to ensure segmentation accuracy.

[0092] It should be noted that K can be a fixed value (e.g., K=3), or it can be a dynamically changing value adjusted according to the actual scenario. For example, if the runway area is significantly larger than other objects, a smaller K value (e.g., K=1 or 2) can be chosen to improve efficiency.

[0093] In this embodiment of the application, considering that the runway usually occupies a large area in the image, preliminary screening is performed based on the area to exclude small areas that are missegmented (such as vegetation and shadows). Among the segmentation masks with similar areas, the high-confidence mask is more likely to accurately fit the runway boundary, avoiding noise introduced by area calculation errors, thereby reducing the risk of missegmentation (such as background or other irrelevant areas).

[0094] In one possible embodiment, the step of generating multiple cue points based on each partially labeled image includes:

[0095] Based on the bounding box of each partially labeled image, multiple prompt points are randomly generated that meet preset conditions; the preset conditions include that the prompt point is not inside the bounding box and the distance between the prompt point and the bounding box is greater than a preset distance.

[0096] In the specific implementation process, the bounding boxes of all foreign objects in each partially labeled image are extracted, and each bounding box is represented by four vertices: ,in, and These are the diagonal vertices of the bounding box. and These are the other diagonal vertices of the bounding box.

[0097] For each partially labeled image, calculate the minimum bounding rectangle that contains all bounding boxes. The vertices of the minimum bounding rectangle are defined as follows: ,in, , .

[0098] Within the minimum bounding rectangle, multiple cue points are randomly selected. If these cue points are too close to the bounding box, the model may over-rely on object information within the bounding box, causing the segmentation results to be influenced by object features. Therefore, these cue points must meet the following two conditions:

[0099] The cue point is not located within any bounding box, i.e. ,in, This represents the region of the k-th bounding box, where n is the total number of bounding boxes in each partially labeled image. Let be the coordinates of the i-th prompt point.

[0100] The distance between the prompt point and each bounding box is greater than the preset distance. .

[0101] In this embodiment, by ensuring that the cue point is far from the bounding box, the model can avoid being affected by objects within the bounding box, thus focusing more on the features of the runway itself. This allows the model to effectively distinguish the differences between the runway and surrounding objects, improving its independent recognition ability of the runway region. Furthermore, maintaining a minimum distance between the cue point and the bounding box allows the model to better identify the specific location of the runway, especially when there are clear boundaries between the runway and surrounding objects. This helps the model accurately grasp the edges and shape of the runway, ensuring a more accurate segmentation mask.

[0102] In one possible embodiment, the specific steps of S202 include:

[0103] Multiple bounding box images of foreign objects are randomly selected from the database. Geometric transformations are performed on the bounding box images of the foreign objects to obtain multiple transformed bounding box images. The transformed bounding box images are pasted onto the runway area of ​​each partially labeled image to obtain multiple labeled images.

[0104] In practice, geometric transformations can be performed on the bounding box images of multiple foreign objects, such as rotating them by a random angle. By factor The bounding box image is scaled to obtain a transformed image, and its size is limited to a minimum of 10 pixels and a maximum of 200 pixels per side. Finally, the transformed bounding box image is pasted onto the runway area.

[0105] In this embodiment, by applying random geometric transformations (such as rotation and scaling), the diversity of pasted objects is increased, thereby increasing the diversity of training data, helping the model learn more diverse features, and improving the model's generalization ability.

[0106] In airport runway foreign object detection tasks, despite the availability of large amounts of manually annotated datasets, some small foreign objects (such as small stones) are often left unlabeled due to their minute size. These missing annotations introduce ambiguity during training, thereby reducing the model's detection performance. Therefore, this application proposes a pseudo-label generation strategy to enhance the training data by automatically generating high-quality pseudo-labels.

[0107] There are two specific implementation methods for S203, which will be described below.

[0108] The first method involves inputting each labeled image into the foreign object detection model and outputting a pseudo-label for each labeled image. The real and pseudo-labels of each labeled image are then fused to obtain a labeled image with fused labels. The foreign object detection model is then trained based on the labeled image with fused labels to obtain the final trained foreign object detection model.

[0109] In this embodiment, by fusing real and pseudo-labels, the model can be trained with more data, enhancing its generalization ability even when labeled data is scarce. Furthermore, the process of generating pseudo-labels requires no manual intervention, thus significantly reducing the workload of data labeling.

[0110] The second approach involves training a foreign object detection model based on multiple labeled images in the first stage to obtain an initially trained model. In the second stage, each labeled image is input into the initially trained model, and a pseudo-label for each image is output. The real and pseudo-labels of each labeled image are then fused to obtain a fused labeled image. The initially trained model is then trained based on the fused labeled image to obtain the final trained model.

[0111] In the embodiments of this application, in the first stage (e.g., rounds 1-100), the foreign object detection model is trained only on labeled images to obtain an initially trained foreign object detection model, thereby establishing basic target detection capabilities. This stage aims to achieve baseline performance, such as average accuracy (mAP), ensuring that the model can reliably identify larger foreign objects before fusing pseudo-labels.

[0112] In the second stage (e.g., rounds 101-200), the foreign object detection model after initial training can generate high-quality pseudo-labels (i.e., bounding boxes predicted by the model) based on the labeled images. By fusing the pseudo-labels with the real labels, that is, merging the pseudo-labels with the real labels into the labels of the labeled images, the model can continue to be trained, enhancing the model's ability to detect smaller foreign objects, especially tiny foreign objects that may be missed in the original annotations (such as small debris on the runway), thereby improving the model's generalization performance.

[0113] In one possible embodiment, each labeled image has multiple pseudo-labels; the step of fusing the real labels and pseudo-labels of each labeled image to obtain a labeled image with fused labels includes:

[0114] Delete pseudo-labels that are below the target confidence threshold from multiple pseudo-labels to obtain the remaining pseudo-labels for each labeled image; fuse the real labels and remaining pseudo-labels for each labeled image to obtain a labeled image with fused labels.

[0115] In the specific implementation process, the model generates multiple pseudo-labels for the labeled image. These pseudo-labels can be filtered. If the confidence level of a pseudo-label is greater than or equal to the target confidence threshold, the pseudo-label is retained as a remaining pseudo-label; otherwise, the pseudo-label is deleted. The real label and the remaining pseudo-labels are merged into the label of the labeled image to obtain the labeled image with fused labels.

[0116] In this embodiment of the application, considering that pseudo-labels with low confidence may cause the model to learn inaccurate information, while pseudo-labels with high confidence usually represent predictions that the model is more confident in, filtering out pseudo-labels with low confidence through a confidence threshold can reduce the negative impact of erroneous pseudo-labels on model training, thereby helping the model to obtain more reliable information during training.

[0117] In one possible embodiment, the real labels and remaining pseudo labels of each labeled image are fused to obtain a labeled image with fused labels, including:

[0118] Calculate the cross-union ratio (CUI) of the bounding boxes of the ground truth labels and the remaining pseudo-labels for each labeled image; if the CUI is less than the CUI threshold, then fuse the ground truth labels and pseudo-labels of each labeled image to obtain a labeled image with fused labels.

[0119] In the specific implementation process, for each labeled image, the bounding boxes of all ground truth labels and all remaining pseudo-labels are obtained. The intersection-over-union (IOU) ratio of the bounding boxes of any ground truth label and any remaining pseudo-label is calculated. The IoU is the ratio of the intersection region to the union region of two bounding boxes. If the IoU of the bounding box is greater than or equal to the IoU threshold, the corresponding remaining pseudo-label is deleted; if the IoU of the bounding box is less than the IoU threshold, the corresponding remaining pseudo-label is retained. This process continues until all ground truth labels and remaining pseudo-labels have been traversed. Then, all ground truth labels are fused with the retained remaining pseudo-labels to obtain a labeled image with fused labels. In other words, all ground truth labels and retained remaining pseudo-labels are used as labels for the labeled image.

[0120] In this embodiment, by calculating the Intersection over Union (IoU) between the real and pseudo-labels, when the IoU is greater than or equal to the Intersection over Union (IoU) threshold, it indicates that the region has been correctly labeled, and the pseudo-labels are deleted. This ensures that the labels in the dataset are unique and accurate, avoiding duplicate labeling of the same foreign objects. When the IoU is less than the IoU threshold, the pseudo-labels are retained and considered as omissions in the labeling. This approach can expand the labeled data, fill in areas not covered by the real labels, and thus improve the accuracy and comprehensiveness of model training.

[0121] As one example, the cross-union ratio (CUI) threshold is set to 0.2. A lower CUI threshold can ensure that even if there is a slight offset between the position of the pseudo-label and the real label, the reliability of the manual label can still be preserved first.

[0122] It should be noted that the target confidence threshold can be a pre-set fixed value (e.g., 0.85) or a dynamically changing value. In one possible embodiment, before deleting pseudo-labels less than the target confidence threshold from multiple pseudo-labels to obtain the remaining pseudo-labels for each labeled image, the method further includes:

[0123] Based on the current training round and the total number of training rounds, the initial confidence threshold is adjusted to obtain the target confidence threshold; the target confidence threshold is negatively correlated with the current training round.

[0124] In practical implementation, the formula for calculating the target confidence threshold is as follows:

[0125]

[0126] Where, θ new θ is the target confidence threshold. initial As the initial confidence threshold (e.g., 0.85), N epochs For the total number of training rounds, This is the current training round.

[0127] In this embodiment, considering that the model becomes stronger with each training iteration, enabling it to better handle high-confidence pseudo-labels while also extracting valuable information from lower-confidence pseudo-labels, the confidence threshold is gradually lowered with each training iteration to incorporate more pseudo-labels while maintaining quality.

[0128] It should be noted that the second stage can be divided into an intermediate training stage (such as rounds 101-150) and a final stage (such as rounds 151-200), which will be introduced below.

[0129] In the first case, the intermediate training phase, and the final phase, a fixed target confidence threshold is used.

[0130] Throughout the entire second phase of training, whether in the intermediate or final stages, the target confidence threshold remains constant, for example, 0.85. Training the model under higher confidence conditions yields higher detection accuracy.

[0131] In the second scenario, dynamic target confidence thresholds are used in the intermediate training phase and the final phase.

[0132] Throughout the second phase of training, the target confidence threshold decreases continuously as the current training epoch increases. That is, the initial confidence threshold is adjusted based on the current training epoch and the total number of training epochs to obtain the target confidence threshold.

[0133] In the third scenario, a fixed target confidence threshold is used during the intermediate training phase, while a dynamic target confidence threshold is used in the final phase.

[0134] In the embodiments of this application, in the early stage of training, a fixed high confidence threshold can ensure that the model learns more important features. In the later stage of training, as the model matures and converges, the confidence threshold can be appropriately reduced, so as to learn more complex details and samples that are not easy to classify.

[0135] In one possible implementation, the central server provides a manual annotation interface to support rapid correction of low-confidence detection results. It can also provide a model monitoring panel to visualize metrics such as average detection accuracy (mAP) and false positive rate during training. Newly annotated foreign object data can be periodically added to the training set, and model weights can be updated through incremental learning. For specific environments (such as rainy or snowy weather), a pre-trained, fine-tuned model can be loaded to adapt to scene changes.

[0136] In summary, this application provides a semi-supervised learning-based method for detecting foreign objects (FOOs) on airport runways. To initialize the YOLOv8 model with high-quality feature representations, the MoCo v2 framework is used for self-supervised pre-training. This helps the model learn useful features from unlabeled images, reducing reliance on manually annotated data. Manual annotations often overlook small FAOs in runways. By dynamically fusing pseudo-labels and real labels, the detection capability for small FAOs is enhanced, especially for tiny FAOs that may be missed in the original annotations. Furthermore, by pasting small FAOs into the runway area, the diversity of training data is increased, further improving the model's ability to detect small FAOs in complex environments.

[0137] Based on the same inventive concept, such as Figure 3 As shown in the figure, this application embodiment also provides an airport runway foreign object detection device based on semi-supervised learning, including:

[0138] The annotation module is used to collect multiple images of the airport runway, manually annotate some of the images, and obtain multiple partially labeled images with real labels; the real labels are the bounding boxes of foreign objects.

[0139] The pasting module is used to randomly select multiple bounding box images of foreign objects from the database, and paste the bounding box images of the multiple foreign objects onto the runway area of ​​each partially marked image to obtain multiple marked images; the database contains bounding box images of foreign objects smaller than a preset size.

[0140] The pre-training module is used to pre-train based on the unlabeled images in the multiple images and the MoCo v2 framework, and transfer the weights obtained from the pre-training to the foreign object detection model;

[0141] The fine-tuning training module is used to fine-tune the foreign object detection model based on the multiple labeled images to obtain the final trained foreign object detection model.

[0142] The detection module is used to input the real-time target runway image into the finally trained foreign object detection model and output the detection result of whether the runway contains foreign objects.

[0143] It should be noted that each module in the airport runway foreign object detection device based on semi-supervised learning in this embodiment corresponds one-to-one with each step in the airport runway foreign object detection method based on semi-supervised learning in the aforementioned embodiment. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned airport runway foreign object detection method based on semi-supervised learning, and will not be repeated here.

[0144] Furthermore, in one embodiment, this application also provides a computer device, the computer device including a processor, a memory and a computer program stored in the memory, the computer program being executed by the processor to implement the aforementioned airport runway foreign object detection method based on semi-supervised learning.

[0145] In addition, in one embodiment, this application also provides a computer storage medium storing a computer program, which is executed by a processor to implement the aforementioned semi-supervised learning-based foreign object detection method for airport runways.

[0146] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a device including one or any combination of the above-mentioned memories. The computer may be a variety of computing devices, including smart terminals and servers.

[0147] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0148] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0149] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0150] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0151] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a multimedia terminal device (which may be a mobile phone, computer, television receiver, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0153] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for detecting foreign objects on airport runways based on semi-supervised learning, characterized in that, include: Multiple images of an airport runway are collected, and some of these images are manually labeled to obtain multiple partially labeled images with real labels; the real labels are bounding boxes of foreign objects, and foreign objects are present in all of the images. Multiple bounding box images of foreign objects are randomly selected from the database, and these images are pasted onto the runway area of ​​each partially marked image to obtain multiple marked images; the database contains bounding box images of foreign objects smaller than a preset size. Before randomly selecting multiple bounding box images of foreign objects from the database, pasting the bounding box images of the multiple foreign objects onto the runway region of each partially labeled image, and obtaining multiple labeled images, the method further includes: Based on each partially labeled image, multiple cue points are generated; the generation of multiple cue points based on each partially labeled image includes: Based on the bounding box of each partially labeled image, multiple prompt points are randomly generated that meet preset conditions; the preset conditions include that the prompt point is not inside the bounding box and the distance between the prompt point and the bounding box is greater than a preset distance. Each partially labeled image and the multiple cue points are input into SAM v2, and multiple segmentation masks are output. The multiple segmentation masks are sorted in descending order of area, and the top K segmentation masks are selected; K is a positive integer greater than 0. The region corresponding to the segmentation mask with the highest confidence among the K segmentation masks is determined as the runway region of each partially labeled image; Pre-training is performed based on the unlabeled images in the multiple images and the MoCo v2 framework, and the weights obtained from the pre-training are transferred to the foreign object detection model. The foreign object detection model is fine-tuned and trained based on the multiple labeled images to obtain the final trained foreign object detection model. The real-time target runway image is input into the finally trained foreign object detection model, which outputs the detection result of whether the runway contains foreign objects.

2. The airport runway foreign object detection method based on semi-supervised learning as described in claim 1, characterized in that, The process involves randomly selecting multiple bounding box images of foreign objects from the database, pasting these bounding box images onto the runway region of each partially labeled image, and obtaining multiple labeled images, including: Randomly select multiple bounding box images of foreign objects from the database, and perform geometric transformations on the bounding box images of the multiple foreign objects respectively to obtain multiple transformed bounding box images; The transformed bounding box images are pasted onto the runway area of ​​each partially labeled image to obtain multiple labeled images.

3. The airport runway foreign object detection method based on semi-supervised learning as described in claim 1, characterized in that, The step of fine-tuning and training the foreign object detection model based on the multiple labeled images to obtain the final trained foreign object detection model includes: In the first stage, the foreign object detection model is trained based on the multiple labeled images to obtain the initially trained foreign object detection model; In the second stage, each labeled image is input into the initially trained foreign object detection model, and pseudo-labels for each labeled image are output. The real and pseudo labels of each labeled image are fused to obtain a labeled image with fused labels; The initially trained foreign object detection model is trained based on the labeled images fused with the labels to obtain the final trained foreign object detection model.

4. The airport runway foreign object detection method based on semi-supervised learning as described in claim 3, characterized in that, Each labeled image has multiple pseudo-labels; the process of fusing the real labels and pseudo-labels of each labeled image to obtain a labeled image with fused labels includes: Delete pseudo-labels that are lower than the target confidence threshold from multiple pseudo-labels to obtain the remaining pseudo-labels for each labeled image; The real labels and remaining pseudo labels of each labeled image are fused to obtain a labeled image with fused labels.

5. The airport runway foreign object detection method based on semi-supervised learning as described in claim 4, characterized in that, The process of fusing the real and pseudo labels of each labeled image to obtain a labeled image with fused labels includes: Calculate the intersection-union ratio (IoU) of the bounding boxes of the ground truth labels and the remaining pseudo labels for each labeled image; If the cross-union ratio of the bounding boxes is less than the cross-union ratio threshold, then the real labels and the remaining pseudo labels of each labeled image are fused to obtain a labeled image with fused labels.

6. The airport runway foreign object detection method based on semi-supervised learning as described in claim 4, characterized in that, Before deleting pseudo-labels below the target confidence threshold from multiple pseudo-labels to obtain the remaining pseudo-labels for each labeled image, the method further includes: Based on the current training round and the total number of training rounds, the initial confidence threshold is adjusted to obtain the target confidence threshold; the target confidence threshold is negatively correlated with the current training round.

7. The airport runway foreign object detection method based on semi-supervised learning as described in claim 6, characterized in that, The formula for calculating the target confidence threshold is as follows: Where, θ new θ is the target confidence threshold. initial N is the initial confidence threshold. epochs The total number of training rounds, This refers to the current training round.

8. A foreign object detection device for airport runways based on semi-supervised learning, characterized in that, include: The annotation module is used to collect multiple images of the airport runway, manually annotate some of the images, and obtain multiple partially labeled images with real labels; the real labels are the bounding boxes of foreign objects, and foreign objects exist in all the images. The pasting module is used to randomly select multiple bounding box images of foreign objects from the database, and paste the bounding box images of the multiple foreign objects onto the runway area of ​​each partially marked image to obtain multiple marked images; the database contains bounding box images of foreign objects smaller than a preset size. Before randomly selecting multiple bounding box images of foreign objects from the database, pasting these bounding box images onto the runway region of each partially labeled image, and obtaining multiple labeled images, the process further includes: Based on each partially labeled image, multiple cue points are generated; the generation of multiple cue points based on each partially labeled image includes: Based on the bounding box of each partially labeled image, multiple prompt points are randomly generated that meet preset conditions; the preset conditions include that the prompt point is not inside the bounding box and the distance between the prompt point and the bounding box is greater than a preset distance. Each partially labeled image and the multiple cue points are input into SAM v2, and multiple segmentation masks are output. The multiple segmentation masks are sorted in descending order of area, and the top K segmentation masks are selected; K is a positive integer greater than 0. The region corresponding to the segmentation mask with the highest confidence among the K segmentation masks is determined as the runway region of each partially labeled image; The pre-training module is used to pre-train based on the unlabeled images in the multiple images and the MoCo v2 framework, and transfer the weights obtained from the pre-training to the foreign object detection model; The fine-tuning training module is used to fine-tune the foreign object detection model based on the multiple labeled images to obtain the final trained foreign object detection model. The detection module is used to input the real-time target runway image into the finally trained foreign object detection model and output the detection result of whether the runway contains foreign objects.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the airport runway foreign object detection method based on semi-supervised learning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the processor executes the computer program to implement the airport runway foreign object detection method based on semi-supervised learning as described in any one of claims 1-7.

Images