A detection method based on vehicle-mounted FOD detection system

Abstract

A detection method based on a vehicle-mounted FOD detection system is provided, which belongs to the field of intelligent detection technology, and includes: constructing and training a foreign object recognition model constructed based on the PatchCore algorithm; a patrol vehicle is driving on the runway, a high-definition camera collects video images of the runway, and a trained foreign object recognition model is used to detect whether there are foreign objects in the video images; if a foreign object on the runway is found, an alarm is triggered and a pop-up window is displayed on the monitoring client in the patrol vehicle, and the position of the foreign object on the runway in the video image is framed, and the actual position of the foreign object on the runway is given; the control personnel verify the identified foreign object on the runway, and if it is a real foreign object, a cleaning operation is arranged according to the actual position of the foreign object on the runway; otherwise, the image of the falsely reported foreign object on the runway is submitted to the model upgrade program, and the model upgrade program is used to upgrade the model online based on the falsely reported image of the foreign object on the runway. The present invention can realize accurate detection of foreign objects on the airport runway with low implementation cost.

Description

Technical Field

[0001] The present invention relates to the field of intelligent detection technology, and in particular to a detection method based on a vehicle-mounted FOD detection system. Background Art

[0002] With the rapid development of the aviation industry, the safety management of airport runways is particularly important. Foreign Object Debris (FOD) is one of the main factors threatening flight safety. Foreign objects on the runway may cause serious damage to the aircraft or even cause a major accident. Therefore, efficient and accurate FOD detection technology has become the key to ensuring flight safety.

[0003] Traditional FOD detection usually includes manual inspection, tower inspection, and edge light inspection:

[0004] Manual inspection: When the runway is not in operation, airport staff drive a patrol vehicle along the runway to manually inspect whether there are foreign objects on the runway. This method of inspection is time-consuming and inefficient, and because the vehicle is in motion, it is difficult for the human eye to observe small foreign objects, so this method also has the risk of missing reports.

[0005] Tower type: A tower is erected along the drainage ditch beside the runway, and sensors such as radar and cameras are installed on the tower to perform real-time scanning of foreign objects on the runway. This method has problems such as high cost (requires a lot of infrastructure), easy false alarms (largely affected by environmental factors such as weather and light), missed alarms (cannot detect thin foreign objects such as patches, coins, etc.; at the same time, when there are aircraft waiting on the taxiway, they will block the sensors on the tower and cannot see the runway behind, resulting in missed alarms), and difficult to handle (a series of follow-up measures are required after the foreign objects are found).

[0006] Edge light type: radar, camera and other sensors are installed along the edge lights on both sides of the runway to scan foreign objects on the runway in real time. This method uses the optical fiber of the edge lights and is closer to the runway than the tower. The overall performance is higher than the tower type, but this method still has problems such as high cost (requires a lot of infrastructure), affects runway operations (requires complete suspension of flights for construction), missed reports (cannot detect thin foreign objects such as iron sheets, coins, etc.), and difficult to handle (a series of follow-up measures are required after foreign objects are found).

[0007] It can be seen that how to improve the accuracy and economy of FOD detection is the main technical problem that still needs to be solved. Summary of the invention

[0008] The present invention aims to provide an efficient, accurate and cost-effective detection method based on a vehicle-mounted FOD detection system. By installing a carbon fiber crossbar and multiple high-definition monitoring cameras on a patrol vehicle, combined with advanced video analysis algorithms, the detection and automatic alarm of foreign objects on the runway can be achieved, overcoming the problems of low accuracy, high cost and difficult processing in the existing technology.

[0009] The present invention provides a detection method based on a vehicle-mounted FOD detection system, wherein the vehicle-mounted FOD detection system comprises a patrol vehicle, a carbon fiber cross bar, and a high-definition camera, wherein the carbon fiber cross bar is transversely mounted on the patrol vehicle, and a plurality of high-definition cameras are mounted on the carbon fiber cross bar; a fill light is also mounted on the carbon fiber cross bar, and the fill light and the high-definition camera are alternately arranged; the method comprises the following steps:

[0010] Build and train a foreign body recognition model based on the PatchCore algorithm;

[0011] The patrol car is driving on the runway, and the high-definition camera collects video images of the runway, and uses the trained foreign object recognition model to detect whether there are foreign objects in the video images;

[0012] If a foreign object is found on the runway, an alarm will be sounded and a window will pop up on the monitoring client in the patrol vehicle, and the position of the foreign object on the runway in the video image will be outlined, and the actual position of the foreign object on the runway will be given;

[0013] The control personnel verify the identified foreign objects on the runway. If they are real foreign objects, they arrange cleaning operations according to the actual location of the foreign objects on the runway. If they are not real foreign objects, the falsely reported image of the runway foreign object is submitted to the model upgrade program, which is used to upgrade the model online based on the falsely reported image of the runway foreign object.

[0014] Optionally, the training of the foreign body recognition model includes:

[0015] Collecting runway images, wherein the runway images are mainly normal runway images and supplemented by images with foreign objects on the runway;

[0016] Dividing the collected runway image into a plurality of small patches, and extracting a patch feature vector from each small patch using a pre-trained model;

[0017] Constructing a memory library, and storing normal feature vectors, abnormal feature vectors, and feature vector sampling in the memory library, and constructing a training data set based on the data stored in the memory library, wherein the training data set includes positive samples and negative samples;

[0018] Training model threshold: Use the training data set and the test data set to train the foreign body recognition model, traverse different thresholds in turn, calculate the accuracy, recall rate, specificity and F1 score according to the performance evaluation indicators, and automatically calculate the optimal threshold through the F1 score until the foreign body recognition model training is completed.

[0019] Optionally, before using the trained foreign body recognition model to detect whether there is a foreign body in the video image, the method further includes:

[0020] A video image of a runway captured by a high-definition camera is obtained, and denoising is performed on the video image.

[0021] Optionally, the using the trained foreign object recognition model to detect whether there is a foreign object in the video image includes:

[0022] Dividing the video image into a plurality of patch blocks, and extracting a real-time patch feature vector from each patch block;

[0023] Compare each real-time patch feature vector of the video image with the patch feature vector in the memory bank, and calculate the distance between them;

[0024] The abnormality score of the corresponding patch is determined according to the distance, and each abnormality score is comprehensively calculated to obtain a comprehensive abnormality score. If the comprehensive abnormality score is greater than an optimal threshold, it is determined that there is a runway foreign object in the video image.

[0025] The present invention also provides an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the method as described in any of the preceding items.

[0026] The present invention also provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, any of the above methods is executed.

[0027] The present invention also provides a computer program product, wherein the computer program product stores a computer program for implementing any of the above methods.

[0028] The beneficial technical effects of the present invention are:

[0029] 1) Whether it is a newly built runway or an already operating runway, there will be a patrol vehicle. The present invention modifies the existing patrol vehicle. The length of the carbon bar cross bar can cover the width of various runways. Therefore, the patrol vehicle can detect foreign objects on various runways without stopping. On the other hand, it also reduces the system cost, so that all types of approaches can use this system.

[0030] 2) Compared with the solution of using only a single camera, the present invention uses multiple high-definition cameras, so that the coverage area of ​​a single high-definition camera is smaller, thereby increasing the area of ​​foreign objects in the image, making it easier for foreign objects to be detected by the algorithm. The high-definition camera has a top-view perspective, so that thin-sheet objects (such as patches, coins, etc.) that cannot be detected by the horizontal perspective in the conventional method also occupy a larger area in the image, so that such foreign objects can also be detected by the algorithm, ensuring that the system is not leaked.

[0031] The high-definition camera is installed at a low height and a fill light is added. At the same time, the relevant parameters are configured to adapt to the vehicle speed and weather environment, so that the system can operate stably under normal navigation conditions and is not affected by factors such as visibility and lighting environment.

[0032] 3) After discovering foreign objects, the control personnel can deal with them in a timely manner. Real foreign objects can be removed immediately, avoiding tedious steps such as remote scheduling in conventional tower and edge light solutions, and also improving the efficiency of on-site control. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 It is a structural schematic diagram of a vehicle-mounted FOD detection system disclosed in an embodiment of the present invention.

[0035] Figure 2 It is a flow chart of a detection method based on a vehicle-mounted FOD detection system disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0038] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, it is based on the orientation or position relationship shown in the accompanying drawings, or is the orientation or position relationship in which the product of the invention is usually placed when used. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the system or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0039] The terms "first", "second", "third", and "fourth" in the specification and claims of the present invention are used to distinguish different objects rather than to describe a specific order of the objects. For example, the first input, the second input, the third input, and the fourth input are used to distinguish different inputs rather than to describe a specific order of the inputs.

[0040] In the embodiments of the present invention, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0041] In the description of the embodiments of the present invention, unless otherwise specified, “multiple” means two or more than two. For example, multiple processing units refer to two or more processing units; multiple elements refer to two or more elements, etc.

[0042] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0043] The embodiment of the present invention provides a detection method based on a vehicle-mounted FOD detection system, such as Figure 1 As shown, the vehicle-mounted FOD detection system includes a patrol vehicle, a carbon fiber cross bar, and a high-definition camera. The carbon fiber cross bar is horizontally installed on the patrol vehicle, and several high-definition cameras are installed on the carbon fiber cross bar; fill lights are also installed on the carbon fiber cross bar, and the fill lights and the high-definition cameras are arranged alternately.

[0044] like Figure 2 As shown, the method comprises the following steps:

[0045] Build and train a foreign body recognition model based on the PatchCore algorithm;

[0046] The patrol car is driving on the runway, and the high-definition camera collects video images of the runway, and uses the trained foreign object recognition model to detect whether there are foreign objects in the video images;

[0047] If a foreign object is found on the runway, an alarm will be sounded and a window will pop up on the monitoring client in the patrol vehicle, and the position of the foreign object on the runway in the video image will be outlined, and the actual position of the foreign object on the runway will be given;

[0048] The control personnel verify the identified foreign objects on the runway. If they are real foreign objects, they arrange cleaning operations according to the actual location of the foreign objects on the runway. If they are not real foreign objects, the falsely reported image of the runway foreign object is submitted to the model upgrade program, which is used to upgrade the model online based on the falsely reported image of the runway foreign object.

[0049] The HD camera needs to be calibrated first: the internal and external parameters of the HD camera must be calibrated, and the relative position of the HD camera and the vehicle must be calibrated at the same time to calculate the position of the target relative to the HD camera, and the real GIS coordinates of the target are calculated based on the GIS coordinates of the vehicle, providing data support for the control personnel to deal with the subsequent foreign objects after they are discovered.

[0050] Exemplarily, the HD camera parameter configuration can be: the HD camera shoots vertically downward, the installation height is about 2 meters, the covered runway width is about 3 meters, and the vehicle is moving (calculated at 60 yards), so the exposure parameters need to be set (1 / 10000 seconds) to solve the image blur caused by vehicle movement. At the same time, the exposure time is short, and the corresponding amount of light input is greatly reduced. Therefore, a fill light must be installed next to the HD camera to increase the amount of light entering the lens and ensure the brightness of the image. At the same time, set the focal length of the HD camera (covering a range of 3 meters, thereby improving the accuracy of the actual target size described by each pixel, laying the foundation for subsequent video AI videos), and turn on anti-shake (to solve the problem of image blur caused by the HD camera head due to vehicle movement).

[0051] The foreign body recognition model in the present invention is built based on the PatchCore algorithm, which has powerful image anomaly detection capabilities and can achieve accurate detection of foreign bodies.

[0052] About the model upgrade procedure: mark the alarm images as correct alarms and false alarms respectively, and submit the images to the model to update the normal feature vectors and abnormal feature vectors in the model, thereby improving the accuracy and robustness of the model's foreign object detection for actual scenarios.

[0053] Optionally, the training of a foreign body recognition model constructed based on the PatchCore algorithm includes:

[0054] Collecting runway images, wherein the runway images are mainly normal runway images and supplemented by images with foreign objects on the runway;

[0055] These images should cover the runway conditions at different times, weather, and lighting conditions to ensure the generalization ability of the model. The collected images are then preprocessed, including resizing and normalization, to facilitate subsequent processing.

[0056] The collected runway image is divided into a plurality of small patches, and a patch feature vector is extracted from each small patch using a pre-trained model.

[0057] The extracted feature vectors can capture the local information of the image and are the basis for subsequent anomaly detection. The pre-trained model can be a convolutional neural network model (WideResNet).

[0058] A memory library is constructed, and normal feature vectors, abnormal feature vectors, and feature vector sampling are stored in the memory library. A training data set is constructed based on the data stored in the memory library, and the training data set includes positive samples and negative samples.

[0059] The normal feature vector refers to the patch feature vector extracted from the normal runway image, and the abnormal feature vector refers to the patch feature vector extracted from all runway images with foreign objects; and, in order to reduce the size of the memory library and speed up the inference speed, the Coreset Sampling and other methods can also be used to sample the two aforementioned patch feature vectors and select the most representative feature vector for storage.

[0060] The most representative feature vectors are extracted from the data stored in the memory bank, including positive samples (no foreign objects) and negative samples (with foreign objects), and a training data set is constructed based on them. In addition, a part of the training data set can be used as a test data set, or a test data set can be collected separately.

[0061] Training model threshold: Use the training data set and the test data set to train the foreign body recognition model, traverse different thresholds in turn, calculate the accuracy, recall rate, specificity and F1 score based on the performance evaluation indicators (True Positive (TP), True Negative (TN), False Positive (FP), False Negative (FN)), and automatically calculate the optimal threshold through the F1 score until the foreign body recognition model training is completed.

[0062] Precision = (TP + TN) / (TP + TN + FP + FN), Recall = TP / (TP + FN)), Specificity = TN / (TN + FP), F1 = (2 * (Recall * Specificity) / (Recall + Specificity).

[0063] Optionally, before using the trained foreign body recognition model to detect whether there is a foreign body in the video image, the method further includes:

[0064] A video image of a runway captured by a high-definition camera is obtained, and denoising is performed on the video image.

[0065] To ensure that the image quality meets the processing requirements, the original video images of the runway captured by the high-definition camera are denoised in preparation for subsequent automatic analysis.

[0066] Optionally, the using the trained foreign object recognition model to detect whether there is a foreign object in the video image includes:

[0067] Dividing the video image into a plurality of patch blocks, and extracting a real-time patch feature vector from each patch block;

[0068] Compare each real-time patch feature vector of the video image with the patch feature vector in the memory bank, and calculate the distance between them;

[0069] This distance can be used as a measure of the degree of abnormality: the larger the distance, the more different the patch in the test image is from the normal sample, and the more likely it is a foreign object.

[0070] The abnormality score of the corresponding patch is determined according to the distance, and each abnormality score is comprehensively calculated to obtain a comprehensive abnormality score. If the comprehensive abnormality score is greater than an optimal threshold, it is determined that there is a runway foreign object in the video image.

[0071] Optionally, dividing the collected runway image into a plurality of patches comprises:

[0072] Determine a runway type corresponding to the runway image, and determine a unit segmentation size according to the runway type;

[0073] The collected runway image is divided into a plurality of small patches according to the unit segmentation size.

[0074] There are many different types of runways in the airport, which are used for taking off and landing different types of aircraft. For example, the AF runway is used for taking off and landing large civil aircraft, the GK runway is used for taking off and landing medium-sized civil aircraft, and the LO runway is used for taking off and landing small propeller aircraft. That is, multiple runway types correspond to aircraft of different specifications. The larger the size of the aircraft, the greater the harm caused by tiny foreign objects on the runway to the aircraft (large aircraft fly faster and weigh more). At this time, the unit segmentation size should be set to be smaller, that is, the recognition of smaller runway foreign objects should be enhanced; and for smaller aircraft, the harm caused by tiny foreign objects on the runway to the aircraft is relatively smaller. At this time, the unit segmentation size can be set to be larger, that is, the recognition of smaller runway foreign objects can be appropriately weakened.

[0075] The type of runway may be pre-specified or determined based on actual recorded data of the types of aircraft taking off and landing on each runway, and the present invention does not make any specific limitation thereto.

[0076] The present invention also discloses an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the method as described in any of the preceding items.

[0077] The present invention also discloses a computer storage medium, on which a computer program is stored. When the computer program is run by a processor, any of the above methods is executed.

[0078] The present invention also discloses a computer program product, wherein the computer program product stores a computer program for implementing any of the above methods.

[0079] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0080] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0081] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0082] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A detection method based on a vehicle-mounted FOD detection system, characterized in that: The vehicle-mounted FOD detection system comprises a patrol vehicle, a carbon fiber crossbar, and a high-definition camera. The carbon fiber crossbar is transversely mounted on the patrol vehicle, and a plurality of high-definition cameras are mounted on the carbon fiber crossbar. A fill light is also mounted on the carbon fiber crossbar, and the fill light and the high-definition camera are arranged alternately. The method comprises the following steps: Build and train a foreign body recognition model based on the PatchCore algorithm; The patrol car is driving on the runway, and the high-definition camera collects video images of the runway, and uses the trained foreign object recognition model to detect whether there are foreign objects in the video images; If a foreign object is found on the runway, an alarm will be sounded and a window will pop up on the monitoring client in the patrol vehicle, and the position of the foreign object on the runway in the video image will be outlined, and the actual position of the foreign object on the runway will be given; The control personnel will verify the identified foreign objects on the runway. If they are real foreign objects, they will arrange the cleaning operation according to the actual location of the foreign objects on the runway; If it is not a real foreign object, submitting the falsely reported image of the runway foreign object to the model upgrade program, wherein the model upgrade program is used to perform online upgrade of the model based on the falsely reported image of the runway foreign object; Training of foreign body recognition models based on the PatchCore algorithm, including: Collecting runway images, wherein the runway images are mainly normal runway images and supplemented by images with foreign objects on the runway; Dividing the collected runway image into a plurality of small patches, and extracting a patch feature vector from each small patch using a pre-trained model; A memory library is constructed, and normal feature vectors, abnormal feature vectors, and feature vector sampling are stored in the memory library, and a training data set is constructed based on the data stored in the memory library, wherein the training data set includes positive samples and negative samples; wherein the normal feature vector refers to a patch feature vector extracted from a normal runway image, and the abnormal feature vector refers to a patch feature vector extracted from all runway images with foreign objects; Training model threshold: Use the training data set and the test data set to train the foreign body recognition model, traverse different thresholds in turn, calculate the accuracy, recall rate, specificity and F1 score according to the performance evaluation indicators, and automatically calculate the optimal threshold through the F1 score until the foreign body recognition model training is completed; The step of dividing the collected runway image into a plurality of patches comprises: Determine a runway type corresponding to the runway image, and determine a unit segmentation size according to the runway type; The collected runway image is divided into a plurality of small patches according to the unit segmentation size.

2. The method according to claim 1, characterized in that: Before using the trained foreign body recognition model to detect whether there is a foreign body in the video image, the method further includes: A video image of a runway captured by a high-definition camera is obtained, and denoising is performed on the video image.

3. The method according to claim 2, characterized in that: The using the trained foreign body recognition model to detect whether there is a foreign body in the video image includes: Dividing the video image into a plurality of patch blocks, and extracting a real-time patch feature vector from each patch block; Compare each real-time patch feature vector of the video image with the patch feature vector in the memory bank, and calculate the distance between them; The abnormality score of the corresponding patch is determined according to the distance, and each abnormality score is comprehensively calculated to obtain a comprehensive abnormality score. If the comprehensive abnormality score is greater than an optimal threshold, it is determined that there is a runway foreign object in the video image.

4. An electronic device comprising: A memory storing executable program code; A processor coupled to the memory; characterized in that: the processor calls the executable program code stored in the memory to execute the method according to any one of claims 1-3.

5. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is executed.

6. A computer program product, characterized in that: The computer program product stores a computer program for implementing the method according to any one of claims 1 to 3.

Images