A system for detecting foreign objects on a runway, a detection system and method thereof

Abstract

The present invention relates to a system for detecting a foreign object on a runway. The system comprises a thermal camera having a first field of view and adapted to capture a thermal image of a region of interest on the runway, a visible light camera having a second field of view and adapted to capture a visible light image of the region of interest on the runway, such that the first field of view overlaps the second field of view. The system further comprises a processor configured to detect a thermal object image in the thermal image, to detect a visible light object image in the visible light image, and to determine that a foreign object is detected when the thermal object image and the visible light object image are detected in the thermal image and the visible light image, respectively.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of Singapore Patent Application No. 10202009789R, filed on October 1, 2020, which is incorporated herein by reference. Technical Field

[0003] This invention relates to a system, a detection system for detecting foreign objects on a runway, and a method for using the system. Background Technology

[0004] Foreign object debris (FOD) on airport runways poses a hazard to aircraft landing and takeoff. FOD detection systems exist that use visible spectrum cameras to perform reliable FOD detection under normal, clear weather conditions. Under normal, clear weather conditions, such as in the absence of fog, FOD detection systems are able to capture high-resolution images of any FOD and process them for high-precision FOD detection. FOD can include engine and aircraft parts, tools, construction debris, rubber materials, natural materials, etc.

[0005] However, the performance of FOD detection systems can be adversely affected and impaired during adverse weather conditions, especially foggy conditions. FOD detection systems may fail to reliably detect FOD in foggy weather conditions (i.e., poor visibility conditions) because they operate only in the visible light spectrum. Therefore, under such conditions, such as through fog, FOD cannot be "seen," as fog typically reduces visibility along the runway to less than 1 km. Visibility conditions can be categorized into different types. For example, Cat II represents standard operations with an associated runway visual range (RVR) ranging from 550 m (1,800 ft) to 300 m (1,000 ft), Cat IIIa represents precision instrument approach and landing operations with an RVR of not less than 175 m (600 ft), Cat IIIb represents precision instrument approach and landing operations with an RVR of less than 175 m (600 ft) but not less than 50 m (600 ft), and Cat IIIc represents precision instrument approach and landing operations with no RVR limit (i.e., even zero visibility). Runway visibility at an airport can vary depending on its geographical location and is classified accordingly. While most FOD detection systems are capable of detecting FOD at airports with Cat II visibility, they cannot be used to detect airports with Cat IIIa, Cat IIIb, and Cat IIIc visibility.

[0006] In addition, FOD detection systems often generate invalid or false alarms. False alarms can be caused by phenomena, primarily light reflection, such as artificial light sources originating from nearby buildings or runway edge lights. This artificial light, reflected from smooth runway surfaces, puddles, or water accumulation on the runway surface, can cause the FOD detection system to identify it as FOD, thus resulting in a false alarm. The number of such false alarms due to reflection typically increases significantly after rainfall, when puddles or water accumulation are prevalent on the runway surface. While such reflections do occur during the day, they are more common at night and during dawn and dusk.

[0007] Therefore, it is important to provide a solution that can detect FOD during poor visibility conditions (such as adverse weather conditions) and prevent or minimize false detections of FOD. Summary of the Invention

[0008] According to various embodiments, a method for detecting foreign objects on a runway is provided. The method includes capturing a thermal image of a region of interest on the runway, capturing a visible light image of the region of interest on the runway, detecting a thermal object image in the thermal image, detecting a visible light object image in the visible light image, and determining that a foreign object has been detected when both the thermal object image and the visible light object image are detected in the thermal image and the visible light object image, respectively.

[0009] According to various embodiments, determining a foreign object may include at least one attribute of generating the foreign object in each of the thermal object image and the visible light object image, comparing at least one attribute of the foreign object in the thermal object image and the visible light object image, such that a foreign object is detected when at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

[0010] According to various embodiments, at least one property of the foreign object may include the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image.

[0011] According to various embodiments, when the distance between the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image is within the position parameter, at least one property of the foreign object in the thermal object image and the visible light object image is the same.

[0012] According to various embodiments, at least one property of the foreign object may include the size of the thermal object image and the visible light object image.

[0013] According to various embodiments, when the difference between the size of the thermal object image in the thermal image and the size of the visible light object image in the visible light image is within a size parameter, at least one property of the foreign object in the thermal object image and the visible light object image is the same.

[0014] According to various embodiments, the method may further include obtaining a magnified image of the thermal object and a magnified image of the visible light object when a foreign object is detected.

[0015] According to various embodiments, the method may further include identifying the object category of a foreign object in a thermal image, such that identifying the object category may include: segmenting the thermal image into multiple thermal image regions, assigning a feature vector to each of the multiple thermal image regions, comparing the feature vector with a multiple reference feature vector, such that each of the multiple reference feature vectors represents an object category, and identifying a reference feature vector that is closest to the feature vector and its object category.

[0016] According to various embodiments, segmenting a thermal image may include labeling each pixel in the thermal image and grouping labeled pixels with the same characteristics into multiple groups to form multiple thermal image regions.

[0017] According to various embodiments, the method may further include identifying the object category of a foreign object in a visible light image, such that identifying the object category may include: segmenting the visible light image into a plurality of visible light image regions; assigning a feature vector to each of the plurality of visible light image regions; comparing the feature vector with a plurality of reference feature vectors, such that each of the plurality of reference feature vectors represents an object category; and identifying a reference feature vector that is closest to the feature vector and its object category.

[0018] According to various embodiments, segmenting a visible light image may include labeling each pixel in the visible light image and grouping labeled pixels with the same characteristics into multiple groups to form multiple visible light image regions.

[0019] According to various embodiments, the method may also include training a thermal camera to detect foreign objects based on visible light images from a visible light camera.

[0020] According to various embodiments, training a thermal camera may include determining the relationship between the object category of a foreign object in a visible light image and the temperature of the foreign object in a thermal image.

[0021] According to various embodiments, a system for detecting foreign objects on a runway is provided. The system includes: a thermal camera having a first field of view and adapted to capture a thermal image of a region of interest on the runway; and a visible light camera having a second field of view and adapted to capture a visible light image of the region of interest on the runway, such that the first field of view overlaps with the second field of view. The system further includes: a processor in communication with the thermal camera and the visible light camera; and a memory in communication with the processor for storing instructions executable by the processor, such that the processor is configured to detect an image of a thermal object in the thermal image, detect an image of a visible light object in the visible light image, and determine that a foreign object has been detected when both the thermal image and the visible light image are detected in the thermal image and the visible light image, respectively.

[0022] According to various embodiments, in order to determine a foreign object, the processor can be configured to generate at least one attribute of the foreign object in each of a thermal object image and a visible light object image, compare at least one attribute of the foreign object in the thermal object image and the visible light object image, such that a foreign object is detected when at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

[0023] According to various embodiments, at least one property of the foreign object may include the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image.

[0024] According to various embodiments, when the distance between the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image is within the position parameter, at least one property of the foreign object in the thermal object image and the visible light object image is the same.

[0025] According to various embodiments, at least one property of the foreign object may include the size of the thermal object image and the visible light object image.

[0026] According to various embodiments, when the difference between the size of the thermal object image in the thermal image and the size of the visible light object image in the visible light image is within a size parameter, at least one property of the foreign object in the thermal object image and the visible light object image is the same.

[0027] According to various embodiments, the processor may also be configured to magnify the thermal camera and the visible light camera when a foreign object is detected to obtain magnified images of the thermal object and magnified images of the visible light object.

[0028] According to various embodiments, the processor can be configured to identify the object category of a foreign object in a thermal image, such that the processor can be configured to segment the thermal image into multiple thermal image regions, assign a feature vector to each of the multiple thermal image regions, compare the feature vector with a multiple reference feature vector, such that each of the multiple reference feature vectors represents an object category, and identify the reference feature vector that is closest to the feature vector and its object category.

[0029] According to various embodiments, in order to segment a thermal image, a processor can be configured to label each pixel in the thermal image and group labeled pixels with the same characteristics into multiple groups to form multiple thermal image regions.

[0030] According to various embodiments, the processor can also be configured to identify the object category of a foreign object in a visible light image, such that the processor can be configured to segment the visible light image into multiple visible light image regions, assign feature vectors to each of the multiple visible light image regions, compare the feature vectors with multiple reference feature vectors such that each of the multiple reference feature vectors represents an object category, and identify the reference feature vector that is closest to the feature vector and its object category.

[0031] According to various embodiments, in order to segment a visible light image, a processor can be configured to label each pixel in the visible light image and group labeled pixels with the same characteristics into multiple groups to form multiple visible light image regions.

[0032] According to various embodiments, the processor can also be configured to train a thermal camera to detect foreign objects based on visible light images from a visible light camera.

[0033] According to various embodiments, in order to train a thermal camera, the processor can be configured to determine the relationship between the object category of a foreign object in a visible light image and the temperature of the foreign object in a thermal image.

[0034] According to various embodiments, a detection system for detecting foreign objects on a runway divided into multiple sectors is provided. The detection system includes multiple sets of cameras spaced apart from each other. Each set of cameras may include: a thermal camera including a first field of view and adapted to capture a thermal image of a region of interest on the runway; a visible light camera including a second field of view and adapted to capture a visible light image of the region of interest on the runway, such that the first field of view overlaps with the second field of view; a processor communicating with the thermal camera and the visible light camera; and a memory communicating with the processor for storing instructions executable by the processor, such that the processor can be configured to detect a thermal object image in the thermal image, detect a visible light object image in the visible light image, and determine that a foreign object has been detected when both the thermal object image and the visible light object image are detected in the thermal image and the visible light object image, respectively, such that each set of cameras can be configured to scan one sector of a plurality of sectors of the runway. Attached Figure Description

[0035] Figure 1 A schematic diagram of an exemplary embodiment of a system for detecting foreign objects on a runway is shown.

[0036] Figure 1AA schematic diagram is shown of a visible light image of a visible light object containing foreign matter and a thermal image of a thermal object containing foreign matter.

[0037] Figure 2A An exemplary embodiment of the system is shown.

[0038] Figure 2B It shows Figure 2A The system scans one of the multiple sectors of the runway.

[0039] Figure 3 An exemplary embodiment of a detection system for detecting foreign objects on a runway divided into multiple sectors is shown.

[0040] Figure 4 A flowchart of an exemplary method for detecting foreign objects on a runway is shown.

[0041] Figure 5 A flowchart of an exemplary method for detecting foreign objects on a runway is shown.

[0042] Figure 6 A flowchart of an exemplary method for detecting foreign objects on a runway is shown.

[0043] Figure 7 A flowchart of an exemplary method for comparing at least one property of a foreign object in an image of a thermal object and an image of a visible light object is shown.

[0044] Figure 8 A flowchart is shown for a method used to identify foreign objects on the runway.

[0045] Figure 9 A flowchart is shown for a method used to identify foreign objects on the runway.

[0046] Figure 10 A flowchart is shown of a method for training an image recognition module to improve the identification of foreign objects on a runway.

[0047] Figure 11 A flowchart is shown for a method of detecting foreign objects using a thermal camera. Detailed Implementation

[0048] In the following examples, reference will be made to the accompanying drawings, in which the same features are indicated by the same reference numerals.

[0049] Figure 1A schematic diagram of an exemplary embodiment of a system 100 for detecting foreign object 20 on a runway is shown. The system 100 includes: a thermal camera 110 having a first field of view 110F and adapted to capture a thermal image 110M of a region of interest 112 on the runway; a visible light camera 120 having a second field of view 120F and adapted to capture a visible light image of the region of interest 112 on the runway, such that the first field of view 110F overlaps with the second field of view 120F; a processor 132 communicating with the thermal camera 110M and the visible light camera 120M; and a memory 134 communicating with the processor 132 for storing instructions executable by the processor 132, such that the processor 132 is configured to detect a thermal object image in the thermal image, detect a visible light object image in the visible light image, and determine that a foreign object 20 has been detected when both the thermal object image and the visible light object image are detected in the thermal image and the visible light object image, respectively. System 100 may include a server comprising a processor 132, memory 134, and an I / O interface 136 configured to provide an interface between the processor 132 and peripheral interface modules (e.g., keyboard, mouse, touchscreen, display, etc.). System 100 may include a communication module 138 configured to facilitate wired or wireless communication between system 100 and other user devices (e.g., mobile devices, laptop computers) via the Internet. System 100 may include a display, such as a monitor or touchscreen, for displaying signals, such as alarm signals, to an operator. System 100 is configured to detect foreign object debris (FOD) on runways, taxiways, aprons, ramps, etc., under ambient light conditions during the day and night without auxiliary lighting, such as visible spectrum illumination, infrared illumination, or laser illumination.

[0050] Figure 1A A schematic diagram is shown of a visible light image 120M of a visible light object image 120B containing foreign matter 20 and a thermal image 110M of a thermal object image 110B containing foreign matter 20.

[0051] System 100 may include an image processing module 134M (see Figure 1 The system 100 is configured to process images 110T and 120T captured from thermal camera 110 and visible light camera 120. The system 100 may include a thermal camera operation module 134T containing operating parameters for operating thermal camera 110. The system 100 may include a visible light camera operation module 134V containing operating parameters for operating visible light camera 120. Modules 134T, 134V, and 134M may be stored in storage device 140 and loaded into memory 134 for processing by processor 132.

[0052] When capturing thermal image 110M and visible light image 120M, images 110M and 120M can be transmitted to processor 132 for processing. Processor 132 can receive and process thermal image 110M and visible light image 120M to detect foreign objects 20 on the runway. System 100 enables the detection of foreign objects 20 at airports with Cat II visibility, Cat IIIa visibility, Cat IIIb visibility, and Cat IIIc visibility, enables the detection of foreign objects during poor visibility conditions (e.g., adverse weather conditions), and prevents or minimizes false detection of foreign objects.

[0053] Figure 2A An exemplary embodiment of system 200 is shown. System 200 may include a set of cameras, namely a visible light camera 220 and a thermal camera 210. The set of cameras 210S may be rigidly mounted on an actuator 250 adapted to move the set of cameras 210S. The set of cameras 210S may be controlled by processor 132 to scan sectors of the runway to detect foreign objects 20 on the surface of the runway.

[0054] Actuator 250 may be a translation and tilt unit (PTU) adapted to simultaneously translate and tilt the group of cameras 210S, enabling the group of cameras 210S to have the same field of view and focus on the same region of interest. Actuator 250 may be adapted to translate the group of cameras 210S in the horizontal direction 210H and / or tilt the group of cameras 210S in the vertical direction 210V. Actuator 250 may communicate with processor 132, such that processor 132 may be configured to remotely control the movement of actuator 250 to translate and tilt the group of cameras 210S to scan the runway. Actuator 250 may be mounted on top of support 252 (e.g., mast structure) typically positioned along the runway. The support may be located at a distance of 304 from the runway centerline (see...). Figure 3 At a distance of 120m-350m.

[0055] Figure 2B It shows Figure 2A The system 100 scans one of multiple sectors 202S of runway 202. Each group of cameras 210S may include a thermal camera 210 and a visible light camera 220, each camera having a field of view 210F, 220F and adapted to capture a region of interest 212. The field of view 210F of the visible light camera 220 may overlap with the field of view 220F of the thermal camera 210. Both the fields of view 210F and 220F of the visible light camera 220 and the thermal camera 210 may cover a specific region of interest 212 within sector 202S of runway 202. Therefore, due to the overlapping fields of view 210F, 220F, both the visible light camera 220 and the thermal camera 210 can detect the same foreign object 20 on the runway sector 202S while scanning the sector 202S.

[0056] Figure 3 An exemplary embodiment of a detection system 300 for detecting foreign objects 20 on a runway 302 divided into multiple sectors 302S is shown. The detection system 300 includes multiple sets of cameras 310S spaced apart from each other. Each set of cameras 310S includes: a thermal camera 210 having a first field of view 210F and adapted to capture a thermal image 110M of a region of interest 212 on the runway 302; and a visible light camera 220 having a second field of view 220F and adapted to capture a visible light image 120M of the region of interest 212 on the runway 302, such that the first field of view 210F overlaps with the second field of view 220F. The detection system 300 also includes a processor and a memory in communication with the processor for storing instructions executable by the processor, such that the processor is configured to detect a thermal object image 110B in a thermal image 110M, detect a visible light object image 120B in a visible light image 120M, and determine that a foreign object 20 has been detected when both the thermal object image 110B and the visible light object image 120B are detected in the thermal image 110M and the visible light object image 120B, respectively, such that each camera in the plurality of camera sets 310S is configured to scan one sector of a plurality of sectors 302S of the runway. Figure 3 As shown, runway 302 can be divided into multiple sectors 302S. Each of the multiple sets of cameras 310S can scan one of the multiple sectors 302S to detect any foreign object 20 on the surface of the corresponding sector 302S. Each of the multiple sectors 302S can be further divided into multiple sub-sectors. Each set of cameras 310S is operable to scan a dedicated sector 302S and scan the sector 302S sub-sector by sub-sector. In this way, when a foreign object 20 is detected, system 300 is able to identify sector 302S based on the set of cameras 310S that scanned sector 302S. The camera set 310S can scan sector 302S in a specific typical scanning direction, such as from the leftmost sub-sector to the rightmost sub-sector or from the rightmost sub-sector to the leftmost sub-sector.

[0057] Thermal camera 210 detects the foreign object 20 on runway 302 by detecting the difference in thermal radiation levels (or temperature) between the foreground (i.e., the foreign object 20) and the background (i.e., the runway surface). Thermal camera 210 operates in the infrared spectrum and requires no ambient light to "see" the foreign object 20. Thermal camera 210 may also be commonly referred to as an infrared thermal camera. Thermal camera 210 can be a mid-wave infrared (MWIR) camera or a long-wave infrared (LWIR) camera. Thermal camera 210 offers the advantage of detecting the foreign object 20 on runway 302 under very low visibility conditions and even under zero illumination conditions (i.e., complete darkness). Therefore, thermal camera 210 provides the advantage of being able to detect the foreign object 20 on runway 302 even in foggy weather conditions. Thermal camera 210 can capture images and monochrome video output and transmit them to processor 132. Thermal camera 210 is entirely passive, with no active transmission or emission, such as radio frequency, microwave, artificial illumination, infrared, laser, and LIDAR. Therefore, the thermal camera 210 offers the following advantages, such as not interfering with existing airport systems / equipment and aircraft systems / equipment, not interfering with future airport systems / equipment and aircraft systems / equipment, and not requiring permission and approval of frequencies / spectrums from airports and spectrum regulators.

[0058] Unlike the thermal camera 210, the visible light camera 220 operates within the visible spectrum and therefore requires a minimum amount of ambient visible light to be able to "see" foreign objects 20 on runway 302. The visible light camera 220 cannot detect any foreign objects 20 when visibility conditions are too poor or under zero illumination conditions. For example, the visible light camera 220 also cannot detect foreign objects 20 when visibility conditions (above the runway surface) are very poor or when fog is present (above the runway surface). The visible light camera 220 is capable of capturing and transmitting full-color and high-resolution images / videos, such as Full HD (FHD) or 4K Ultra HD (4K UHD) resolution. High-resolution color images enable reliable and accurate visual verification and confirmation of detected foreign objects 20 by an operator, and reliable and accurate identification / classification of detected foreign objects 20 by system 300. Therefore, the combined use of the visible light camera 220 and the thermal camera 210 enables the system 300 to operate under very low visibility conditions (e.g., foggy weather conditions), allowing the system 300 to accurately and reliably detect foreign objects 20 on the surface of runway 302. The visible light camera 220 is configured to capture color and high-resolution visible light images 120M and output them to the processor 132. The visible light camera 220 does not require any transmission of infrared illumination, visible spectrum illumination, or laser illumination to operate. Being passive, the system 300 offers the advantage that it does not pose any danger or cause any interference to other airport systems and / or aircraft systems, such as when aircraft are landing / taking off from runway 302. The system 300 provides the advantages of not interfering with existing airport systems / equipment and aircraft systems / equipment, not interfering with future airport systems / equipment and aircraft systems / equipment, and not requiring the license and approval of frequencies / spectrums needed by airports and spectrum regulators.

[0059] Figure 4A flowchart of an exemplary method 1000 for detecting a foreign object 20 on a runway is shown. The method includes capturing a thermal image 110M of a region of interest 112 on the runway in block 1010, capturing a visible light image 120M of the region of interest 112 on the runway in block 1020, detecting a thermal object image 110B in the thermal image 110M in block 1030, detecting a visible light object image 120B in the visible light image 120M in block 1040, and determining that a foreign object 20 has been detected when the thermal object image 110B and the visible light object image 120B are detected in the thermal image 110M and the visible light object image 120B, respectively, in block 1050. The thermal object image 110B may be a portion of the thermal image 110M representing the foreign object 20 in the thermal image 110M, and may be arbitrarily referred to as the foreign object 20 in the thermal image 110M. The visible light object image 120B may be a part of the visible light image 120M representing the foreign object 20 in the visible light image 120M, and may be arbitrarily referred to as the foreign object 20 in the visible light image 120M.

[0060] Before capturing thermal image 110M and visible light image 120M, the method may include scanning the runway using thermal camera 110 and visible light camera 120. As thermal camera 110 and visible light camera 120 scan sectors of the runway, they capture thermal image 110M and visible light image 120M along multiple regions of interest 112 of that sector. To detect foreign object 20, image processing module 134M may process thermal image 110M and visible light image 120M to determine the presence of foreign object 20 in both images. Upon detection of foreign object 20, image processing module 134M may be configured to identify thermal object image 110B and visible light object image 120B within thermal image 110M and visible light image 120M, respectively. Upon identification of foreign object 20, system 100 may generate an alarm signal.

[0061] To detect foreign object 20, the method may include generating at least one attribute of foreign object 20 in each of thermal object image 110B and visible light object image 120B, comparing at least one attribute of foreign object 20 in thermal object image 110B and visible light object image 120B, such that foreign object 20 is detected when at least one attribute of foreign object 20 in thermal object image 110B and visible light object image 120B is the same or within a specified parameter or threshold level. System 100 may be configured to obtain magnified thermal object image 110B and magnified visible light object image 120B when foreign object 20 is detected by zooming visible light camera 120 and thermal camera 110 onto the detected foreign object 20.

[0062] Figure 5A flowchart of an exemplary method 2000 for detecting foreign objects 20 on a runway is shown. System 100 can be configured to designate a visible light camera 120 as the primary detector and a thermal camera 110 as the secondary detector. Reference Figure 5 In block 2110, a visible light camera 120 can be configured to scan one of multiple sectors on the runway. The visible light camera 120 can be configured to scan sub-sectors of that sector partition by partition. The visible light camera 120 can capture multiple visible light images 120M within each sector. The multiple visible light images 120M can be processed by the image processing module 134M to detect foreign objects 20. In block 2210, a thermal camera 110 can be configured to scan the same sector on the runway scanned by the visible light camera 120. The thermal camera 110 can be configured to scan sub-sectors of that sector partition by partition. The thermal camera 110 can capture multiple thermal images 110M within the same sector. The multiple thermal images 110M can be processed by the image processing module 134M to detect foreign objects 20. The thermal camera 110 and the visible light camera 120 can be configured to scan sectors simultaneously. The image processing module 134M for visible light images and thermal images 110M can be a separate module for processing visible light images 120M and thermal images 110M, respectively.

[0063] In block 2120, system 100 can detect foreign object 20 after processing visible light image 120M. System 100 can identify visible light object image 120B within the visible image. In block 2220, system 100 can detect foreign object 20 after processing thermal image 110M. System 100 can identify thermal object image 110B within thermal image 110M. Thermal image 110M and visible light image 120M can be processed simultaneously by processor 132. If system 100 detects foreign object 20 in visible light image 120M, in block 2130, system 100 can generate a "suspected FOD" alarm signal to notify the operator that foreign object 20 has been detected in visible light image 120M. Similarly, if system 100 detects a foreign object 20 in thermal image 110M, system 100 can generate a "suspected FOD" alarm signal in box 2230 to notify the operator that a foreign object 20 has been detected in thermal image 110M, since the detection of foreign object 20 has not yet been verified. A "suspected FOD" signal can be generated for each of visible light image 120M and thermal image 110M. System 100 can display thermal object image 110B and / or visible light object image 120B on a display for operator viewing. System 100 can generate at least one attribute of visible light object image 120B and thermal object image 110B. At least one attribute may include the position of visible light object image 120B in visible light image 120M, the position of thermal object image 110B in thermal image 110M, the size of visible light object image 120B, and / or the size of thermal object image 110B. For example, system 100 can generate the position of a visible light object image 120B in a visible light image 120M and the position of a thermal object image 110B in a thermal image 110M, and / or the dimensions of the visible light object image 120B and the thermal object image 110B. In block 2140, system 100 can be configured to determine whether a foreign object 20 is detected in the visible light image 120M and the thermal image 110M by comparing at least one attribute of the visible light object image 120B and the thermal object image 110B. Details of this comparison step are provided in... Figure 7As shown in block 2150. If the attributes of the visible light object image 120B and the thermal object image 110B match in block 2150, then system 100 determines that a foreign object 20 has been detected in the visible light image 120B and the thermal image 110B. System 100 can receive operator verification input via a peripheral interface module to verify the detection of the foreign object 20 after viewing the visible light image 120B and / or the thermal image 110B on the display. In block 2160, system 100 can identify the foreign object 20 based on at least one attribute. Once the foreign object 20 is detected and / or identified, system 100 can generate an alarm signal, such as a "FOD confirmed" signal, in block 2170. Otherwise, system 100 can generate an "FOD unconfirmed" alarm signal. System 100 can send the alarm signal to the operator's mobile device or display the alarm signal on the display for the operator to view. System 100 can generate an alarm signal upon receiving operator verification input that a foreign object 20 has been detected.

[0064] Figure 6 A flowchart of an exemplary method 3000 for detecting foreign objects 20 on a runway is shown. Method 3000 and Figure 5 The method is the same as in 2000, except that system 100 is configured to designate thermal camera 110 as the primary detector and visible light camera 120 as the secondary detector. Figure 5 and Figure 6 The same reference numerals in the figures indicate the same steps. (See also...) Figure 6 In block 3110, thermal camera 110 can be configured to scan one of multiple sectors on the runway. Thermal camera 110 can be configured to scan sub-sectors of a sector one by one. Thermal camera 110 can capture multiple thermal images 110M within each sector. The multiple thermal images 110M can be processed by image processing module 134M to detect foreign object 20. In block 3210, visible light camera 120 can be configured to scan the same sector on the runway scanned by thermal camera 110. Visible light camera 120 can be configured to scan sub-sectors of a sector one by one. Visible light camera 120 can capture multiple visible light images 120M within that sector. The multiple visible light images 120M can be processed by image processing module 134M to detect foreign object 20. Thermal camera 110 and visible light camera 120 can be configured to scan sectors simultaneously. In block 3120, system 100 can detect foreign object 20 after processing thermal images 110M. System 100 can identify thermal object image 110B within thermal image 110M. In block 3230, system 100 can detect foreign object 20 after processing visible light image 120M. System 100 can identify visible light object image 120B within visible light image 120M. Thermal image 110M and visible light image 120M can be processed simultaneously by processor 132. Blocks 3140 to 3170 are related to... Figure 5The boxes 2140 to 2170 are the same.

[0065] Figure 7 A flowchart of an exemplary method 4140 for comparing at least one property of a foreign object 20 in a thermal object image 110B and a visible light object image 120B is shown. Figure 5 Method 2000 and Figure 6 Method 3000 uses method 4140 in blocks 2140 and 3140 respectively. At least one attribute of the foreign object 20 may include the position of the thermal object image 110B in the thermal image 110M and the position of the visible light object image 120B in the visible light image 120M. In block 4141, when the distance between the position of the thermal object image 110B in the thermal image 110M and the position of the visible light object image 120B in the visible light image 120M is within the position parameter, the foreign object 20 in the thermal object image 110B and at least one attribute in the visible light object image 120B can be considered the same. For example, processor 132 identifies the position difference between the positions of the visible light object image 120B and the thermal object image 110B in the visible light image 120M and the thermal image 110M, and determines whether the position difference between the positions is within the position parameter (i.e., a predefined position threshold level). The position parameter can be determined based on statistical analysis of the detected positions of all detected foreign object 20 samples. If the position difference is within the position parameters, the processor 132 can generate a "position match" alarm signal in box 4142.

[0066] At least one attribute of the foreign object 20 may include the size of the thermal object image 110B and the visible light object image 120B. In block 4143, when the difference between the size of the thermal object image 110B in the thermal image 110M and the size of the visible light object image 120B in the visible light image 120M is within a size parameter, at least one attribute of the foreign object 20 in the thermal object image 110B and the visible light object image 120B can be considered identical. For example, processor 132 identifies the size difference between the sizes of the visible light object image 120B and the thermal object image 110B in the visible light image 120M and the thermal object image 110B in the thermal image 110M, and determines whether the size difference between locations is within a size parameter (i.e., a predefined size threshold level). The size parameter can be determined based on a statistical analysis of the measured sizes of all detected foreign object 20 samples. If the size difference is within the size parameter, processor 132 can generate a “size match” alarm signal in block 4144.

[0067] Depending on the configuration of system 100, the process can detect foreign object 20 based on the position and / or size of thermal object image 110B and visible light object image 120B. For example, when using both the position and size of thermal object image 110B and visible light object image 120B, foreign object 20 is detected when the position and size of thermal object image 110B and visible light object image 120B are within the position parameter and size parameter, respectively (i.e., matched). System 100 can generate an alarm signal when foreign object 20 is detected, such as generating an "attribute match" signal when attribute matching occurs in box 4145. System 100 can generate alarm signals when both the "position match" alarm signal and the "size match" alarm signal are enabled or generated.

[0068] The exemplary system 100 and method described above provide a solution that enables the detection of foreign objects 20 during adverse weather conditions and prevents or minimizes false detections of foreign objects 20. For example, reflections from puddles or puddles after rainfall, or reflections from a smooth runway surface, appear within the visible spectrum. Because the visible light camera 120 operates only within the visible light spectrum, the system 100 may easily misinterpret these reflections as foreign objects 20 or “hyper-image FODs.” This would cause the system 100 to generate invalid or false alarms. Therefore, by comparing and detecting foreign objects 20 using both the thermal image 110M and the visible light image 120M, the system 100 is able to provide more accurate detection of foreign objects 20 and prevent or minimize false detections of foreign objects 20.

[0069] refer to Figure 5 Method 2000 and Figure 6 Method 3000. When comparing at least one attribute of a visible light object image 120B and a thermal object image 110B, system 100 can determine the detection of an unconfirmed foreign object 20, i.e., at least one attribute in the visible light object image 120B does not match at least one attribute in the thermal image 110B. System 100 can then generate an "Unconfirmed FOD" alarm signal. In this case, when system 100 "suspects" the detection of a foreign object 20 but identifies it as unconfirmed, system 100 can be configured to identify the event as a invalid alarm or a false alarm. Therefore, system 100 can be configured to store at least one of the alarm signal, attribute, feature, and image of the event in a database (e.g., a invalid alarm database) for post-event analysis and investigation.

[0070] Figure 8A flowchart of a method 5000 for identifying foreign object 20 on a runway is shown. An image processing module 134M can be configured to execute method 5000. System 100 can store multiple reference feature vectors and the object category associated with each of the multiple reference feature vectors in a reference feature vector database, which can be stored in storage device 140. To identify foreign object 20, system 100 is configured to identify the object category of foreign object 20. Reference Figure 8 The method may include capturing a visible light image 120M and a thermal image 110M via a visible light camera 120 and a thermal camera 110 in block 5302. To identify the object category of the foreign object 20 in the thermal image 110M, the method includes segmenting the thermal image 110M into multiple thermal image regions in block 5304. The method may also include segmenting the visible light image 120M into multiple visible light image regions in block 5304. Segmenting the thermal image 110M may include labeling each pixel in the thermal image 110M and grouping labeled pixels with the same characteristics into multiple groups to form multiple thermal image regions. Segmenting the visible light image 120M may include labeling each pixel in the visible light image 120M and grouping labeled pixels with the same characteristics into multiple groups to form multiple visible light image regions. The processor 132 may be configured to assign a label to each pixel in the thermal image 110M and the visible light image 120M, such that pixels with the same label share certain common characteristics or attributes. During segmentation, the thermal image 110M and the visible light image 120M consist of multiple thermal image regions and multiple visible light image regions that jointly cover the respective images. Pixels within each of the multiple regions are similar in some characteristics, features, or attributes (such as texture, color, or intensity). Adjacent regions of the multiple image regions differ significantly from each other regarding the same characteristics. The segmented thermal image 110M and visible light image 120M can be used to detect and locate one or more regions in the image that may contain the suspicious foreign object 20. In block 5306, the method may include detecting and extracting features from the thermal image 110M and the visible light image 120M. System 100 can be configured to assign feature vectors to each of the multiple thermal image regions (e.g., thermal feature vectors) and each of the multiple visible light image regions (e.g., visible light feature vectors). Features may refer to patterns or different structures found in the image, such as points, spots, patches, corners, or edges. Features are represented by image regions that differ from image regions in their immediate surrounding environment, for example, by texture, color, or intensity. Features can be extracted, grouped, and represented by feature vectors. Foreign object 20 can be represented by a set of features, which can be represented by feature vectors.

[0071] In box 5308, the method may include comparing a feature vector with a plurality of reference feature vectors, such that each of the plurality of reference feature vectors is associated with an object category. In box 5308, system 100 may match the feature vector with the plurality of reference feature vectors. Each object category, such as a rubber tire, machine tool, aircraft part, vehicle part, etc., may be represented by a specific reference feature vector stored in a reference feature vector database. Each extracted feature vector may be matched with the plurality of reference feature vectors in the reference feature vector database. In box 5310, the method may include detecting a foreign object 20. If a match exists between the feature vector and one or more of the plurality of reference feature vectors, system 100 may determine that a foreign object 20 has been detected. System 100 may generate an "Inspected FOD" alarm signal. In box 2312, the method may include identifying the object category of the foreign object 20. System 100 may be configured to identify the reference feature vector that is closest to the feature vector and its object category. Based on one or more matches among a plurality of reference feature vectors, system 100 can identify or classify foreign object 20 based on the closest match between the feature vector and one or more of the plurality of reference feature vectors (e.g., the “shortest distance” between the feature vector and a particular reference feature vector). Furthermore, the “shortest distance” can be used to determine the match or probability of the foreign object 20 being accurately classified. There may potentially be more than one reference feature vector that can match the feature vector. Matching can be based on fuzzy matching. System 100 can be configured to identify and classify foreign object 20 based on object category. System 100 can identify the object category of foreign object 20 in a visible light image 120M. Based on the matched reference feature vectors, object categories labeled to the matched reference feature vectors can be retrieved, and foreign object 20 can be identified or classified. Upon identification of foreign object 20, system 100 can generate and send an alarm signal.

[0072] Figure 9 A flowchart of a method 6000 for identifying foreign objects 20 on a runway is shown. Method 6000 and... Figure 8 The method is the same as that in 5000, except that system 100 is configured to automatically detect and extract features from thermal image 110M and visible light image 120M, and to match feature vectors with multiple reference feature vectors and detect foreign object 20 in box 6306. Figure 8 and Figure 9 The same reference numerals in the figures indicate the same steps. System 100 can be configured to train image processing module 134M using a deep learning module and automatically execute the steps in box 6306.

[0073] The thermal camera 110 can detect the foreign object 20 by detecting a temperature difference (i.e., the infrared thermal radiation of the foreground (e.g., foreign object 20) relative to the background (e.g., the surface of runway 202)). Different categories or types of foreign objects 20 are made of different materials, such as metal, rubber, plastic, concrete, etc., and will have different energy absorption rates, reflectivities, and emissivities. Thus, different categories of foreign objects 20 will result in different temperatures, i.e., different levels of infrared thermal radiation relative to the background (i.e., the runway surface). The temperature difference between the foreign object 20 and the runway can be detected by the thermal camera 110.

[0074] Therefore, it is beneficial to "train" the thermal camera 110, or more precisely, the thermal camera operation module 134T, to distinguish different categories of foreign objects 20 by identifying the type of material the foreign object 20 is made of, such as rubber, metal, plastic, concrete, asphalt, etc. Since foreign objects 20 made of different types of materials will have different emissivity, resulting in different temperature levels and different temperature contrast levels relative to the background (i.e., the runway), a "well-trained" thermal camera 110 will be able to identify foreign objects 20 more accurately.

[0075] Training thermal camera 110. Under normal, clear weather conditions, thermal camera 110 can be placed in the initial "training" period, whereby thermal camera 110 can operate in "training" mode to enable thermal camera 110 to "learn" from the visible light image 120M of visible light camera 120. After initial "training," thermal camera 110 can be sufficiently "learned" to enable thermal camera 110 to provide reliable and accurate detection of foreign object 20 with a relatively high level of accuracy. With a high level of accuracy, it will then be possible to enable system 100 with a "standalone" thermal camera 110 instead of a set of visible light cameras 120 and thermal camera 110. In this way, system 100 will be applicable under adverse weather conditions and / or very low visibility conditions without the need for visible light camera 120.

[0076] Figure 10 A flowchart illustrating a method for training an image processing module 134M to improve the identification of foreign object 20 on a runway is shown. In block 7110, a visible light camera 120 can be configured to scan one of multiple sectors on the runway. The visible light camera 120 can be configured to scan sub-sectors of that sector partition by partition. The visible light camera 120 can capture multiple visible light images 120M of the sector. The multiple visible light images 120M can be processed by the image processing module 134M to detect the foreign object 20. In block 7210, a thermal camera 110 can be configured to scan the same sector on the runway scanned by the visible light camera 120. The thermal camera 110 can be configured to scan sub-sectors of that sector partition by partition. The thermal camera 110 can capture multiple thermal images 110M of the sector.

[0077] Multiple thermal images 110M can be processed by image processing module 134M to detect foreign object 20. Thermal camera 110 and visible light camera 120 can be configured to scan sectors simultaneously. In block 7120, system 100 can detect foreign object 20 and identify visible light object image 120B after processing visible light image 120M. In block 7220, system 100 can detect foreign object 20 and identify thermal object image 110B after processing thermal image 110M. Thermal image 110M and visible light image 120M can be processed simultaneously by processor 132. If system 100 detects foreign object 20 in visible light image 120M, system 100 can generate a "suspected FOD" alarm signal to notify the operator that foreign object 20 has been detected in visible light image 120M. Similarly, if system 100 detects a foreign object 20 in thermal image 110M, system 100 can generate a "suspected FOD" signal in box 7230 to notify the operator that a foreign object 20 has been detected in thermal image 110M, since the detection of foreign object 20 has not yet been verified. A "suspected FOD" alarm signal can be generated for each of visible light image 120M and thermal image 110M. System 100 can display thermal object image 110B and / or visible light object image 120B on a display for operator viewing. System 100 can generate at least one attribute of visible light object image 120B in box 7130 and at least one attribute of thermal object image 110B in box 7230.

[0078] At least one attribute may include the position of the visible light object image 120B in the visible light image 120M, the position of the thermal object image 110B in the thermal image 110M, the size of the visible light object image 120B, the size of the thermal object image 110B, and / or the temperature of the thermal object image 110B. For example, system 100 may generate the position of the visible light object image 120B in the visible light image 120M and / or the size of the visible light object image 120B. For example, system 100 may generate at least one of the position of the thermal object image 110B in the thermal image 110M, the size of the thermal object image 110B, and the temperature of the foreign object 20. In blocks 7132 and 7232, system 100 may be configured to store at least one of the alarm signal, the attributes, characteristics, and image of the event in the foreign object alarm signal and event database 742. In block 7140, system 100 can be configured to determine the presence of foreign object 20 in visible light image 120B and thermal image 110B by comparing at least one attribute of visible light object image 120B and thermal image 110B. The method for comparing at least one attribute can be... Figure 7Method 4140 is shown in the diagram. If the attributes of the visible light object image 120B and the thermal object image 110B match in block 7150, then system 100 determines that a foreign object 20 has been detected in both the visible light image 120B and the thermal image 110B. If the foreign object 20 has been detected, system 100 may generate and display an alarm signal, such as a "Confirmed FOD" alarm signal, in block 7170. Otherwise, system 100 may generate and send an "Unconfirmed FOD" alarm signal. In block 7172, system 100 may be configured to store at least one of the signals, attributes, features, and images in a foreign object alarm signal and event database 742. In block 7180, system 100 may be configured to optimize the detection configuration parameters of the thermal camera 110. System 100 may optimize the detection configuration parameters based on data stored in the foreign object alarm signal and event database 742. System 100 may run statistical analysis and / or optimization modules to optimize the detection configuration parameters based on the data. System 100 can use artificial intelligence to optimize detection configuration parameters based on data. In block 7182, system 100 can be configured to store the optimized detection configuration parameters of thermal camera 110 in a database 744 for foreign object 20 detection configuration parameters of thermal camera 110.

[0079] System 100 can determine the relationship between the temperature and object category of foreign object 20 in thermal image 110M for all detected / verified foreign object samples. Processor 132 can also be configured to train thermal camera 110 to detect foreign objects 20 based on visible light image 120M from visible light camera 120. Since foreign objects 20 in visible light image 120M are essentially easier to identify and classify, system 100 can establish a relationship between the object category in visible light object image 120B obtained from visible light camera 120 and the temperature in thermal object image 110B from thermal camera 110. Therefore, with a sufficiently large foreign object sample size, system 100 will be able to determine the relationship between different foreign object categories, such as foreign objects made of different materials (e.g., metal, plastic, rubber, etc.) and their corresponding temperatures. System 100 can then be able to construct a "FOD-type thermal distribution model," which can be used to map various foreign object types (i.e., made of different materials) to their corresponding temperature ranges. In this way, the system can more easily identify foreign objects 20 based on its thermal object image 110B.

[0080] The "FOD-type thermal distribution model" will enable system 100 to determine the category or type of foreign object, including the specific type of material from which the foreign object 20 is made, such as metal, rubber, plastic, etc. Any detected foreign object 20 is analyzed based on its temperature detected by thermal camera 110. The development of the "FOD-type thermal distribution model" can be based on mathematical and / or statistical methods, such as statistical correlation analysis. Alternatively, the development of the "FOD-type thermal distribution model" can be based on artificial intelligence and machine learning techniques. This "FOD-type thermal distribution model" can be used to optimize the detection configuration parameters of thermal camera 110.

[0081] To optimize the performance of thermal camera 110, its detection configuration parameters need to be optimized. These parameters can be a set of operating parameters related to thermal camera 110 to enable it to detect foreign object 20 with optimal and high accuracy. Operating parameters may include sensitivity, gain, brightness, contrast, shutter timing settings, etc. In this way, as system 100 trains the thermal camera operation module 134T, the detection performance of thermal camera 110 will be improved over time to a level where it can operate as a “standalone” and sole foreign object detector for system 100, i.e., without visible light camera 120. Optimizing the performance of thermal camera 110 will be beneficial under adverse weather conditions and / or in very low visibility conditions.

[0082] The thermal camera 110 can detect various temperature contrast levels caused by different types of foreign materials. This allows the thermal camera 110 to accurately detect the foreign object 20. It also enables the system 100 to classify or identify different categories or types of foreign objects 20 based on the different types of materials that make up the foreign object 20.

[0083] Database 742 may contain alarm signals such as “suspected FOD”, “confirmed FOD”, and events occurring in the methods used with both the visible light camera 120 and the thermal camera 110. Database 742 may store detected and / or calculated foreign object 20 attributes such as category, size, location, temperature, etc.

[0084] Figure 11 A flowchart of a method 8000 for detecting a foreign object 20 using a thermal camera 110 is shown. After "training," the thermal camera 110 can be able to provide reliable and accurate detection of the foreign object 20 with a relatively high level of accuracy. At this high level of accuracy, it will be possible to use the thermal camera 110 without a visible light camera 120. (Reference) Figure 11In block 8402, system 100 can be configured to detect foreign object 20 by utilizing optimized detection configuration parameters of thermal camera 110 stored in detection configuration parameter database 844 of thermal camera 110. Thermal camera 110 can be configured as the sole foreign object detector to detect foreign object 20.

[0085] Technicians will understand that the features described in an example are not limited to that example and can be combined with any of the other examples.

[0086] The present invention relates to system 100, detection system 100 for detecting foreign objects on a runway, and method of system 100, as described herein with reference to the accompanying drawings and / or shown in the drawings.

Claims

1. A method for identifying foreign objects on a runway based on thermal object images, characterized in that, The method includes: Capture thermal images of regions of interest on the runway; Capture visible light images of the region of interest on the runway; Detecting thermal object images in the thermal image; Detecting visible light object images in the visible light image; When the thermal image and the visible light object image are detected respectively in the thermal image and the visible light object image, it is determined that a foreign object has been detected; Identify and categorize the visible light object image into the object category of foreign matter; Determine the relationship between the object category of the foreign object in the visible light image and the thermal object image of the foreign object; When a thermal object image of a foreign object is detected, the foreign object is identified by mapping the object category to the thermal object image based on the aforementioned relationship.

2. The method according to claim 1, wherein, Determining that the foreign object was detected includes: In each of the thermal object image and the visible light object image, at least one attribute of the foreign object is generated; the at least one attribute of the foreign object in the thermal object image and the visible light object image is compared; wherein the foreign object is detected when the at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

3. The method according to claim 2, wherein, At least one attribute of the foreign object includes the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image.

4. The method according to claim 3, wherein, When the distance between the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image is within the position parameter, the at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

5. The method according to any one of claims 2-4, wherein, The at least one property of the foreign object includes the size of the thermal object image and the visible light object image.

6. The method according to claim 5, wherein, When the difference between the size of the thermal object image in the thermal image and the size of the visible light object image in the visible light image is within a size parameter, the foreign object in the thermal object image and the visible light object image have the same at least one attribute.

7. The method according to claim 6, wherein, It also includes obtaining magnified thermal object images and magnified visible light object images when the foreign object is detected.

8. The method according to claim 7, wherein, It also includes identifying the object category of the foreign object in the thermal image, wherein identifying the object category includes: The thermal image is divided into multiple thermal image regions; Assign feature vectors to each of the plurality of thermal image regions; The feature vector is compared with a plurality of reference feature vectors, each of which represents an object category; Identify the reference feature vector that is closest to the feature vector and its object category.

9. The method according to claim 8, wherein, Segmenting the thermal image includes labeling each pixel in the thermal image and grouping labeled pixels with the same characteristics into multiple groups to form the multiple thermal image regions.

10. The method according to claim 9, wherein, Identifying the object category includes: The visible light image is divided into multiple visible light image regions; The feature vector is assigned to each of the plurality of visible light image regions; The feature vector is compared with a plurality of reference feature vectors, wherein each of the plurality of reference feature vectors represents an object category; And to identify the reference feature vector that is closest to the feature vector and its object category.

11. The method according to claim 10, wherein, Segmenting the visible light image includes labeling each pixel in the visible light image and grouping labeled pixels with the same characteristics into multiple groups to form the multiple visible light image regions.

12. The method according to claim 11, wherein, It also includes training a thermal camera to detect the foreign object based on the visible light image from a visible light camera.

13. The method according to claim 12, wherein, It also includes generating a thermal distribution model of the foreign object based on the relationship, and optimizing the detection configuration parameters of the thermal camera based on the thermal distribution model.

14. A system for detecting foreign objects on a runway based on thermal object image recognition, characterized in that, The system includes: A thermal camera, the thermal camera including a first field of view and adapted to capture thermal images of a region of interest on the runway; A visible light camera, the visible light camera including a second field of view and adapted to capture a visible light image of the region of interest on the runway, wherein the first field of view overlaps with the second field of view; A processor that communicates with the thermal camera and the visible light camera; A memory, which communicates with the processor to store instructions executable by the processor; wherein the processor is configured to: Detecting thermal object images in the thermal image; Detecting visible light object images in the visible light image; When the thermal image and the visible light object image are detected respectively in the thermal image and the visible light object image, it is determined that the foreign object has been detected; Identify and categorize visible light object images into foreign object categories; Determine the relationship between the object category of a foreign object in a visible light image and the thermal object image of the foreign object; When a thermal object image of a foreign object is detected, the foreign object is identified by mapping the object category to the thermal object image based on the aforementioned relationship.

15. The detection system according to claim 14, wherein, To identify the foreign object, the processor is configured to generate at least one attribute of the foreign object in each of the thermal object image and the visible light object image, compare the at least one attribute of the foreign object in the thermal object image and the visible light object image, wherein the foreign object is detected when the at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

16. The detection system according to claim 15, wherein, The at least one attribute of the foreign object includes the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image.

17. The detection system according to claim 16, wherein, When the distance between the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image is within the position parameter, the at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

18. The detection system according to any one of claims 15-17, wherein, At least one property of the foreign object includes the size of the thermal object image and the visible light object image.

19. The detection system according to claim 18, wherein, When the difference between the size of the thermal object image in the thermal image and the size of the visible light object image in the visible light image is within a size parameter, the foreign object in the thermal object image and the visible light object image have the same at least one attribute.

20. The detection system according to claim 19, wherein, The processor is also configured to, upon detecting the foreign object, magnify the thermal camera and the visible light camera to obtain magnified images of the thermal object and the visible light object.

21. The detection system according to claim 20, wherein, The processor is configured to identify the object category of the foreign object in the thermal image, wherein the processor is configured to: The thermal image is divided into multiple thermal image regions; Assign feature vectors to each of the plurality of thermal image regions; The feature vector is compared with a plurality of reference feature vectors, each of which represents an object category; Identify the reference feature vector that is closest to the feature vector and its object category.

22. The detection system according to claim 21, wherein, In order to segment the thermal image, the processor is configured to label each pixel in the thermal image and group labeled pixels with the same characteristics into multiple groups to form the multiple thermal image regions.

23. The detection system according to claim 22, wherein, In order to identify the object category of the foreign object in the visible light image, the processor is configured to: The visible light image is divided into multiple visible light image regions; The feature vector is assigned to each of the plurality of visible light image regions; The feature vector is compared with a plurality of reference feature vectors, each of which represents an object category; Identify the reference feature vector that is closest to the feature vector and its object category.

24. The detection system according to claim 23, wherein, In order to segment the visible light image, the processor is configured to label each pixel in the visible light image and group labeled pixels with the same characteristics into multiple groups to form the multiple visible light image regions.

25. The detection system according to claim 24, wherein, The processor is also configured to train the thermal camera to detect the foreign object based on the visible light image from the visible light camera.

26. The detection system according to claim 22, wherein, The processor is also configured to generate a thermal distribution model of the foreign object and to optimize the detection configuration parameters of the thermal camera based on the thermal distribution model.

27. A foreign object detection system for a runway divided into multiple sectors, characterized in that, The detection system includes multiple sets of cameras spaced apart from each other, each set of cameras comprising: A thermal camera, which includes a first field of view and is adapted to capture thermal images of the region of interest on the runway; A visible light camera, which includes a second field of view and is adapted to capture visible light images of a region of interest on a runway, wherein the first field of view overlaps with the second field of view; A processor that communicates with multiple sets of cameras; Memory, which communicates with the processor to store instructions that can be executed by the processor; The processor is configured as follows: Detecting thermal objects in thermal images; Detect visible light object images in visible light images; and determine that a foreign object has been detected when thermal object images and visible light object images are detected in thermal images and visible light object images, respectively; Identify and categorize visible light object images into foreign object categories; Determine the relationship between the object category of a foreign object in a visible light image and the thermal object image of the foreign object; When a thermal object image of a foreign object is detected, the foreign object is identified by mapping the object category to the thermal object image based on the aforementioned relationship. Each of the multiple camera groups is configured to scan one of the multiple sectors of the runway.

Images