System, detection system for detecting foreign objects on runways, and method for the system

Abstract

This concerns a system for detecting foreign objects on runways. [Solution] This system comprises a thermal camera having a first field of view and configured to capture a thermal image of a target area on the runway, and a visible light camera having a second field of view and configured to capture a visible light image of a target area on the runway, wherein the first field of view overlaps with the second field of view. The system further comprises a processor and a memory that communicates with the processor and stores instructions that can be executed by the processor. The processor is configured to detect thermal object images in the thermal image and visible light object images in the visible light image, and to determine that a foreign object has been detected when a thermal object image and a visible light object image are detected in the thermal image and visible light object image, respectively.

Description

Technical Field

[0001] The present invention relates to a system, a detection system for detecting foreign objects on a runway, and a method of the system.

Background Art

[0002] Foreign objects and debris (FOD) on airport runways pose risks to aircraft landing and takeoff on the runway. To reliably detect FOD under normal clear weather conditions, a FOD detection system using a visible light spectrum camera is used. Under normal clear weather conditions, such as when there is no fog, the FOD detection system can detect FOD with high accuracy by taking and processing high-resolution images of FOD. FOD includes, for example, engine and aircraft parts, tools, construction waste, rubber materials, natural substances, and the like.

[0003] However, in bad weather, especially in foggy weather, the operation of the FOD detection system may be adversely affected and degraded. Since the FOD detection system operates only in the visible light spectrum, there is a risk that FOD cannot be reliably detected under foggy weather conditions, that is, under poor visibility conditions. Under such conditions caused by fog or the like, the visibility on the runway usually drops below 1 km, making it impossible to "see" FOD. Visibility conditions are classified into multiple categories. For example, Cat II represents standard operations corresponding to a runway visual range (RVR) in the range of 550 meters (1,800 feet) to 300 meters (1,000 feet). Cat IIIa represents precision instrument approach and landing operations where the RVR is 175 meters (600 feet) or more. Cat IIIb represents precision instrument approach and landing operations where the RVR is less than 175 meters (600 feet) and 50 meters (200 feet) or more. Cat IIIc represents precision instrument approach and landing operations without RVR restrictions, including cases where the visibility is zero. The visibility of airport runways varies depending on the geographical location of the airport and is appropriately classified. Many FOD detection systems can detect FOD within the airport at Cat II visibility, but cannot be used at airports with Cat IIIa, Cat IIIb, and Cat IIIc visibility.

[0004] Furthermore, FOD detection systems often generate false alarms or incorrect warnings. False warnings can be caused by phenomena such as light reflection from artificial light sources, including lights at nearby buildings or the runway edge. When such artificial light reflects off the smooth surface of the runway or off puddles or standing water on the runway surface, the FOD detection system may identify it as an FOD and issue a false warning. The number of such false warnings due to reflection often increases significantly after rainfall, when puddles or standing water form on the runway surface. While such reflections occur during the day, they are even more frequent at night, and during the early morning and evening hours.

[0005] Therefore, it is important to provide a solution that can detect FOD even under poor visibility conditions such as bad weather, and that can prevent or minimize false FOD detections. [Overview of the Initiative]

[0006] Various embodiments provide a method for detecting foreign objects on a runway. This method comprises capturing a thermal image of a target area on the runway, capturing a visible light image of the target area on the runway, detecting a thermal object image within the thermal image, detecting a visible light object image within the visible light image, and determining that a foreign object has been detected when a thermal object image and a visible light object image are detected in the thermal image and visible light object image, respectively.

[0007] In various embodiments, determining a foreign object may involve generating at least one attribute of the foreign object in both the thermal object image and the visible light object image, comparing at least one attribute of the foreign object in the thermal object image and the visible light object image, and detecting the foreign object if at least one attribute of the foreign object in the thermal object image and the visible light object image are the same.

[0008] In various embodiments, at least one attribute of the foreign object may include the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image.

[0009] In various embodiments, if the distance between the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image is within the position parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image may be the same.

[0010] In various embodiments, at least one attribute of the foreign object may include the size of the thermal object image and the visible light object image. In various embodiments, if the difference between the size of the thermal object image in the thermal image and the size of the visible light object image in the visible light image is within the size parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image may be the same.

[0011] In various embodiments, the method may further include obtaining an enlarged thermal image of the object and an enlarged visible light image of the object when a foreign object is detected. In various embodiments, the method further includes identifying the object category of foreign objects in a thermal image, the identification of an object category may further include segmenting the thermal image into a plurality of thermal image regions, assigning a feature vector to each of the plurality of thermal image regions, comparing the feature vectors with a plurality of reference feature vectors, each of the plurality of reference feature vectors representing an object category, and identifying the reference feature vector closest to the feature vector and its object category.

[0012] In various embodiments, segmenting a thermal image may include labeling each pixel in the thermal image and grouping the labeled pixels having the same characteristics into multiple groups to form multiple thermal image regions.

[0013] In various embodiments, the method further includes identifying the object category of a foreign object in a visible light image, wherein identifying the object category may include segmenting the visible light image into a plurality of visible light image regions, assigning a feature vector to each of the plurality of visible light image regions, comparing the feature vector with a plurality of reference feature vectors, each of which represents an object category, and identifying the reference feature vector closest to the feature vector and its object category.

[0014] In various embodiments, segmenting a visible light image may include labeling each pixel in the visible light image and grouping the labeled pixels having the same characteristics into multiple groups to form multiple visible light image regions.

[0015] In various embodiments, the method may further include training a thermal camera to detect foreign objects based on visible light images from a visible light camera. In various embodiments, training a thermal camera may include determining the relationship between the object category of a foreign object in a visible light image and the temperature of the foreign object in a thermal image.

[0016] In various embodiments, a system for detecting foreign objects on a runway is provided. The system comprises a thermal camera having a first field of view and configured to capture a thermal image of a target area on the runway, and a visible light camera having a second field of view and configured to capture a visible light image of a target area on the runway, wherein the first field of view overlaps with the second field of view. The system further comprises a processor that communicates with the thermal camera and the visible light camera, and a memory that communicates with the processor and stores instructions that can be executed by the processor. The processor is configured to detect thermal object images in the thermal image and visible light object images in the visible light image, and to determine that a foreign object has been detected when a thermal object image and a visible light object image are detected in the thermal image and visible light object image, respectively.

[0017] In various embodiments, in order to determine a foreign object, the processor may be configured to generate at least one attribute of the foreign object in both the thermal object image and the visible light object image, compare the at least one attribute of the foreign object in the thermal object image and the visible light object image, and detect the foreign object if the at least one attribute of the foreign object in the thermal object image and the visible light object image are the same.

[0018] In various embodiments, at least one attribute of the foreign object may include the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image.

[0019] In various embodiments, if the distance between the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image is within the position parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image may be the same.

[0020] In various embodiments, at least one attribute of the foreign object may include the size of the thermal object image and the visible light object image. In various embodiments, if the difference between the size of the thermal object image in the thermal image and the size of the visible light object image in the visible light image is within the size parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image may be the same.

[0021] In various embodiments, the processor may be further configured to zoom in on the thermal camera and the visible light camera to obtain magnified thermal and visible light images of the object when a foreign object is detected.

[0022] In various embodiments, the processor may be configured to identify the object category of foreign objects in a thermal image, and the processor may be configured to segment the thermal image into a plurality of thermal image regions, assign a feature vector to each of the plurality of thermal image regions, and compare the feature vector with a plurality of reference feature vectors, wherein each of the plurality of reference feature vectors represents an object category, and to identify the reference feature vector and its object category that is closest to the feature vector.

[0023] In various embodiments, in order to segment the thermal image, the processor may be configured to label each pixel in the thermal image, group the labeled pixels having the same characteristics into multiple groups, and form multiple thermal image regions.

[0024] In various embodiments, the processor may be further configured to identify the object category of foreign objects in a thermal image, and the processor may be configured to segment a visible light image into a plurality of visible light image regions, assign a feature vector to each of the plurality of visible light image regions, compare the feature vector with a plurality of reference feature vectors, each of which represents an object category, and identify the reference feature vector and its object category that is closest to the feature vector.

[0025] In various embodiments, to segment a visible light image, the processor may be configured to label each pixel in the visible light image and group the labeled pixels having the same characteristics into a plurality of groups to form a plurality of visible light image regions.

[0026] In various embodiments, the processor may be further configured to train a thermal camera to detect foreign objects based on a visible light image from a visible light camera.

[0027] In various embodiments, to train a thermal camera, the processor may be configured to determine a relationship between an object category of a foreign object in a visible light image and a temperature of the foreign object in a thermal image.

[0028] In various embodiments, a detection system is provided for detecting foreign objects on a runway divided into a plurality of sectors. The detection system includes a plurality of camera sets arranged at intervals from each other, each having a thermal camera configured to have a first field of view and capture a thermal image of a target area on the runway, and a visible light camera configured to have a second field of view and capture a visible light image of the target area on the runway, the visible light camera having the first field of view overlapping the second field of view, a processor communicating with the thermal camera and the visible light camera, and a memory communicating with the processor and storing instructions executable by the processor. The processor may be configured to detect a thermal object image in the thermal image, detect a visible light object image in the visible light image, and determine that a foreign object is detected when the thermal object image and the visible light object image are respectively detected in the thermal image and the visible light object image. The plurality of camera sets may each be configured to scan one of the plurality of sectors of the runway.

Brief Description of the Drawings

[0029] [Figure 1] A schematic diagram illustrating an exemplary embodiment of a system for detecting foreign objects on a runway. [Figure 1A] A schematic diagram showing a visible light image containing a visible light object image of a foreign object, and a thermal image containing a thermal object image of the foreign object. [Figure 2A] A diagram illustrating an exemplary embodiment of the system. [Figure 2B] Figure 2A shows the system scanning one of several sectors of the runway. [Figure 3] A diagram illustrating an exemplary embodiment of a detection system for detecting foreign objects on a runway divided into multiple sectors. [Figure 4] A flowchart illustrating an exemplary method for detecting foreign objects on a runway. [Figure 5] A flowchart illustrating an exemplary method for detecting foreign objects on a runway. [Figure 6] A flowchart illustrating an exemplary method for detecting foreign objects on a runway. [Figure 7] A flowchart illustrating an exemplary method for comparing at least one attribute of a foreign object in thermal and visible light object images. [Figure 8] A flowchart illustrating a method for identifying foreign objects on a runway. [Figure 9] A flowchart illustrating a method for identifying foreign objects on a runway. [Figure 10] A flowchart illustrating how to train an image recognition module to improve the identification of foreign objects on runways. [Figure 11] A flowchart illustrating a method for detecting foreign objects using a thermal camera. [Modes for carrying out the invention]

[0030] In the drawings referenced in the following examples, identical features are denoted by the same reference numerals. Figure 1 is a schematic diagram showing an exemplary embodiment of a system 100 for detecting foreign objects 20 on a runway. The system 100 includes a thermal camera 110 having a first field of view 110F and configured to capture a thermal image 110M of a target area 112 on the runway; a visible light camera 120 having a second field of view 120F and configured to capture a visible light image of the target area 112 on the runway, with the first field of view 110F overlapping with the second field of view 120F; a processor 132 that communicates with the thermal camera 110M and the visible light camera 120M; and a memory 134 that communicates with the processor 132 and stores instructions that can be executed by the processor 132. The processor 132 is configured to detect thermal object images in the thermal image and visible light object images in the visible light image, and to determine that foreign objects 20 have been detected when thermal object images and visible light object images are detected in the thermal image and visible light object images, respectively. System 100 includes a server containing a processor 132, memory 134, and an I / O interface 136 configured to connect the processor 132 with peripheral interface modules (keyboard, mouse, touchscreen, display, etc.). System 100 also includes a communication module 138 configured to facilitate wired or wireless communication between System 100 and other user devices (mobile devices, laptops, etc.) over the internet. System 100 includes a storage device 140 configured to store data. System 100 also includes a display (monitor, touchscreen, etc.) for displaying signals such as warning signals to the operator. System 100 is configured to detect foreign objects and debris (FOD) on runways, taxiways, aprons, ramps, etc., under both day and night ambient light conditions without requiring auxiliary lighting such as visible spectrum illumination, infrared illumination, or laser illumination.

[0031] Figure 1A is a schematic diagram showing a visible light image 120M including a visible light object image 120B of the foreign object 20, and a thermal image 110M including a thermal object image 110B of the foreign object. System 100 includes an image processing module 134M (see Figure 1) configured to process images 110T and 120T captured by the thermal camera 110 and the visible light camera 120. System 100 also includes a thermal camera operation module 134T containing operating parameters for the thermal camera 110. System 100 also includes a visible light camera operation module 134V containing operating parameters for the visible light camera 120. Modules 134T, 134V, and 134M are stored in a storage device 140, loaded into memory 134, and processed by the processor 132.

[0032] When thermal image 110M and visible light image 120M are captured, images 110M and 120M are transmitted to processor 132 for processing. Processor 132 receives and processes thermal image 110M and visible light image 120M to detect foreign objects 20 on the runway. Since system 100 can detect foreign objects 20 at airports with Cat II visibility, Cat IIIa visibility, Cat IIIb visibility, and Cat IIIc visibility, it can detect foreign objects even in poor visibility conditions such as bad weather, and can prevent or minimize false detections of foreign objects.

[0033] Figure 2A shows an exemplary embodiment of system 200. System 200 comprises a camera set, namely a visible light camera 220 and a thermal camera 210. The camera set 210S is rigidly mounted on an actuator 250 configured to move the camera set 210S. The camera set 210S is controlled by a processor 132 to scan a sector of the runway to detect foreign objects 20 on the runway surface.

[0034] The actuator 250 may be a pan / tilt unit (PTU) configured to simultaneously pan and tilt the camera set 210S so that the field of view of the camera set 210S is the same and the same target area can be focused on. The actuator 250 is configured to pan the camera set 210S horizontally 210H and / or tilt the camera set 210S vertically 210V. The actuator 250 is able to communicate with the processor 132, which is configured to remotely control the movement of the actuator 250 to pan and tilt the camera set 210S to scan the runway. The actuator 250 is usually installed on top of a support structure 252, such as a mast structure, which is located along the runway. The support structure is installed at a distance of 120m to 350m from the runway centerline 304 (see Figure 3).

[0035] Figure 2B shows the system 100 of Figure 2A scanning one of several sectors 202S of runway 202. Each camera set 210S consists of a thermal camera 210 and a visible light camera 220, each having a field of view 210F and 220F, respectively, and is configured to capture a target area 212. The field of view 210F of the visible light camera 220 overlaps with the field of view 220F of the thermal camera 210. The fields of view 210F and 220F of both the visible light camera 220 and the thermal camera 210 cover a specific target area 212 within sector 202S of runway 202. Because the fields of view 210F and 220F overlap in this way, both the visible light camera 220 and the thermal camera 210 can simultaneously detect the same foreign object 20 on sector 202S of the runway while scanning the sector 202S.

[0036] Figure 3 shows an exemplary embodiment of a detection system 300 for detecting foreign objects 20 on a runway 302 divided into multiple sectors 302S. The detection system 300 has multiple camera sets 310S spaced apart from each other. Each camera set 310S has a thermal camera 210 having a first field of view 210F and configured to capture a thermal image 110M of a target area 212 on the runway 302, and a visible light camera 220 having a second field of view 220F and configured to capture a visible light image 120M of the target area 212 on the runway 302. The first field of view 210F overlaps with the second field of view 220F. The detection system 300 further includes a processor and a memory that communicates with the processor and stores instructions that can be executed by the processor. The processor is configured to detect a thermal object image 110B in the thermal image 110M and a visible light object image 120B in the visible light image 120M, and to determine that a foreign object 20 has been detected when the thermal object image 110B and the visible light object image 120B are detected in the thermal image 110M and the visible light object image 120B, respectively. Multiple camera sets 310S are configured to scan one of multiple sectors 302S of the runway 302. As shown in Figure 3, the runway 302 is divided into multiple sectors 302S. Each of the multiple camera sets 310S detects a foreign object 20 on the surface of each sector 302S by scanning one of these multiple sectors 302S. Each sector 302S may be further divided into multiple subsectors. Each camera set 310S is capable of scanning the corresponding sector 302S and scanning that sector 302S subsector by subsector. In this way, when a foreign object 20 is detected, the system 300 can identify the sector 302S based on the camera set 310S scanning the sector 302S. The camera set 310S can scan the sector 302S in a specific common scanning direction, for example, from the leftmost subsector to the rightmost subsector, or from the rightmost subsector to the leftmost subsector.

[0037] The thermal camera 210 detects foreign objects 20 on runway 302 by detecting the difference in thermal radiation levels (or temperature) between the foreground (i.e., foreign objects 20) and the background (i.e., the runway surface). The thermal camera 210 operates in the infrared spectrum and does not require ambient light to "see" the foreign objects 20. The thermal camera 210 is also commonly referred to as an infrared thermal camera. The thermal camera 210 is a medium-wave infrared (MWIR) camera or a long-wave infrared (LWIR) camera. The thermal camera 210 has the advantage of being able to detect foreign objects 20 on runway 302 even in very poor visibility conditions, and even in zero illumination, i.e., complete darkness. For this reason, the thermal camera 210 has the advantage of being able to detect foreign objects 20 on runway 302 even in foggy weather conditions. The thermal camera 210 captures monochrome images and video output and transmits them to the processor 132. The thermal camera 210 is a completely passive camera that does not actively transmit or emit radio frequencies, microwaves, artificial lighting, infrared, lasers, LIDAR, etc. Therefore, the thermal camera 210 has advantages such as not interfering with existing airport systems / equipment and aircraft systems / equipment, not interfering with airport systems / equipment and aircraft systems / equipment in the future, and not requiring frequency / spectrum licenses and approvals from airport and frequency spectrum regulatory authorities.

[0038] Unlike the thermal camera 210, the visible light camera 220 operates within the visible spectrum, so in order to "see" foreign objects 20 on the runway 302, the ambient visible spectral light must be above a certain minimum amount. The visible light camera 220 cannot detect foreign objects 20 if visibility conditions are very poor or if the illumination is zero. For example, the visible light camera 220 cannot detect foreign objects 20 if visibility conditions (on the runway surface) are very poor or if fog is present (on the runway surface). The visible light camera 220 can capture and transmit full-color and high-resolution images / videos (e.g., HD (FHD) or 4K Ultra HD (4K UHD) resolution). High-resolution color images not only allow the system 300 to reliably and accurately recognize and classify the detected foreign objects 20, but also allow the operator to reliably and accurately perform visual verification and confirmation of the detected foreign objects 20. Therefore, by using both the visible light camera 220 and the thermal camera 210 in combination, the system 300 can function even under very poor visibility conditions, such as foggy weather conditions, and can accurately and reliably detect foreign objects 20 on the surface of runway 302. The visible light camera 220 is configured to capture a color, high-resolution visible light image 120M and output it to the processor 132. The visible light camera 220 can operate without the need for infrared illumination, visible spectrum illumination, or laser illumination. Because the system 300 is passive, it has the advantage of not posing a danger to or interfering with other airport systems or aircraft systems during aircraft landings / takeoffs on runway 302. The system 300 has the advantages of not interfering with existing airport systems / equipment and aircraft systems / equipment, not interfering with airport systems / equipment and aircraft systems / equipment in the future, and not requiring frequency / spectrum licenses and approvals from airport and frequency spectrum regulatory authorities.

[0039] Figure 4 is a flowchart illustrating an exemplary method for detecting foreign objects on a runway. This method includes taking a thermal image 110M of a target area 112 on the runway in block 1010, taking a visible light image 120M of the target area 112 on the runway in block 1020, detecting a thermal object image 110B within the thermal image 110M in block 1030, detecting a visible light object image 120B within the visible light image 120M in block 1040, and determining in block 1050 that a foreign object 20 has been detected if a thermal object image 110B and a visible light object image 120B are detected within the thermal image 110M and visible light object image 120B, respectively. The thermal object image 110B is a part of the thermal image 110M that represents the foreign object 20 within the thermal image 110M, and may also be simply referred to as the foreign object 20 in the thermal image 110M. The visible light object image 120B is a part of the visible light image 120M that represents the foreign object 20 within the visible light image 120M, and may also be simply referred to as the foreign object 20 within the visible light image 120M.

[0040] This method may include scanning the runway with the thermal camera 110 and the visible light camera 120 before capturing the thermal image 110M and the visible light image 120M. As the thermal camera 110 and the visible light camera 120 scan the runway sectors, they capture thermal images 110M and visible light images 120M of multiple target areas 112 along the sectors. To detect foreign objects 20, the image processing module 134M processes the thermal image 110M and the visible light image 120M to determine whether or not foreign objects 20 are present in the thermal image 110M and the visible light image 120M. When foreign objects 20 are detected, the image processing module 134M is configured to identify the thermal object image 110B and the visible light object image 120B, respectively, within the thermal image 110M and the visible light image 120M. When the foreign object 20 is identified, the system 100 generates a warning signal.

[0041] To detect the foreign object 20, the above method may include generating at least one attribute of the foreign object 20 in each of the thermal object image 110B and the visible light object image 120B, and comparing at least one attribute of the foreign object 20 in the thermal object image 110B and the visible light object image 120B. The foreign object 20 is detected if at least one attribute of the foreign object 20 in the thermal object image 110B and the visible light object image 120B is the same or falls within a specific parameter or threshold level. The system 100 may be configured to obtain an enlarged thermal object image 110B and an enlarged visible light object image 120B by zooming in on the detected foreign object 20 with the visible light camera 120 and the thermal camera 110 when the foreign object 20 is detected.

[0042] Figure 5 shows a flowchart of an exemplary method for detecting foreign objects 20 on a runway. System 100 designates a visible light camera 120 as the primary detector and a thermal camera 110 as the secondary detector. As shown in Figure 5, in block 2110, the visible light camera 120 is configured to scan one of several sectors on the runway. The visible light camera 120 is configured to scan the sector sub-sector by subsector. The visible light camera 120 captures multiple visible light images 120M within each sector. The image processing module 134M processes the multiple visible light images 120M to detect the foreign objects 20. In block 2210, the thermal camera 110 is configured to scan the same sector on the runway scanned by the visible light camera 120. The thermal camera 110 is configured to scan the sector sub-sector by subsector. The thermal camera 110 captures multiple thermal images 110M within the same sector. The image processing module 134M processes multiple thermal images 110M to detect the foreign object 20. The thermal camera 110 and the visible light camera 120 are configured to scan sectors simultaneously. The image processing modules 134M for the visible image and the thermal image 110M may be separate modules for processing the visible light image 120M and the thermal image 110M, respectively.

[0043] In block 2120, system 100 detects the foreign object 20 after processing the visible light image 120M. System 100 identifies the visible light object image 120B within the visible image. In block 2220, system 100 detects the foreign object 20 after processing the thermal image 110M. System 100 identifies the thermal object image 110B within the thermal image 110M. The thermal image 110M and the visible light image 120M can be processed simultaneously by processor 132. If system 100 detects the foreign object 20 within the visible light image 120M, system 100 generates a "suspected FOD" warning signal in block 2130 to notify the operator that the foreign object 20 has been detected within the visible light image 120M. Similarly, if system 100 detects a foreign object 20 in the thermal image 110M, system 100 generates a “suspected FOD” warning signal in block 2230 to notify the operator that a foreign object 20 has been detected in the thermal image 110M, since the detection of the foreign object 20 has not yet been verified. The “suspected FOD” signal is generated for both the visible light image 120M and the thermal image 110M. System 100 may display the thermal object image 110B and / or the visible light object image 120B on a display for the operator to visually inspect. System 100 generates at least one attribute for the visible light object image 120B and the thermal object image 110B. At least one attribute includes the location of the visible light object image 120B in the visible light image 120M, the location of the thermal object image 110B in the thermal image 110M, the size of the visible light object image 120B, and / or the size of the thermal object image 110B. For example, system 100 generates the position of the visible light object image 120B in the visible light image 120M, the position of the thermal object image 110B in the thermal image 110M, and / or the sizes of the visible light object image 120B and the thermal object image 110B. In block 2140, system 100 is configured to determine whether or not a foreign object 20 has been detected in the visible light image 120M and the thermal image 110M by comparing at least one attribute of the visible light object image 120B and the thermal object image 110B. Details of this comparison step are shown in Figure 7.If the attributes of the visible light object image 120B and the thermal object image 110B match in block 2150, the system 100 determines that a foreign object 20 has been detected in the visible light image 120M and the thermal image 110M. The system 100 may receive confirmation input from the operator via a peripheral interface module to confirm the detection of the foreign object 20 after visually inspecting the visible light image 120M and / or the thermal image 110M on the display. In block 2160, the system 100 identifies the foreign object 20 based on at least one attribute. Once the foreign object 20 is detected and / or identified, the system 100 generates a warning signal in block 2170, for example, a "Confirmed FOD" signal. Otherwise, the system 100 generates a "No Confirmed FOD" warning signal. The system 100 may transmit the warning signal to the operator's mobile device or display the warning signal on the display for the operator's confirmation. The system 100 may generate a warning signal when it receives input indicating that the operator has confirmed the detection of a foreign object 20.

[0044] Figure 6 shows a flowchart of an exemplary method 3000 for detecting foreign objects 20 on the runway. Method 3000 is the same as method 2000 in Figure 5, except that the system 100 is configured to designate the thermal camera 110 as the primary detector and the visible light camera 120 as the secondary detector. In Figures 5 and 6, the same steps are denoted by the same reference numerals. As shown in Figure 6, in block 3110, the thermal camera 110 is configured to scan one of several sectors on the runway. The thermal camera 110 is configured to scan the sector sub-sector by sub-sector. The thermal camera 110 takes multiple thermal images 110M within each sector. The image processing module 134M processes the multiple thermal images 110M to detect the foreign objects 20. In block 3210, the visible light camera 120 is configured to scan the same sector on the runway scanned by the thermal camera 110. The visible light camera 120 is configured to scan the sector sub-sector by sub-sector. The visible light camera 120 captures multiple visible light images 120M within the sector. The image processing module 134M processes the multiple visible light images 120M to detect the foreign object 20. The thermal camera 110 and the visible light camera 120 are configured to scan the sector simultaneously. In block 3120, the system 100 detects the foreign object 20 after processing the thermal image 110M. The system 100 identifies the thermal object image 110B within the thermal image 110M. In block 3230, the system 100 detects the foreign object 20 after processing the visible light image 120M. The system 100 identifies the visible light object image 120B within the visible light image 120M. The thermal image 110M and the visible light image 120M can be processed simultaneously by the processor 132. Blocks 3140-3170 are the same as blocks 2140-2170 in Figure 5.

[0045] Figure 7 is a flowchart illustrating an exemplary method 4140 for comparing at least one attribute of a foreign object 20 in a thermal object image 110B and a visible light object image 120B. Method 4140 is used in blocks 2140 and 3140 of Method 2000 in Figure 5 and Method 3000 in Figure 6. At least one attribute of the foreign object 20 includes the position of the thermal object image 110B in the thermal image 110M and the position of the visible light object image 120B in the visible light image 120M. In block 4141, it is determined that at least one attribute of the foreign object 20 in the thermal object image 110B and the visible light object image 120B is the same if the distance between the position of the thermal object image 110B in the thermal image 110M and the position of the visible light object image 120B in the visible light image 120M is within the position parameter. For example, the processor 132 identifies the positional difference between the visible light object image 120B and the thermal object image 110B in the visible light image 120M and the thermal image 110M, and determines whether the positional difference is within a positional parameter, i.e., a predetermined positional threshold level. The positional parameter can be determined by statistical analysis of the detection positions of all detected foreign object samples 20. If the positional difference is within the range of the positional parameter, the processor 132 generates a "position match" warning signal in block 4142.

[0046] At least one attribute of the foreign object 20 includes the size in the thermal object image 110B and the visible light object image 120B. In block 4143, if the difference between the size of the thermal object image 110B in the thermal image 110M and the size of the visible light object image 120B in the visible light image 120M is within the size parameter, it is determined that at least one attribute of the foreign object 20 in the thermal object image 110B and the visible light object image 120B is the same. For example, the processor 132 identifies the size difference between the sizes of the visible light object image 120B and the thermal object image 110B in the visible light image 120M and the thermal image 110M, and determines whether the size difference is within the size parameter, i.e., a predetermined size threshold level. The size parameter can be determined based on a statistical analysis of the measured size of all detected foreign object 20 samples. If the size difference is within the range of the size parameter, the processor 132 generates a "size match" warning signal in block 4144.

[0047] Depending on the configuration of system 100, the process detects foreign objects 20 based on the position and / or size of the thermal object image 110B and the visible light object image 120B. For example, if both the position and size of the thermal object image 110B and the visible light object image 120B are used, foreign objects 20 are detected if the position and size of the thermal object image 110B and the visible light object image 120B are within the range of the position parameter and size parameter, respectively, i.e., they match. When foreign objects 20 are detected, system 100 generates a warning signal, for example, an "attribute match" signal when attributes match in block 4145. System 100 may also generate warning signals if the "position match" warning signal and the "size match" warning signal are turned on or generated.

[0048] The exemplary system 100 and method described above provide a solution that enables the detection of foreign objects 20 during adverse weather conditions and prevents or minimizes false detections of foreign objects 20. For example, reflections from puddles or standing water after rain, or reflections from the smooth surface of a runway, occur within the visible light spectrum. Because the visible light camera 120 operates only within the visible light spectrum, system 100 tends to mistakenly identify these reflections as foreign objects 20, i.e., "suspected FODs." As a result, system 100 may issue inappropriate warnings or false detection alarms. Therefore, by comparing and detecting foreign objects 20 using both the thermal image 110M and the visible light image 120M, system 100 can more accurately detect foreign objects 20 and prevent or minimize false detections of foreign objects 20.

[0049] Refer to Method 2000 in Figure 5 and Method 3000 in Figure 6. When comparing at least one attribute of the visible light object image 120B and the thermal object image 110B, the system 100 may determine that no foreign object 20 is detected, i.e., that at least one attribute of the visible light object image 120B does not match at least one attribute of the thermal image 110M. In this case, the system 100 generates a warning signal of “No Confirmed FOD”. In such a situation, the system 100 identifies that it “suspected” the detection of foreign object 20 but did not “confirm” it, and is therefore configured to identify this event as an improper warning or false positive alert. For this reason, the system 100 can be configured to save at least one of the warning signal, attribute, feature and image of this event to a database (e.g., a database of improper warnings) for post-event analysis and investigation.

[0050] Figure 8 is a flowchart showing a method 5000 for identifying foreign objects 20 on a runway. An image processing module 134M can be configured to perform method 5000. System 100 stores a plurality of reference feature vectors and the object categories associated with each of the plurality of reference feature vectors in a reference feature vector database. The reference feature vector database is stored in a storage device 140. To identify the foreign object 20, system 100 is configured to identify the object category of the foreign object 20. As shown in Figure 8, this method includes capturing a visible light image 120M and a thermal image 110M by a visible light camera 120 and a thermal camera 110 in block 5302. This method includes segmenting the thermal image 110M into a plurality of thermal image regions in block 5304 in order to identify the object category of the foreign object 20 in the thermal image 110M. This method includes segmenting the visible light image 120M into a plurality of visible light image regions in block 5304. Segmenting the thermal image 110M includes labeling each pixel in the thermal image 110M and grouping labeled pixels with the same characteristics into multiple groups to form multiple thermal image regions. Segmenting the visible light image 120M includes labeling each pixel in the visible light image 120M and grouping labeled pixels with the same characteristics into multiple groups to form multiple visible light image regions. The processor 132 assigns labels to each pixel in the thermal image 110M and visible light image 120M such that pixels with common specific characteristics or properties have the same label. The segmented thermal image 110M and visible light image 120M consist of multiple thermal image regions and multiple visible light image regions that collectively encompass each image. Pixels in each of the multiple regions are similar in some characteristic, feature, or property such as texture, color, or intensity. Adjacent regions in the multiple image regions differ significantly from each other in the same characteristics.The segmented thermal image 110M and visible light image 120M can be used to detect and identify regions containing suspicious foreign objects 20 within the images. In block 5306, this method detects and extracts features from the thermal image 110M and visible light image 120M. System 100 is configured to assign a feature vector (e.g., a thermal feature vector) to each of the multiple thermal image regions and a feature vector (e.g., a visible light feature vector) to each of the multiple visible light image regions. A feature refers to a pattern or unique structure present in the image, such as a point, clump, small patch, corner, or edge. Features are represented by image regions whose texture, color, intensity, etc., differ from the surrounding image regions. Features are extracted, grouped, and represented by feature vectors. The foreign object 20 can be represented by a group of features represented by feature vectors.

[0051] This method includes comparing a feature vector with multiple reference feature vectors in block 5308. Each of the multiple reference feature vectors is associated with an object category. System 100 matches the feature vector with the multiple reference feature vectors in block 5308. Each object category (e.g., rubber tire, mechanic's tools, aircraft part, vehicle part) can be represented by a specific reference feature vector stored in the reference feature vector database. Each extracted feature vector is matched with multiple reference feature vectors in the reference feature vector database. This method includes detecting foreign object 20 in block 5310. If the feature vector matches one or more of the multiple reference feature vectors, System 100 determines that foreign object 20 has been detected. System 100 generates a "suspected FOD" warning signal. This method identifies the object category of foreign object 20 in block 2312. System 100 is configured to identify the reference feature vector and its object category that is closest to the feature vector. System 100 identifies or classifies foreign objects 20 based on one or more of the matched reference feature vectors, specifically the "shortest distance" between the feature vector and a particular reference feature vector. The "shortest distance" may also be used to determine the match or the probability that the foreign object 20 is accurately classified. Multiple reference feature vectors may match the feature vector. Matching may be performed based on fuzzy matching. System 100 is configured to recognize and classify foreign objects 20 based on their object categories. System 100 identifies the object category of the foreign object 20 in the visible light image 120M. Based on the matched reference feature vectors, the object categories tagged with the matched reference feature vectors are retrieved, allowing for the identification or classification of the foreign object 20. Once the foreign object 20 is identified, System 100 generates and transmits a warning signal.

[0052] Figure 9 is a flowchart illustrating method 6000 for identifying foreign objects 20 on the runway. Method 6000 is the same as method 5000 in Figure 8, except that system 100 is configured to automatically detect and extract features from thermal images 110M and visible light images 120M, match the feature vectors with a plurality of reference feature vectors, and detect the foreign objects 20 in block 6306. In Figures 8 and 9, the same steps are denoted by the same reference numerals. System 100 can be configured to train an image processing module 134M using a deep learning module and to automatically execute the steps in block 6306.

[0053] The thermal camera 110 detects foreign objects 20 by detecting the difference in temperature, i.e., infrared thermal radiation, between the foreground (e.g., foreign objects 20) and the background (e.g., the surface of the runway 202). Foreign objects 20 of different categories or types are made of different materials such as metal, rubber, plastic, and concrete, and have different energy absorption rates, reflectances, and emissivity. Therefore, different categories of foreign objects 20 may result in different levels of temperature, i.e., infrared thermal radiation, relative to the background (i.e., the runway surface). The temperature difference between the foreign objects 20 and the runway can be detected by the thermal camera 110.

[0054] Therefore, it is beneficial to "train" the thermal camera 110 or thermal camera operating module 134T to distinguish between different categories of foreign objects 20 by identifying the type of material (e.g., rubber, metal, plastic, concrete, asphalt, etc.) that makes up the foreign object 20. Since foreign objects 20 made of different materials have different emissivity, and consequently different temperature levels and temperature contrast levels relative to the background, i.e., the runway, a "well-trained" thermal camera 110 can more accurately identify the foreign objects 20.

[0055] The thermal camera 110 is trained. Under normal sunny conditions, the thermal camera 110 undergoes an initial "training" period. During this training, the thermal camera 110 operates in "training" mode so that it can "learn" from the visible light image 102M of the visible light camera 120. After this initial "training," the thermal camera 110 has properly "learned" and can reliably and accurately detect foreign objects 20 with a relatively high level of accuracy. By increasing accuracy, it becomes possible to realize a system 100 with a "standalone" thermal camera 110 instead of using the visible light camera 120 and thermal camera 110 as a set. In this way, the system 100 can be applied even in bad weather or in situations with very poor visibility without requiring the visible light camera 120.

[0056] Figure 10 shows a flowchart of how to train the image processing module 134M to improve the identification of foreign objects 20 on the runway. In block 7110, the visible light camera 120 is configured to scan one of several sectors on the runway. The visible light camera 120 is configured to scan the sector sub-sector by subsector. The visible light camera 120 takes multiple visible light images 120M of the sector. The image processing module 134M processes the multiple visible light images 120M to detect the foreign object 20. In block 7210, the thermal camera 110 is configured to scan the same sector on the runway scanned by the visible light camera 120. The thermal camera 110 is configured to scan the sector sub-sector by subsector. The thermal camera 110 takes multiple thermal images 110M within the sector.

[0057] The image processing module 134M processes multiple thermal images 110M to detect the foreign object 20. The thermal camera 110 and the visible light camera 120 are configured to scan sectors simultaneously. In block 7120, the system 100 processes the visible light image 120M to identify the visible light object image 102B and then detects the foreign object 20. In block 7220, the system 100 processes the thermal image 110M to identify the thermal object image 110B and then detects the foreign object 20. The thermal image 110M and the visible light image 120M can be processed simultaneously by the processor 132. If the system 100 detects the foreign object 20 in the visible light image 120M, the system 100 generates a "suspected FOD" warning signal to notify the operator that the foreign object 20 has been detected in the visible light image 120M. Similarly, if system 100 detects a foreign object 20 in the thermal image 110M, system 100 generates a "suspected FOD" signal in block 7230 to notify the operator that a foreign object 20 has been detected in the thermal image 110M, since the detection of the foreign object 20 has not yet been verified. The "suspected FOD" warning signal is generated for both the visible light image 120M and the thermal image 110M. System 100 may display the thermal object image 110B and / or the visible light object image 120B on a display for the operator to visually inspect. System 100 generates at least one attribute of the visible light object image 120B in block 7130 and at least one attribute of the thermal object image 110B in block 7230.

[0058] At least one attribute includes the location of the visible light object image 120B in the visible light image 120M, the location of the thermal object image 110B in the thermal image 110M, the size of the visible light object image 120B, the size of the thermal object image 110B, and / or the temperature of the thermal object image 110B. For example, system 100 generates the location of the visible light object image 120B in the visible light image 120M and / or the size of the visible light object image 120B. For example, system 100 generates at least one of the location of the thermal object image 110B in the thermal image 110M, the size of the thermal object image 110B, and the temperature of the foreign object 20. In blocks 7132 and 7232, system 100 is configured to store at least one of the warning signal, attribute, feature, and image of this event in the foreign object warning signal and event database 742. In block 7140, system 100 is configured to determine whether or not foreign objects 20 are present in the visible light image 120M and the thermal image 110M by comparing at least one attribute of the visible light object image 120B and the thermal object image 110B. The method for comparing at least one attribute is shown in method 4140 in Figure 7. If the attributes of the visible light object image 120B and the thermal object image 110B match in block 7150, system 100 determines that foreign objects 20 have been detected in the visible light image 120M and the thermal image 110M. If foreign objects 20 are detected, system 100 generates and displays a warning signal in block 7170, for example, a "Confirmed FOD" warning signal. Otherwise, system 100 generates and transmits a "No Confirmed FOD" warning signal. In block 7172, system 100 is configured to store at least one of the signals, attributes, features, and images in the foreign object warning signal and event database 742. In block 7180, system 100 is configured to optimize the detection configuration parameters of the thermal camera 110. System 100 can optimize the detection configuration parameters based on foreign object warning signals and data stored in the event database 742.System 100 can run statistical analysis modules and / or optimization modules to optimize detection configuration parameters based on data. System 100 may use artificial intelligence to optimize detection configuration parameters based on data. In block 7182, System 100 is configured to store the optimized detection configuration parameters of the thermal camera 110 in the foreign object detection configuration parameter database 744 of the thermal camera 110.

[0059] System 100 determines the relationship between the object category of the foreign object 20 in the thermal image 110M and its temperature for all detected / verified foreign object samples. The processor 132 may further be configured to train the thermal camera 110 to detect foreign objects 20 based on the visible light image 120M from the visible light camera 120. Since it is much easier to identify and classify foreign objects 20 in the visible light image 120M, System 100 can determine the relationship between the object category of the visible light object image 120B acquired from the visible light camera 120 and the temperature of the thermal object image 110B from the thermal camera 110. Thus, if the size of the foreign object samples is sufficiently large, System 100 can determine the relationship between different foreign object categories (e.g., foreign objects made of different materials such as metal, plastic, and rubber) and their corresponding temperatures. System 100 then constructs a "FOD type thermal profile model" that can be used to map various types of foreign objects (i.e., foreign objects made of different materials) to their corresponding temperature ranges. In this way, the system can more easily identify the foreign object 20 based on the thermal object image 110B.

[0060] System 100 can identify foreign object categories or types using an "FOD type thermal profile model," and based on the temperature of the foreign object 20 detected by the thermal camera 110, it can identify the specific material type (metal, rubber, plastic, etc.) of the detected foreign object 20. The "FOD type thermal profile model" can be constructed using mathematical methods and / or statistical methods such as statistical correlation analysis. Alternatively, the "FOD type thermal profile model" may be constructed using artificial intelligence or machine learning techniques. This "FOD type thermal profile model" can be used to optimize the detection configuration parameters of the thermal camera 110.

[0061] To optimize the performance of the thermal camera 110, it is necessary to optimize its detection configuration parameters. The detection configuration parameters of the thermal camera 110 may be a set of operating parameters for the thermal camera 110 configured to enable the thermal camera 110 to detect foreign objects 20 with optimized and high accuracy. The operating parameters of the thermal camera 110 may include sensitivity, gain, brightness, contrast, shutter timing settings, etc. In this way, as the system 100 trains the thermal camera operating module 134T, the detection performance of the thermal camera 110 will improve over time, reaching a level where it can operate as the system 100's only "standalone" foreign object detector, that is, without the visible light camera 120. By optimizing the performance of the thermal camera 110, it becomes possible to function effectively even in bad weather or in situations with very poor visibility.

[0062] The thermal camera 110 can detect different levels of temperature contrast depending on the type of foreign object. Therefore, the thermal camera 110 can accurately detect the foreign object 20. As a result, the system 100 can classify or identify different categories or types of foreign objects 20 based on the different types of materials that make up the foreign object 20.

[0063] Database 742 stores information such as "suspected FOD," "confirmed FOD," and events that occurred using both the visible light camera 120 and the thermal camera 110. Database 742 also stores the attributes (category, size, location, temperature, etc.) of the detected and / or calculated foreign objects 20.

[0064] Figure 11 is a flowchart of method 8000 for detecting foreign objects 20 using a thermal camera 110. After "training," the thermal camera 110 can reliably and accurately detect foreign objects 20 with a relatively high level of accuracy. By establishing high accuracy, it becomes possible to use the thermal camera 110 without a visible light camera 120. As shown in Figure 11, the system 100 may be configured in block 8402 to detect foreign objects 20 using optimized detection configuration parameters of the thermal camera 110 stored in the detection configuration parameter database 844 for the thermal camera 110. The thermal camera 110 can be configured as the sole foreign object detector for detecting foreign objects 20.

[0065] Those skilled in the art will understand that the features described in one embodiment are not limited to that embodiment and can be combined with any of the other embodiments. The present invention relates to a system 100, a detection system 300 for detecting foreign objects on a runway, and a method for the system 100, as described herein with reference to the accompanying drawings and / or illustrated in the drawings.

Claims

1. A method for identifying foreign objects on a runway based on a thermal image of the foreign object, To capture a thermal image of the target area on the runway, To capture a visible light image of the target area on the runway, The detection of thermal object images within the thermal image and the detection of visible light object images within the visible light image, In the thermal image and the visible light object image, if the thermal image and the visible light object image are detected, it is determined that the foreign object has been detected. Identifying the visible light object image and classifying it into the object category of the foreign object, A method comprising: determining the relationship between the object category of the foreign object in the visible light image and the thermal object image of the foreign object; and, when the thermal object image of the foreign object is detected, identifying the foreign object by mapping the object category to the thermal object image based on the relationship.

2. The method according to claim 1, wherein determining the foreign object includes generating at least one attribute of the foreign object in each of the thermal object image and the visible light object image, and comparing the at least one attribute of the foreign object in the thermal object image and the visible light object image, wherein the foreign object is detected if the at least one attribute of the foreign object in the thermal object image and the visible light object image are the same.

3. The method according to claim 2, wherein the at least one attribute of the foreign object includes the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image.

4. The method according to claim 3, wherein if the distance between the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image is within the position parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

5. The method according to any one of claims 2 to 4, wherein the at least one attribute of the foreign object includes the size of the thermal object image and the visible light object image.

6. The method according to claim 5, wherein if the difference between the size of the thermal object image in the thermal image and the size of the visible light object image in the visible light image is within the size parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

7. The method according to any one of claims 1 to 6, further comprising obtaining an enlarged thermal object image and an enlarged visible light object image when the foreign object is detected.

8. This further includes identifying the object category of foreign objects in the thermal image, and identifying the object category is The thermal image is segmented into multiple thermal image regions, Assigning a feature vector to each of the aforementioned multiple thermal image regions, The process involves comparing the aforementioned feature vector with a plurality of reference feature vectors, wherein each of the plurality of reference feature vectors represents an object category. The method according to any one of claims 1 to 7, comprising identifying the reference feature vector and its object category that is closest to the feature vector.

9. The method according to claim 8, wherein segmenting the thermal image includes labeling each pixel in the thermal image, and grouping the labeled pixels having the same characteristics into a plurality of groups to form the plurality of thermal image regions.

10. Identifying the aforementioned object category The visible light image is segmented into multiple visible light image regions, Assigning a feature vector to each of the aforementioned multiple visible light image regions, The process involves comparing the aforementioned feature vector with a plurality of reference feature vectors, wherein each of the plurality of reference feature vectors represents an object category. The method according to any one of claims 1 to 11, comprising identifying the reference feature vector and its object category that is closest to the feature vector.

11. The method according to claim 10, wherein segmenting the visible light image includes labeling each pixel in the visible light image, and grouping the labeled pixels having the same characteristics into a plurality of groups to form a plurality of visible light image regions.

12. The method according to claim 10 or 11, further comprising training a thermal camera to detect the foreign object based on the visible light image from the visible light camera.

13. The method according to any one of claims 1 to 12, further comprising generating a thermal profile model of the foreign object based on the aforementioned relationship and optimizing the detection configuration parameters of a thermal camera based on the thermal profile model.

14. A detection system for identifying foreign objects on a runway based on thermal images of those foreign objects, A thermal camera having a first field of view and configured to capture a thermal image of a target area on the runway, A visible light camera having a second field of view and configured to capture a visible light image of the target area on the runway, wherein the first field of view overlaps with the second field of view, A processor that communicates with the thermal camera and the visible light camera, The system includes a memory that communicates with the processor and stores instructions that can be executed by the processor, The aforementioned processor, To detect the thermal object image within the thermal image, To detect a visible light object image within the aforementioned visible light image, In the thermal image and the visible light object image, if the thermal image and the visible light object image are detected, it is determined that the foreign object has been detected. Identifying the visible light object image and classifying it into the object category of the foreign object, The system is configured to determine the relationship between the object category of the foreign object in the visible light image and the thermal object image of the foreign object, A detection system that identifies a foreign object by mapping the object category to the thermal object image based on the relationship when a thermal object image of the foreign object is detected.

15. The detection system according to claim 14, wherein the processor is configured to generate at least one attribute of the foreign object in each of the thermal object image and the visible light object image, and to compare the at least one attribute of the foreign object in the thermal object image and the visible light object image, and the foreign object is detected if the at least one attribute of the foreign object in the thermal object image and the visible light object image are the same.

16. The detection system according to claim 15, wherein the at least one attribute of the foreign object includes the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image.

17. The detection system according to claim 16, wherein if the distance between the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image is within the position parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

18. The detection system according to any one of claims 15 to 17, wherein the at least one attribute of the foreign object includes the size of the thermal object image and the visible light object image.

19. The detection system according to claim 18, wherein if the difference between the size of the thermal object image in the thermal image and the size of the visible light object image in the visible light image is within the size parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

20. The detection system according to any one of claims 14 to 19, wherein the processor is further configured to zoom in on the thermal camera and the visible light camera to obtain an enlarged thermal image and an enlarged visible light image of the object when the foreign object is detected.

21. The processor is configured to identify the object category of the foreign object in the thermal image, The aforementioned processor, The thermal image is segmented into multiple thermal image regions, Assigning a feature vector to each of the aforementioned multiple thermal image regions, The process involves comparing the aforementioned feature vector with a plurality of reference feature vectors, wherein each of the plurality of reference feature vectors represents an object category. A detection system according to any one of claims 14 to 20, configured to identify the reference feature vector and its object category that is closest to the feature vector.

22. The detection system according to claim 21, wherein the processor is configured to segment the thermal image by labeling each pixel in the thermal image, and by grouping the labeled pixels having the same characteristics into a plurality of groups to form the plurality of thermal image regions.

23. In order to identify the object category of the foreign object in the visible light image, the processor: The visible light image is segmented into multiple visible light image regions, Assigning a feature vector to each of the aforementioned multiple visible light image regions, The process involves comparing the aforementioned feature vector with a plurality of reference feature vectors, wherein each of the plurality of reference feature vectors represents an object category. A detection system according to any one of claims 14 to 22, configured to identify the reference feature vector and its object category that is closest to the feature vector.

24. The detection system according to claim 23, wherein the processor is configured to segment the visible light image by labeling each pixel in the visible light image and grouping the labeled pixels having the same characteristics into a plurality of groups to form a plurality of visible light image regions.

25. The detection system according to any one of claims 23 to 24, wherein the processor is further configured to train the thermal camera to detect the foreign object based on the visible light image from the visible light camera.

26. The detection system according to any one of claims 14 to 22, wherein the process is further configured to generate a thermal profile model of the foreign object and to optimize the detection configuration parameters of the thermal camera based on the thermal profile model.

27. A system for detecting foreign objects on a runway divided into multiple sectors, Multiple camera sets arranged at intervals from each other, A thermal camera having a first field of view and configured to capture a thermal image of a target area on the runway, A plurality of camera sets each having a visible light camera having a second field of view and configured to capture a visible light image of the target area on the runway, wherein the first field of view overlaps with the second field of view, A processor that communicates with the aforementioned multiple camera sets, The system includes a memory that communicates with the processor and stores instructions that can be executed by the processor, The aforementioned processor, To detect the thermal object image within the thermal image, To detect a visible light object image within the aforementioned visible light image, In the thermal image and the visible light object image, if the thermal image and the visible light object image are detected, it is determined that the foreign object has been detected. Identifying the visible light object image and classifying it into the object category of the foreign object, The system is configured to determine the relationship between the object category of the foreign object in the visible light image and the thermal object image of the foreign object, When a thermal object image of the foreign object is detected, the foreign object is identified by mapping the object category to the thermal object image based on the relationship. A system in which each of the multiple camera sets is configured to scan one of the multiple sectors of the runway.

Images