System and method for detecting foreign objects on a runway

Abstract

This solution provides the ability to detect FOD (Foreign Object Deposition) even under poor visibility conditions such as bad weather, and to prevent or minimize false FOD detections. [Solution] This system includes a thermal camera that captures a thermal image of a first view of a target area of ​​a runway, a visible light camera that has a second field of view and captures a visible light image of the first view, a processor, and memory. The processor is configured to determine that the converted visible light image and the converted thermal image are images of the second view of the target area, to detect thermal object images in the thermal image, to detect visible light object images in the visible light image, and to determine that a foreign object has been detected when a thermal object image and a visible light object image are detected in the thermal image and visible light object image, respectively.

Description

Technical Field

[0001] The present invention relates to a system for detecting foreign objects on a runway and a method for the system.

Background Art

[0002] Foreign objects and debris (FOD) on the airport runway pose a danger to the landing and takeoff of aircraft on the runway. To reliably detect FOD under normal sunny conditions, a FOD detection system using a visible light spectrum camera is used. Under normal sunny conditions, such as when there is no fog, the FOD detection system can detect FOD with high accuracy by taking and processing high-resolution images of FOD. FOD includes, for example, engine and aircraft parts, tools, construction waste, rubber materials, natural substances, etc.

[0003] However, in the case of bad weather, especially foggy weather, the operation of the FOD detection system may be adversely affected and degraded. Since the FOD detection system operates only in the visible light spectrum, there is a risk that FOD cannot be reliably detected under foggy weather conditions, that is, under poor visibility conditions. Under such conditions caused by fog or the like, the visibility on the runway usually drops below 1 km, making it impossible to "see" FOD. Visibility conditions are classified into multiple categories. For example, Cat II represents a standard operation corresponding to a runway visual range (RVR) in the range of 550 meters (1,800 feet) to 300 meters (1,0OO feet). Cat IIIa represents a precision instrument approach and landing operation where the RVR is 175 meters (600 feet) or more. Cat IIIb represents a precision instrument approach and landing operation where the RVR is less than 175 meters (600 feet) and 50 meters (200 feet) or more. Cat IIIc represents a precision instrument approach and landing operation without RVR restrictions, including the case where the visibility is zero. Depending on the geographical location of the airport, the visibility of the airport runway varies and is appropriately classified. Many FOD detection systems can detect FOD within the airport at Cat II visibility, but cannot be used at airports with Cat IIIa, Cat IIIb, and Cat IIIc visibility.

[0004] Furthermore, FOD detection systems often generate false alarms or incorrect warnings. False warnings can be caused by phenomena such as light reflection from artificial light sources, including lights at nearby buildings or the runway edge. When such artificial light reflects off the smooth surface of the runway or off puddles or standing water on the runway surface, the FOD detection system may identify it as an FOD and issue a false warning. The number of such false warnings due to reflection often increases significantly after rainfall, when puddles or standing water form on the runway surface. While such reflections occur during the day, they are even more frequent at night, and during the early morning and evening hours.

[0005] When an aircraft is landing on a runway, the presence of FOD (Foreign Object Debris) can make it impossible for the aircraft to land safely. As an aircraft approaches a runway, it flies along the "final approach" of its flight path before reaching the landing zone. At this point, the aircraft is positioned close to the runway. The final approach is the last section of the flight path that an aircraft approaches in order to land on the runway. The final approach flight path is a flight path that descends in the direction of landing from the base leg towards the runway along the extension of the runway centerline. The aircraft needs to align itself with the extension of the runway centerline in preparation for the subsequent descent and landing on the runway. The aircraft typically turns from the base leg to the final approach within approximately 1,609 meters (1 mile) to 3,218 meters (2 miles) from the airport. Typically, the aircraft proceeds along the approach descent path on the final approach flight path and eventually lands in the runway landing zone. The approach descent path is generally at a 3-degree angle from the horizontal. Poor visibility on the runway could prevent pilots from seeing FODs (Foreign Object Debris) on the runway, potentially leading to a major disaster if an aircraft collides with an FOD or if an aircraft engine ingests FOD.

[0006] Therefore, it is important to provide a solution that can detect FOD even under poor visibility conditions such as bad weather, and that can prevent or minimize false FOD detections. [Overview of the project]

[0007] Various embodiments provide a method for detecting foreign objects on a runway. This method comprises: capturing a thermal image of a first view of a target area of ​​the runway from one side of the runway; capturing a visible light image of the first view of the target area of ​​the runway from the other side of the runway; converting at least one of the visible light image and the thermal image into a converted visible light image and a converted thermal image, respectively, wherein the converted visible light image and the converted thermal image are images of a second view of the target area; detecting a thermal object image in the thermal image; detecting a visible light object image in the visible light image; and determining that a foreign object has been detected when a thermal object image and a visible light object image are detected in the thermal image and the visible light object image, respectively.

[0008] In various embodiments, the first view may be an oblique view. In various embodiments, the second view may be a cockpit view as seen from the cockpit of the aircraft.

[0009] In various embodiments, transforming a visible light image may include rotating the visible light image. In various embodiments, transforming a thermal image may include rotating the thermal image.

[0010] In various embodiments, transforming a visible light image may include distorting the visible light image. In various embodiments, transforming a thermal image may include distorting the thermal image.

[0011] In various embodiments, the method may further include displaying at least one of the converted visible light image and the converted thermal image on a display in the cockpit of an aircraft.

[0012] In various embodiments, determining a foreign object may involve generating at least one attribute of the foreign object in both the thermal object image and the visible light object image, comparing at least one attribute of the foreign object in the thermal object image and the visible light object image, and detecting the foreign object if at least one attribute of the foreign object in the thermal object image and the visible light object image are the same.

[0013] In various embodiments, at least one attribute of the foreign object may include the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image.

[0014] In various embodiments, if the distance between the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image is within the position parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

[0015] In various embodiments, at least one attribute of the foreign object may include the size of the thermal object image and the visible light object image. In various embodiments, if the difference between the size of the thermal object image in the thermal image and the size of the visible light object image in the visible light image is within the size parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

[0016] Various embodiments provide a method for detecting foreign objects on a runway. This method comprises: capturing a plurality of thermal images of a first view of a target area of ​​the runway from one side of the runway; capturing a plurality of visible light images of the first view of the target area of ​​the runway from one side of the runway; converting at least one of the plurality of visible light images and the plurality of thermal images into a plurality of converted visible light images and a plurality of converted thermal images, respectively, wherein the plurality of converted visible light images and the plurality of converted thermal images are images of a second view of the target area; detecting a thermal object image in the thermal image; detecting a visible light object image in the visible light image; and determining that a foreign object has been detected when a thermal object image and a visible light object image are detected in the thermal image and the visible light object image, respectively.

[0017] In various embodiments, the first view may include a perspective view. In various embodiments, the second view may include a cockpit view as seen from the cockpit of the aircraft.

[0018] In various embodiments, transforming multiple visible light images may include rotating multiple visible light images. In various embodiments, transforming multiple thermal images may include rotating multiple thermal images.

[0019] In various embodiments, transforming multiple visible light images may include distorting multiple visible light images. In various embodiments, transforming multiple thermal images may include distorting multiple thermal images.

[0020] In various embodiments, the method may further include stitching together a plurality of converted visible light images to form a single visible light image. In various embodiments, the method may further include stitching together a plurality of converted thermal images to form a single thermal image.

[0021] In various embodiments, the method may further include displaying at least one of a single visible light image and a single thermal image on a display in the cockpit of an aircraft.

[0022] In various embodiments, a system is provided for detecting foreign objects on a runway. This system comprises a thermal camera having a first field of view and configured to capture a thermal image of a first view of a target area of ​​the runway from one side of the runway; a visible light camera having a second field of view and configured to capture a visible light image of a first view of a target area of ​​the runway from one side of the runway, wherein the first field of view of the visible light camera overlaps with the second field of view; a processor that communicates with the thermal camera and the visible light camera; and a memory that communicates with the processor and stores instructions that can be executed by the processor. The processor is configured to convert at least one of the visible light image and the thermal image into a converted visible light image and a converted thermal image, respectively, and to determine that the converted visible light image and the converted thermal image are images of the second view of the target area; to detect a thermal object image in the thermal image; to detect a visible light object image in the visible light image; and to determine that a foreign object has been detected when a thermal object image and a visible light object image are detected in the thermal image and the visible light object image, respectively.

[0023] In various embodiments, the first view may include a perspective view. In various embodiments, the second view may include a cockpit view as seen from the cockpit of the aircraft.

[0024] In various embodiments, the processor may be configured to rotate the visible light image in order to transform the visible light image. In various embodiments, the processor may be configured to rotate the thermal image in order to convert the thermal image.

[0025] In various embodiments, the processor may be configured to distort the visible light image in order to convert the visible light image. In various embodiments, the processor may be configured to distort the thermal image in order to convert the thermal image.

[0026] In various embodiments, the processor may be further configured to display at least one of the converted visible light image and the converted thermal image on a display within the aircraft cockpit.

[0027] In various embodiments, to determine a foreign object, the processor generates at least one attribute of the foreign object in each of the thermal object image and the visible light object image, compares at least one attribute of the foreign object in the thermal object image and the visible light object image, and may be configured such that a foreign object is detected when at least one attribute of the foreign object in the thermal object image and the visible light object image is the same. <m

[0028] In various embodiments, at least one attribute of the foreign object may include the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image.

[0029] In various embodiments, when the distance between the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image is within the position parameter, at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

[0030] In various embodiments, at least one attribute of the foreign object may include the sizes of the thermal object image and the visible light object image. In various embodiments, if the difference between the size of the thermal object image in the thermal image and the size of the visible light object image in the visible light image is within the size parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

[0031] In various embodiments, a system is provided for detecting foreign objects on a runway divided into multiple sectors. The system comprises multiple camera sets spaced apart from each other, each having a thermal camera having a first field of view and configured to capture a thermal image of a first view of a target area of ​​the runway from one side of the runway, and a visible light camera having a second field of view and configured to capture a visible light image of a first view of a target area of ​​the runway from one side of the runway, with the first field of view overlapping with the second field of view; a processor that communicates with the thermal camera and the visible light camera; and a memory that communicates with the processor and stores instructions that can be executed by the processor. The processor may be configured to convert at least one of a visible light image and a thermal image into a converted visible light image and a converted thermal image, respectively, and to determine that the converted visible light image and the converted thermal image are images of a second view of the target area, to detect a thermal object image in the thermal image, to detect a visible light object image in the visible light image, and to determine that a foreign object has been detected when a thermal object image and a visible light object image are detected in the thermal image and the visible light object image, respectively, and the multiple camera sets may each be configured to scan one of multiple sectors of the runway.

[0032] In various embodiments, the processor may be configured to stitch together multiple converted visible light images from multiple camera sets to form a single visible light image.

[0033] In various embodiments, the processor may be configured to stitch together multiple converted thermal images from multiple camera sets to form a single thermal image. [Brief explanation of the drawing]

[0034] [Figure 1] A schematic diagram illustrating an exemplary embodiment of a system for detecting foreign objects on a runway. [Figure 1A] A schematic diagram showing a visible light image containing a visible light object image of a foreign object, and a thermal image containing a thermal object image of the foreign object. [Figure 2A] A diagram showing an exemplary embodiment of the system. [Figure 2B] Figure 2A shows the system scanning one of several sectors of the runway. [Figure 2C] A diagram illustrating an exemplary embodiment of a first view of the area of ​​interest. [Figure 2D] A diagram illustrating an exemplary embodiment of a second view of the target area on the display. [Figure 3] A flowchart illustrating an exemplary method for detecting foreign objects on a runway. [Figure 3A] A diagram illustrating an exemplary embodiment in which a visible light image or a thermal image is converted into a rotated visible light image or a rotated thermal image. [Figure 3B] A diagram illustrating an exemplary embodiment of a system for removing foreign objects from a runway divided into multiple sectors. [Figure 4] A diagram illustrating an exemplary embodiment of stitching together multiple rotated visible light images or multiple rotated thermal images to form a single visible light image or a single thermal image, respectively. [Figure 4A] A perspective view showing an aircraft along its final approach flight path. [Figure 5] A flowchart illustrating an exemplary method for detecting foreign objects on a runway. [Figure 6] A flowchart illustrating an exemplary method for detecting foreign objects on a runway. [Figure 7]A flowchart illustrating an exemplary method for comparing at least one attribute of a foreign object in thermal and visible light object images. [Figure 8] A flowchart illustrating a method for identifying foreign objects on a runway. [Figure 9] A flowchart illustrating a method for identifying foreign objects on a runway. [Figure 10] A flowchart illustrating how to train an image recognition module to improve the identification of foreign objects on runways. [Figure 11] A flowchart illustrating a method for detecting foreign objects using a thermal camera. [Modes for carrying out the invention]

[0035] In the drawings referenced in the following examples, identical features are denoted by the same reference numerals. Figure 1 is a schematic diagram showing an exemplary embodiment of a system 100 for detecting foreign objects 20 on a runway. The system 100 includes a thermal camera 110 having a first field of view 110F and configured to capture a thermal image 110M of a first view of a target area 112 of the runway from one side of the runway; a visible light camera 120 having a second field of view 120F and configured to capture a visible light image of a first view of a target area 112 of the runway from one side of the runway, wherein the first field of view 110F overlaps with the second field of view 120F; a processor 132 that communicates with the thermal camera 110M and the visible light camera 120M; and a device that communicates with the processor 132 and executes instructions that can be executed by the processor 132. The system 100 includes a memory 134 for storage, and the processor 132 is configured to convert at least one of a visible light image and a thermal image into a converted visible light image and a converted thermal image, respectively, and to determine that the converted visible light image and the converted thermal image are images of a second view of the target area, to detect a thermal object image in the thermal image, to detect a visible light object image in the visible light image, and to determine that a foreign object 20 has been detected when a thermal object image and a visible light object image are detected in the thermal image and the visible light object image, respectively. The system 100 includes a server including the processor 132, memory 134, and an I / O interface 136 configured to connect the processor 132 and peripheral interface modules (keyboard, mouse, touchscreen, display, etc.). The system 100 includes a communication module 138 configured to facilitate wired or wireless communication between the system 100 and other user devices (mobile devices, laptops, etc.) over the internet. The system 100 includes a storage device 140 configured to store data. System 100 is equipped with a display (monitor, touchscreen, etc.) for displaying signals such as warning signals and images to the operator.System 100 is configured to detect foreign objects and debris (FOD) on runways, taxiways, aprons, ramps, etc., under both daytime and nighttime ambient light conditions, without requiring auxiliary lighting such as visible spectrum illumination, infrared illumination, or laser illumination.

[0036] Figure 1A is a schematic diagram showing a visible light image 120M including a visible light object image 120B of the foreign object 20, and a thermal image 110M including a thermal object image 110B of the foreign object. System 100 has an image processing module 134M (see Figure 1) configured to process images 110T and 120T captured by the thermal camera 110 and the visible light camera 120. System 100 has a thermal camera operation module 134T which contains operating parameters for the thermal camera 110 for operating the thermal camera 110. System 100 has a visible light camera operation module 134V which contains operating parameters for the visible light camera 120 for operating the visible light camera 120. Modules 134T, 134V, and 134M are stored in a storage device 140, loaded into memory 134, and processed by the processor 132. The image processing module 134 is configured to convert visible light images and / or thermal images into converted visible light images and converted thermal images.

[0037] When the thermal image 110M and the visible light image 120M are captured, the images 110M and 120M are transmitted to the processor 132 for processing. The processor 132 receives and processes the thermal image 110M and the visible light image 120M to detect foreign objects 20 on the runway.

[0038] Figure 2A shows an exemplary embodiment of system 200. System 200 comprises a camera set, namely a visible light camera 220 and a thermal camera 210. The camera set 210S is rigidly mounted on an actuator 250 configured to move the camera set 210S. The camera set 210S is controlled by a processor 132 to scan a sector of the runway to detect foreign objects 20 on the runway surface.

[0039] The actuator 250 may be a pan / tilt unit (PTU) configured to simultaneously pan and tilt the camera set 210S so that the field of view of the camera set 210S is the same and the same target area can be focused on. The actuator 250 is configured to pan the camera set 210S horizontally 210H and / or tilt the camera set 210S vertically 210V. The actuator 250 is able to communicate with the processor 132, which is configured to remotely control the movement of the actuator 250 to pan and tilt the camera set 210S to scan the runway. The actuator 250 is usually installed on top of a support structure 252, such as a mast structure, which is located along the runway. The support structure is installed at a distance of 120m to 350m from the runway centerline 304 (see Figure 3).

[0040] Figure 2B shows the system 100 of Figure 2A scanning one of several sectors 202S of runway 202. Each camera set 210S consists of a thermal camera 210 and a visible light camera 220, each having a field of view 210F and 220F, respectively, and is configured to capture a target area 212. The field of view 210F of the visible light camera 220 overlaps with the field of view 220F of the thermal camera 210. The fields of view 210F and 220F of both the visible light camera 220 and the thermal camera 210 cover a specific target area 212 within sector 202S of runway 202. Because the fields of view 210F and 220F overlap in this way, both the visible light camera 220 and the thermal camera 210 can simultaneously detect the same foreign object 20 on sector 202S of the runway while scanning the sector 202S.

[0041] Figure 2C shows an exemplary embodiment of a first view of the area of ​​interest 212. The first view is in perspective. Camera set 210S is positioned on one side of runway 202 and captures a first view of the area of ​​interest 212. Camera set 210S is positioned away from runway 202, for example, 120 to 150 meters from the centerline of runway 202.

[0042] Figure 2D shows an exemplary embodiment of a second view of the display 260 of the area of ​​interest 212. The display 260 is located inside the aircraft cockpit. The second view is the cockpit view, which is the view a pilot sees when looking out from the cockpit. As shown in Figure 2D, the distal end of runway 202 is narrower than the proximal end of runway 202. The view shows the aircraft looking at the distal end of runway 202 from the proximal end of runway 202. The system 100 can detect foreign objects 20 on runway 202 and display them on the display 260 installed in the aircraft cockpit. The foreign objects 20 are highlighted in a graphic box 260B to make them more visible to the pilot. This allows the pilot to take appropriate measures to avoid the foreign objects 20. The pilot can delay the landing procedure until the foreign objects 20 are removed from the runway, or choose other methods to avoid the foreign objects 20.

[0043] System 100 may be configured to generate supplementary augmented reality images and videos by augmenting live video and images of the airport airside captured by the aircraft to assist pilots in confirming runway 202. System 100 can add visible light object images 120B and / or thermal object images 110B to the live video captured by the aircraft. By augmenting the video of runway 202 captured by System 100, it is possible to display video as if it were taken from the perspective of the cockpit of a landing aircraft, based on the aircraft's position relative to runway 202. This information is transmitted in real time to the cockpit of the approaching aircraft to assist the pilot during landing.

[0044] The high-resolution visible light and / or hyperspectral thermal images captured by the system can depict runway 202 better than what a human can see under low-visibility conditions such as fog, haze, rain, and low light. Therefore, by transmitting augmented reality images to the display 260, pilots can be shown clearer and higher-resolution images of runway 202 compared to aircraft that do not use system 100. Useful information for pilots, such as the presence of foreign objects 20 detected on runway 202, can also be overlaid on the augmented image. System 100 can detect foreign objects 20 at airports with Cat II, Cat IIIa, Cat IIIb, and Cat IIIc visibility.

[0045] Figure 3 is a flowchart illustrating an exemplary method 1000 for detecting foreign objects 20 on runway 202. This method involves taking a thermal image 110M of a first view of a target area 112 of runway 202 from one side of the runway in block 1010, taking a visible light image 120M of a first view of a target area 112 of runway 202 from one side of the runway in block 1020, and converting at least one of the visible light image 120M and the thermal image 110M into a converted visible light image and a converted thermal image, respectively, in block 1030. The process includes: the replaced thermal image being an image of a second view of the target region 112; detecting a thermal object image 110B within the thermal image 110M in block 1040; detecting a visible light object image 120B within the visible light image 120M in block 1050; and determining in block 1060 that a foreign object 20 has been detected if the thermal object image 110B and the visible light object image 120B are detected in the thermal image 110M and the visible light object image 120B, respectively. The thermal object image 110B is a part of the thermal image 110M representing the foreign object 20 within the thermal image 110M, and may also be simply referred to as the foreign object 20 within the thermal image 110M. The visible light object image 120B is a part of the visible light image 120M representing the foreign object 20 within the visible light image 120M, and may also be simply referred to as the foreign object 20 within the visible light image 120M.

[0046] This method may include scanning the runway with the thermal camera 110 and the visible light camera 120 before capturing the thermal image 110M and the visible light image 120M. As the thermal camera 110 and the visible light camera 120 scan the runway sectors, they capture thermal images 110M and visible light images 120M of multiple target areas 112 along the sectors. To detect foreign objects 20, the image processing module 134M processes the thermal image 110M and the visible light image 120M to determine whether or not foreign objects 20 are present in the thermal image 110M and the visible light image 120M. When foreign objects 20 are detected, the image processing module 134M is configured to identify the thermal object image 110B and the visible light object image 120B, respectively, within the thermal image 110M and the visible light image 120M. When the foreign object 20 is identified, the system 100 generates a warning signal.

[0047] Figure 3A shows an exemplary embodiment in which a visible light image 120M or a thermal image 110M is converted to a rotated visible light image 120T or a rotated thermal image 110R. The same conversion can be performed to convert a thermal image 110M to a converted thermal image. As shown in Figure 2C, the visible light image 120M is captured by a visible light camera 120. Referring to Figure 3A, the visible light image 120M is received by a processor 132. The processor 132 executes an image processing module 134M to convert the image. The processor 132 is configured to distort the visible light image 120M (as shown by the curved arrow) at 1202 and to obtain a distorted visible light image at 1204. The processor 132 is configured to recognize and adjust the details of the visible light image 120M at 1206. For example, processor 132 moves the centerline 202C of runway 202 downward, positioning the centerline 202C towards the center of runway 202. This results in a visible light image of runway 202 that resembles a top view of runway 202, as shown in 1208. If processor 132 detects a visible light object image (not shown in Figure 3A), it may move the visible light object image vertically, i.e., perpendicular to the centerline. Processor 132 rotates the visible light image 120M by 90 degrees counterclockwise, for example, in 1210, generating the rotated visible light image 120R in 1212. Processor 132 may also distort the visible light image 120M in 1212, converting the visible light image 120M into the transformed visible light image 120T, and configure it to display the runway in a perspective view, as shown in Figure 2D. The processor 132 is configured to display at least one of the converted visible light image 120T and the converted thermal image 110T on a display 260 (not shown in Figure 3A) in the aircraft cockpit. The converted visible light image 1202T and the converted thermal image 110T may be images of one of several sectors 202S of runway 202. The conversion may also be performed on the visible light object image 120B and / or the thermal object image 110B.

[0048] To detect the foreign object 20, the above method includes generating at least one attribute of the foreign object 20 in each of the thermal object image 110B and the visible light object image 120B, and comparing at least one attribute of the foreign object 20 in the thermal object image 110B and the visible light object image 120B. The foreign object 20 is detected if at least one attribute of the foreign object 20 in the thermal object image 110B and the visible light object image 120B is the same or falls within a specific parameter or threshold level. The system 100 may be configured to obtain an enlarged thermal object image 110B and an enlarged visible light object image 120B by zooming in on the detected foreign object 20 with the visible light camera 120 and the thermal camera 110 when the foreign object 20 is detected.

[0049] Figure 3B shows an exemplary embodiment of a system 300 for detecting foreign objects 20 on a runway 302 divided into multiple sectors 302S. The system 300 has multiple camera sets 310S spaced apart from each other. Each camera set 310S has a thermal camera 210 configured to have a first field of view 210F and capture a thermal image 110M of a first view of a target area 212 of the runway 302 from one side of the runway 302, and a visible light camera 220 configured to have a second field of view 220F and capture a visible light image 120M of a first view of a target area 212 of the runway 302 from one side of the runway 302. The first field of view 210F overlaps with the second field of view 220F. The system 300 further includes a processor 132 that communicates with a plurality of visible light cameras 220 and a plurality of thermal cameras 210, and a memory that communicates with the processor 132 and stores instructions that can be executed by the processor. The processor is configured to convert at least one of the visible light image and the thermal image into a converted visible light image and a converted thermal image, respectively, and to determine that the converted visible light image and the converted thermal image are images of a second view of the target area, to detect a thermal object image 110B in the thermal image 110M, to detect a visible light object image 120B in the visible light image 120M, and to determine that a foreign object 20 has been detected when the thermal object image 110B and the visible light object image 120B are detected in the thermal image 110M and the visible light object image 120B, respectively. The plurality of camera sets 310S are each configured to scan one of a plurality of sectors 302S of the runway 302. As shown in Figure 3, the runway 302 is divided into multiple sectors 302S. Each of the multiple camera sets 310S detects foreign matter 20 on the surface of each sector 302S by scanning one of these multiple sectors 302S. Each sector 302S may be further divided into multiple subsectors. Each camera set 310S is operable to scan the corresponding sector 302S and then scan that sector 302S subsector by subsector.In this way, when a foreign object 20 is detected, the system 300 can identify the sector 302S based on the camera set 310S scanning the sector 302S. The camera set 310S can scan the sector 302S in a specific common scanning direction, for example, from the leftmost subsector to the rightmost subsector, or from the rightmost subsector to the leftmost subsector.

[0050] Figure 4 shows an exemplary embodiment in which multiple rotated visible light images 120R or multiple rotated thermal images 110R are stitched together to form a single visible light image 120U or a single thermal image 110U, respectively. Multiple camera sets 210S capture multiple visible light images 120M of a first view of the target area 212 of the runway 202 from one side of the runway 202, and multiple thermal images 110M of the first view of the target area 212 of the runway 202 from the other side of the runway 202. The processor 132 converts at least one of the multiple visible light images 120M and the multiple thermal images 110M into multiple converted visible light images 120T and multiple converted thermal images 110T, respectively. The multiple converted visible light images and multiple converted thermal images are images of a second view of the target area. As shown in Figure 4, the processor 132 executes the image processing module 134M to stitch together multiple rotated visible light images 120R to form a single visible light image 120U, and / or stitch together multiple rotated thermal images 110R to form a single thermal image 110U. Thus, the multiple rotated visible light images 120R and the multiple rotated thermal images 110R represent multiple sectors 202S, and when stitched together, the single visible light image 120U and the single thermal image 110U represent a top-down view of the runway 202. As previously mentioned, the processor 132 can distort the single visible light image 120U and / or the single thermal image 110U to generate a second view of the runway 202, i.e., a cockpit view.

[0051] Figure 4A is a perspective view of aircraft 10 along the aircraft's final approach flight path. As shown in Figure 4A, the cockpit view 10F of aircraft 10 overlaps with sector 202S of runway 202. When the converted visible light image 120T and / or converted thermal image 110T are displayed on display 260, the pilot can see the foreign object 20 from the display in the cockpit of aircraft 10. By directing the pilot's attention to the graphics box 260B, the pilot can locate the foreign object 20.

[0052] System 100 receives the position coordinates (e.g., 3D position, GPS coordinates) of the aircraft 10 along runway 202. Based on the position of the aircraft 10, System 100 is configured to identify a converted visible light image and / or a converted thermal image representing the view from the cockpit of the aircraft 10, and to transmit the identified image to the cockpit display 260. The image may include position coordinates recorded as metadata embedded in the image. The position coordinates are the position of the camera set 210S and / or the position of the target area 212 captured in the image, etc. System 100 is configured to identify a second view, i.e., a cockpit view image, by processing based on the position coordinates of the aircraft and the image, and to transmit that image to the display 260. If the image includes a visible light object image 120B and a thermal object image 110B of the foreign object 20, the pilot can see the foreign object 20 on the display 260.

[0053] The thermal camera 210 detects foreign objects 20 on runway 302 by detecting the difference in thermal radiation levels (or temperature) between the foreground (i.e., foreign objects 20) and the background (i.e., the runway surface). The thermal camera 210 operates in the infrared spectrum and does not require ambient light to "see" the foreign objects 20. The thermal camera 210 is also commonly referred to as an infrared thermal camera. The thermal camera 210 is a medium-wave infrared (MWIR) camera or a long-wave infrared (LWIR) camera. The thermal camera 210 has the advantage of being able to detect foreign objects 20 on runway 302 even in very poor visibility conditions, and even in zero illumination, i.e., complete darkness. For this reason, the thermal camera 210 has the advantage of being able to detect foreign objects 20 on runway 302 even in foggy weather conditions. The thermal camera 210 captures monochrome images and video output and transmits them to the processor 132. The thermal camera 210 is a completely passive camera that does not actively transmit or emit radio frequencies, microwaves, artificial lighting, infrared, lasers, LIDAR, etc. Therefore, the thermal camera 210 has advantages such as not interfering with existing airport systems / equipment and aircraft systems / equipment, not interfering with airport systems / equipment and aircraft systems / equipment in the future, and not requiring frequency / spectrum licenses and approvals from airport and frequency spectrum regulatory authorities.

[0054] Unlike the thermal camera 210, the visible light camera 220 operates within the visible spectrum, so in order to "see" foreign objects 20 on the runway 302, the ambient visible spectral light must be above a certain minimum amount. The visible light camera 220 cannot detect foreign objects 20 if visibility conditions are very poor or if the illumination is zero. For example, the visible light camera 220 cannot detect foreign objects 20 if visibility conditions (on the runway surface) are very poor or if fog is present (on the runway surface). The visible light camera 220 can capture and transmit full-color and high-resolution images / videos (e.g., HD (FHD) or 4K Ultra HD (4K UHD) resolution). High-resolution color images not only allow the system 300 to reliably and accurately recognize and classify the detected foreign objects 20, but also allow the operator to reliably and accurately perform visual verification and confirmation of the detected foreign objects 20. Therefore, by using both the visible light camera 220 and the thermal camera 210 in combination, the system 300 can function even under very poor visibility conditions, such as foggy weather conditions, and can accurately and reliably detect foreign objects 20 on the surface of runway 302. The visible light camera 220 is configured to capture a color, high-resolution visible light image 120M and output it to the processor 132. The visible light camera 220 can operate without the need for infrared illumination, visible spectrum illumination, or laser illumination. Because the system 300 is passive, it has the advantage of not posing a danger to or interfering with other airport systems or aircraft systems during aircraft landings / takeoffs on runway 302. The system 300 has the advantages of not interfering with existing airport systems / equipment and aircraft systems / equipment, not interfering with airport systems / equipment and aircraft systems / equipment in the future, and not requiring frequency / spectrum licenses and approvals from airport and frequency spectrum regulatory authorities.

[0055] Figure 5 shows a flowchart of an exemplary method for detecting foreign objects 20 on a runway. System 100 designates a visible light camera 120 as the primary detector and a thermal camera 110 as the secondary detector. As shown in Figure 5, in block 2110, the visible light camera 120 is configured to scan one of several sectors on the runway. The visible light camera 120 is configured to scan the sector sub-sector by subsector. The visible light camera 120 captures multiple visible light images 120M within each sector. The image processing module 134M processes the multiple visible light images 120M to detect the foreign objects 20. In block 2210, the thermal camera 110 is configured to scan the same sector on the runway scanned by the visible light camera 120. The thermal camera 110 is configured to scan the sector sub-sector by subsector. The thermal camera 110 captures multiple thermal images 110M within the same sector. The image processing module 134M processes multiple thermal images 110M to detect the foreign object 20. The thermal camera 110 and the visible light camera 120 are configured to scan sectors simultaneously. The image processing modules 134M for the visible image and the thermal image 110M may be separate modules for processing the visible light image 120M and the thermal image 110M, respectively.

[0056] In block 2120, system 100 detects the foreign object 20 after processing the visible light image 120M. System 100 identifies the visible light object image 120B within the visible image. In block 2220, system 100 detects the foreign object 20 after processing the thermal image 110M. System 100 identifies the thermal object image 110B within the thermal image 110M. The thermal image 110M and the visible light image 120M can be processed simultaneously by processor 132. If system 100 detects the foreign object 20 within the visible light image 120M, system 100 generates a "suspected FOD" warning signal in block 2130 to notify the operator that the foreign object 20 has been detected within the visible light image 120M. Similarly, if system 100 detects a foreign object 20 in the thermal image 110M, system 100 generates a “suspected FOD” warning signal in block 2230 to notify the operator that a foreign object 20 has been detected in the thermal image 110M, since the detection of the foreign object 20 has not yet been verified. The “suspected FOD” signal is generated for both the visible light image 120M and the thermal image 110M. System 100 may display the thermal object image 110B and / or the visible light object image 120B on a display for the operator to visually inspect. System 100 generates at least one attribute for the visible light object image 120B and the thermal object image 110B. At least one attribute includes the location of the visible light object image 120B in the visible light image 120M, the location of the thermal object image 110B in the thermal image 110M, the size of the visible light object image 120B, and / or the size of the thermal object image 110B. For example, system 100 generates the position of the visible light object image 120B in the visible light image 120M, the position of the thermal object image 110B in the thermal image 110M, and / or the sizes of the visible light object image 120B and the thermal object image 110B. In block 2140, system 100 is configured to determine whether or not a foreign object 20 has been detected in the visible light image 120M and the thermal image 110M by comparing at least one attribute of the visible light object image 120B and the thermal object image 110B. Details of this comparison step are shown in Figure 7.If the attributes of the visible light object image 120B and the thermal object image 110B match in block 2150, the system 100 determines that a foreign object 20 has been detected in the visible light image 120M and the thermal image 110M. The system 100 may receive confirmation input from the operator via a peripheral interface module to confirm the detection of the foreign object 20 after visually inspecting the visible light image 120M and / or the thermal image 110M on the display. In block 2160, the system 100 identifies the foreign object 20 based on at least one attribute. Once the foreign object 20 is detected and / or identified, the system 100 generates a warning signal in block 2170, for example, a "Confirmed FOD" signal. Otherwise, the system 100 generates a "No Confirmed FOD" warning signal. The system 100 may transmit the warning signal to the operator's mobile device or display the warning signal on the display for the operator's confirmation. The system 100 may generate a warning signal when it receives input indicating that the operator has confirmed the detection of a foreign object 20.

[0057] Figure 6 shows a flowchart of an exemplary method 3000 for detecting foreign objects 20 on the runway. Method 3000 is the same as method 2000 in Figure 5, except that the system 100 is configured to designate the thermal camera 110 as the primary detector and the visible light camera 120 as the secondary detector. In Figures 5 and 6, the same steps are denoted by the same reference numerals. As shown in Figure 6, in block 3110, the thermal camera 110 is configured to scan one of several sectors on the runway. The thermal camera 110 is configured to scan the sector sub-sector by sub-sector. The thermal camera 110 takes multiple thermal images 110M within each sector. The image processing module 134M processes the multiple thermal images 110M to detect the foreign objects 20. In block 3210, the visible light camera 120 is configured to scan the same sector on the runway scanned by the thermal camera 110. The visible light camera 120 is configured to scan the sector sub-sector by sub-sector. The visible light camera 120 captures multiple visible light images 120M within the sector. The image processing module 134M processes the multiple visible light images 120M to detect the foreign object 20. The thermal camera 110 and the visible light camera 120 are configured to scan the sector simultaneously. In block 3120, the system 100 detects the foreign object 20 after processing the thermal image 110M. The system 100 identifies the thermal object image 110B within the thermal image 110M. In block 3230, the system 100 detects the foreign object 20 after processing the visible light image 120M. The system 100 identifies the visible light object image 120B within the visible light image 120M. The thermal image 110M and the visible light image 120M can be processed simultaneously by the processor 132. Blocks 3140-3170 are the same as blocks 2140-2170 in Figure 5.

[0058] Figure 7 is a flowchart illustrating an exemplary method 4140 for comparing at least one attribute of a foreign object 20 in a thermal object image 110B and a visible light object image 120B. Method 4140 is used in blocks 2140 and 3140 of Method 2000 in Figure 5 and Method 3000 in Figure 6. At least one attribute of the foreign object 20 includes the position of the thermal object image 110B in the thermal image 110M and the position of the visible light object image 120B in the visible light image 120M. In block 4141, it is determined that at least one attribute of the foreign object 20 in the thermal object image 110B and the visible light object image 120B is the same if the distance between the position of the thermal object image 110B in the thermal image 110M and the position of the visible light object image 120B in the visible light image 120M is within the position parameter. For example, the processor 132 identifies the positional difference between the visible light object image 120B and the thermal object image 110B in the visible light image 120M and the thermal image 110M, and determines whether the positional difference is within a positional parameter, i.e., a predetermined positional threshold level. The positional parameter can be determined by statistical analysis of the detection positions of all detected foreign object samples 20. If the positional difference is within the range of the positional parameter, the processor 132 generates a "position match" warning signal in block 4142.

[0059] At least one attribute of the foreign object 20 includes the size in the thermal object image 110B and the visible light object image 120B. In block 4143, if the difference between the size of the thermal object image 110B in the thermal image 110M and the size of the visible light object image 120B in the visible light image 120M is within the size parameter, it is determined that at least one attribute of the foreign object 20 in the thermal object image 110B and the visible light object image 120B is the same. For example, the processor 132 identifies the size difference between the sizes of the visible light object image 120B and the thermal object image 110B in the visible light image 120M and the thermal image 110M, and determines whether the size difference is within the size parameter, i.e., a predetermined size threshold level. The size parameter can be determined based on a statistical analysis of the measured size of all detected foreign object 20 samples. If the size difference is within the range of the size parameter, the processor 132 generates a "size match" warning signal in block 4144.

[0060] Depending on the configuration of system 100, the process detects foreign objects 20 based on the position and / or size of the thermal object image 110B and the visible light object image 120B. For example, if both the position and size of the thermal object image 110B and the visible light object image 120B are used, foreign objects 20 are detected if the position and size of the thermal object image 110B and the visible light object image 120B are within the range of the position parameter and size parameter, respectively, i.e., they match. When foreign objects 20 are detected, system 100 generates a warning signal, for example, an "attribute match" signal when attributes match in block 4145. System 100 may also generate warning signals if the "position match" warning signal and the "size match" warning signal are turned on or generated.

[0061] The exemplary system 100 and method described above provide a solution that enables the detection of foreign objects 20 during adverse weather conditions and prevents or minimizes false detections of foreign objects 20. For example, reflections from puddles or standing water after rain, or reflections from the smooth surface of a runway, occur within the visible light spectrum. Because the visible light camera 120 operates only within the visible light spectrum, system 100 tends to mistakenly identify these reflections as foreign objects 20, i.e., "suspected FODs." As a result, system 100 may issue inappropriate warnings or false detection alarms. Therefore, by comparing and detecting foreign objects 20 using both the thermal image 110M and the visible light image 120M, system 100 can more accurately detect foreign objects 20 and prevent or minimize false detections of foreign objects 20.

[0062] Refer to Method 2000 in Figure 5 and Method 3000 in Figure 6. When comparing at least one attribute of the visible light object image 120B and the thermal object image 110B, the system 100 may determine that no foreign object 20 is detected, i.e., that at least one attribute of the visible light object image 120B does not match at least one attribute of the thermal image 110M. In this case, the system 100 generates a warning signal of “No Confirmed FOD”. In such a situation, the system 100 identifies that it “suspected” the detection of foreign object 20 but did not “confirm” it, and is therefore configured to identify this event as an improper warning or false positive alert. For this reason, the system 100 can be configured to save at least one of the warning signal, attribute, feature and image of this event to a database (e.g., a database of improper warnings) for post-event analysis and investigation.

[0063] Figure 8 is a flowchart showing a method 5000 for identifying foreign objects 20 on a runway. An image processing module 134M can be configured to perform method 5000. System 100 stores a plurality of reference feature vectors and the object categories associated with each of the plurality of reference feature vectors in a reference feature vector database. The reference feature vector database is stored in a storage device 140. To identify the foreign object 20, system 100 is configured to identify the object category of the foreign object 20. As shown in Figure 8, this method includes capturing a visible light image 120M and a thermal image 110M by a visible light camera 120 and a thermal camera 110 in block 5302. This method includes segmenting the thermal image 110M into a plurality of thermal image regions in block 5304 in order to identify the object category of the foreign object 20 in the thermal image 110M. This method includes segmenting the visible light image 120M into a plurality of visible light image regions in block 5304. Segmenting the thermal image 110M includes labeling each pixel in the thermal image 110M and grouping labeled pixels with the same characteristics into multiple groups to form multiple thermal image regions. Segmenting the visible light image 120M includes labeling each pixel in the visible light image 120M and grouping labeled pixels with the same characteristics into multiple groups to form multiple visible light image regions. The processor 132 assigns labels to each pixel in the thermal image 110M and visible light image 120M such that pixels with common specific characteristics or properties have the same label. The segmented thermal image 110M and visible light image 120M consist of multiple thermal image regions and multiple visible light image regions that collectively encompass each image. Pixels in each of the multiple regions are similar in some characteristic, feature, or property such as texture, color, or intensity. Adjacent regions in the multiple image regions differ significantly from each other in the same characteristics.The segmented thermal image 110M and visible light image 120M can be used to detect and identify regions containing suspicious foreign objects 20 within the images. In block 5306, this method detects and extracts features from the thermal image 110M and visible light image 120M. System 100 is configured to assign a feature vector (e.g., a thermal feature vector) to each of the multiple thermal image regions and a feature vector (e.g., a visible light feature vector) to each of the multiple visible light image regions. A feature refers to a pattern or unique structure present in the image, such as a point, clump, small patch, corner, or edge. Features are represented by image regions whose texture, color, intensity, etc., differ from the surrounding image regions. Features are extracted, grouped, and represented by feature vectors. The foreign object 20 can be represented by a group of features represented by feature vectors.

[0064] This method includes comparing a feature vector with multiple reference feature vectors in block 5308. Each of the multiple reference feature vectors is associated with an object category. System 100 matches the feature vector with the multiple reference feature vectors in block 5308. Each object category (e.g., rubber tire, mechanic's tools, aircraft part, vehicle part) can be represented by a specific reference feature vector stored in the reference feature vector database. Each extracted feature vector is matched with multiple reference feature vectors in the reference feature vector database. This method includes detecting foreign object 20 in block 5310. If the feature vector matches one or more of the multiple reference feature vectors, System 100 determines that foreign object 20 has been detected. System 100 generates a "suspected FOD" warning signal. This method identifies the object category of foreign object 20 in block 2312. System 100 is configured to identify the reference feature vector and its object category that is closest to the feature vector. System 100 identifies or classifies foreign objects 20 based on one or more of the matched reference feature vectors, specifically the "shortest distance" between the feature vector and a particular reference feature vector. The "shortest distance" may also be used to determine the match or the probability that the foreign object 20 is accurately classified. Multiple reference feature vectors may match the feature vector. Matching may be performed based on fuzzy matching. System 100 is configured to recognize and classify foreign objects 20 based on their object categories. System 100 identifies the object category of the foreign object 20 in the visible light image 120M. Based on the matched reference feature vectors, the object categories tagged with the matched reference feature vectors are retrieved, allowing for the identification or classification of the foreign object 20. Once the foreign object 20 is identified, System 100 generates and transmits a warning signal.

[0065] Figure 9 is a flowchart illustrating method 6000 for identifying foreign objects 20 on the runway. Method 6000 is the same as method 5000 in Figure 8, except that system 100 is configured to automatically detect and extract features from thermal images 110M and visible light images 120M, match the feature vectors with a plurality of reference feature vectors, and detect the foreign objects 20 in block 6306. In Figures 8 and 9, the same steps are denoted by the same reference numerals. System 100 can be configured to train an image processing module 134M using a deep learning module and to automatically execute the steps in block 6306.

[0066] The thermal camera 110 detects foreign objects 20 by detecting the difference in temperature, i.e., infrared thermal radiation, between the foreground (e.g., foreign objects 20) and the background (e.g., the surface of the runway 202). Foreign objects 20 of different categories or types are made of different materials such as metal, rubber, plastic, and concrete, and have different energy absorption rates, reflectances, and emissivity. Therefore, different categories of foreign objects 20 may result in different levels of temperature, i.e., infrared thermal radiation, relative to the background (i.e., the runway surface). The temperature difference between the foreign objects 20 and the runway can be detected by the thermal camera 110.

[0067] Therefore, it is beneficial to "train" the thermal camera 110 or thermal camera operating module 134T to distinguish between different categories of foreign objects 20 by identifying the type of material (e.g., rubber, metal, plastic, concrete, asphalt, etc.) that makes up the foreign object 20. Since foreign objects 20 made of different materials have different emissivity, and consequently different temperature levels and temperature contrast levels relative to the background, i.e., the runway, a "well-trained" thermal camera 110 can more accurately identify the foreign objects 20.

[0068] The thermal camera 110 is trained. Under normal sunny conditions, the thermal camera 110 undergoes an initial "training" period. During this training, the thermal camera 110 operates in "training" mode so that it can "learn" from the visible light image 102M of the visible light camera 120. After this initial "training," the thermal camera 110 has properly "learned" and can reliably and accurately detect foreign objects 20 with a relatively high level of accuracy. By increasing accuracy, it becomes possible to realize a system 100 with a "standalone" thermal camera 110 instead of using the visible light camera 120 and thermal camera 110 as a set. In this way, the system 100 can be applied even in bad weather or in situations with very poor visibility without requiring the visible light camera 120.

[0069] Figure 10 shows a flowchart of how to train the image processing module 134M to improve the identification of foreign objects 20 on the runway. In block 7110, the visible light camera 120 is configured to scan one of several sectors on the runway. The visible light camera 120 is configured to scan the sector sub-sector by subsector. The visible light camera 120 takes multiple visible light images 120M of the sector. The image processing module 134M processes the multiple visible light images 120M to detect the foreign object 20. In block 7210, the thermal camera 110 is configured to scan the same sector on the runway scanned by the visible light camera 120. The thermal camera 110 is configured to scan the sector sub-sector by subsector. The thermal camera 110 takes multiple thermal images 110M within the sector.

[0070] The image processing module 134M processes multiple thermal images 110M to detect the foreign object 20. The thermal camera 110 and the visible light camera 120 are configured to scan sectors simultaneously. In block 7120, the system 100 processes the visible light image 120M to identify the visible light object image 102B and then detects the foreign object 20. In block 7220, the system 100 processes the thermal image 110M to identify the thermal object image 110B and then detects the foreign object 20. The thermal image 110M and the visible light image 120M can be processed simultaneously by the processor 132. If the system 100 detects the foreign object 20 in the visible light image 120M, the system 100 generates a "suspected FOD" warning signal to notify the operator that the foreign object 20 has been detected in the visible light image 120M. Similarly, if system 100 detects a foreign object 20 in the thermal image 110M, system 100 generates a "suspected FOD" signal in block 7230 to notify the operator that a foreign object 20 has been detected in the thermal image 110M, since the detection of the foreign object 20 has not yet been verified. The "suspected FOD" warning signal is generated for both the visible light image 120M and the thermal image 110M. System 100 may display the thermal object image 110B and / or the visible light object image 120B on a display for the operator to visually inspect. System 100 generates at least one attribute of the visible light object image 120B in block 7130 and at least one attribute of the thermal object image 110B in block 7230.

[0071] At least one attribute includes the location of the visible light object image 120B in the visible light image 120M, the location of the thermal object image 110B in the thermal image 110M, the size of the visible light object image 120B, the size of the thermal object image 110B, and / or the temperature of the thermal object image 110B. For example, system 100 generates the location of the visible light object image 120B in the visible light image 120M and / or the size of the visible light object image 120B. For example, system 100 generates at least one of the location of the thermal object image 110B in the thermal image 110M, the size of the thermal object image 110B, and the temperature of the foreign object 20. In blocks 7132 and 7232, system 100 is configured to store at least one of the warning signal, attribute, feature, and image of this event in the foreign object warning signal and event database 742. In block 7140, system 100 is configured to determine whether or not foreign objects 20 are present in the visible light image 120M and the thermal image 110M by comparing at least one attribute of the visible light object image 120B and the thermal object image 110B. The method for comparing at least one attribute is shown in method 4140 in Figure 7. If the attributes of the visible light object image 120B and the thermal object image 110B match in block 7150, system 100 determines that foreign objects 20 have been detected in the visible light image 120M and the thermal image 110M. If foreign objects 20 are detected, system 100 generates and displays a warning signal in block 7170, for example, a "Confirmed FOD" warning signal. Otherwise, system 100 generates and transmits a "No Confirmed FOD" warning signal. In block 7172, system 100 is configured to store at least one of the signals, attributes, features, and images in the foreign object warning signal and event database 742. In block 7180, system 100 is configured to optimize the detection configuration parameters of the thermal camera 110. System 100 can optimize the detection configuration parameters based on foreign object warning signals and data stored in the event database 742.System 100 can run statistical analysis modules and / or optimization modules to optimize detection configuration parameters based on data. System 100 may use artificial intelligence to optimize detection configuration parameters based on data. In block 7182, System 100 is configured to store the optimized detection configuration parameters of the thermal camera 110 in the foreign object detection configuration parameter database 744 of the thermal camera 110.

[0072] System 100 determines the relationship between the object category of the foreign object 20 in the thermal image 110M and its temperature for all detected / verified foreign object samples. The processor 132 may further be configured to train the thermal camera 110 to detect foreign objects 20 based on the visible light image 120M from the visible light camera 120. Since it is much easier to identify and classify foreign objects 20 in the visible light image 120M, System 100 can determine the relationship between the object category of the visible light object image 120B acquired from the visible light camera 120 and the temperature of the thermal object image 110B from the thermal camera 110. Thus, if the size of the foreign object samples is sufficiently large, System 100 can determine the relationship between different foreign object categories (e.g., foreign objects made of different materials such as metal, plastic, and rubber) and their corresponding temperatures. System 100 then constructs a "FOD type thermal profile model" that can be used to map various types of foreign objects (i.e., foreign objects made of different materials) to their corresponding temperature ranges. In this way, the system can more easily identify the foreign object 20 based on the thermal object image 110B.

[0073] System 100 can identify foreign object categories or types using an "FOD type thermal profile model," and based on the temperature of the foreign object 20 detected by the thermal camera 110, it can identify the specific material type (metal, rubber, plastic, etc.) of the detected foreign object 20. The "FOD type thermal profile model" can be constructed using mathematical methods and / or statistical methods such as statistical correlation analysis. Alternatively, the "FOD type thermal profile model" may be constructed using artificial intelligence or machine learning techniques. This "FOD type thermal profile model" can be used to optimize the detection configuration parameters of the thermal camera 110.

[0074] To optimize the performance of the thermal camera 110, it is necessary to optimize its detection configuration parameters. The detection configuration parameters of the thermal camera 110 may be a set of operating parameters for the thermal camera 110 configured to enable the thermal camera 110 to detect foreign objects 20 with optimized and high accuracy. The operating parameters of the thermal camera 110 may include sensitivity, gain, brightness, contrast, shutter timing settings, etc. In this way, as the system 100 trains the thermal camera operating module 134T, the detection performance of the thermal camera 110 will improve over time, reaching a level where it can operate as the system 100's only "standalone" foreign object detector, that is, without the visible light camera 120. By optimizing the performance of the thermal camera 110, it becomes possible to function effectively even in bad weather or in situations with very poor visibility.

[0075] The thermal camera 110 can detect different levels of temperature contrast depending on the type of foreign object. Therefore, the thermal camera 110 can accurately detect the foreign object 20. As a result, the system 100 can classify or identify different categories or types of foreign objects 20 based on the different types of materials that make up the foreign object 20.

[0076] Database 742 stores information such as "suspected FOD," "confirmed FOD," and events that occurred using both the visible light camera 120 and the thermal camera 110. Database 742 also stores the attributes (category, size, location, temperature, etc.) of the detected and / or calculated foreign objects 20.

[0077] Figure 11 is a flowchart of method 8000 for detecting foreign objects 20 using a thermal camera 110. After "training," the thermal camera 110 can reliably and accurately detect foreign objects 20 with a relatively high level of accuracy. By establishing high accuracy, it becomes possible to use the thermal camera 110 without a visible light camera 120. As shown in Figure 11, the system 100 may be configured in block 8402 to detect foreign objects 20 using optimized detection configuration parameters of the thermal camera 110 stored in the detection configuration parameter database 844 for the thermal camera 110. The thermal camera 110 can be configured as the sole foreign object detector for detecting foreign objects 20.

[0078] Those skilled in the art will understand that the features described in one embodiment are not limited to that embodiment and can be combined with any of the other embodiments. The present invention relates to a system for detecting foreign objects on a runway and a method for such a system, as described herein with reference to and / or illustrated in the accompanying drawings.

Claims

1. A method for detecting foreign objects on a runway divided into multiple sectors, Multiple thermal images of the first view of the multiple sectors of the runway are taken from one side of the runway, The method involves capturing multiple visible light images of the first view of the multiple sectors of the runway from one side of the runway, wherein the first view overlaps with the second view. To detect thermal object images within the aforementioned plurality of thermal images, To detect visible light object images within the aforementioned plurality of visible light images, In one of the plurality of thermal images and one of the plurality of visible light images, if the thermal object image and the visible light object image are detected, it is determined that the foreign object has been detected. The process involves converting the plurality of visible light images and the plurality of thermal images into a plurality of converted visible light images and a plurality of converted thermal images, wherein the plurality of converted visible light images and the plurality of converted thermal images are cockpit view images, and the cockpit view is a view of the runway from the cockpit of an aircraft. The process involves combining the multiple converted visible light images to form a single visible light image, The process involves combining the multiple converted thermal images to form a single thermal image, A method comprising transmitting the single thermal image and the single visible light image to a display in the cockpit of the aircraft.

2. The method according to claim 1, wherein the first view includes a perspective view.

3. The method according to claim 1 or 2, wherein the second view includes a cockpit view as seen from the cockpit of an aircraft.

4. The method according to any one of claims 1 to 3, wherein the conversion of the plurality of visible light images includes rotating the plurality of visible light images.

5. The method according to any one of claims 1 to 4, wherein the conversion of the plurality of thermal images includes rotating the plurality of thermal images.

6. The method according to any one of claims 1 to 5, wherein converting the plurality of visible light images includes distorting the plurality of visible light images.

7. The method according to any one of claims 1 to 6, wherein the conversion of the plurality of thermal images includes distorting the plurality of thermal images.

8. The method according to any one of claims 1 to 7, wherein determining the foreign object includes generating at least one attribute of the foreign object in each of the thermal object image and the visible light object image, and comparing the at least one attribute of the foreign object in the thermal object image and the visible light object image, and detecting the foreign object if the at least one attribute of the foreign object in the thermal object image and the visible light object image are the same.

9. The method according to claim 8, wherein the at least one attribute of the foreign object includes the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image.

10. The method according to claim 9, wherein if the distance between the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image is within the position parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

11. The method according to any one of claims 8 to 10, wherein the at least one attribute of the foreign object includes the size of the thermal object image and the visible light object image.

12. The method according to claim 11, wherein if the difference between the size of the thermal object image in the thermal image and the size of the visible light object image in the visible light image is within the size parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

13. A system for detecting foreign objects on a runway divided into multiple sectors, Multiple camera sets arranged at intervals from each other, A thermal camera having a first field of view and configured to capture a thermal image of a first view of one sector of the runway from one side of the runway, A plurality of camera sets each having a visible light camera having a second field of view and configured to capture a visible light image of the first view of one of the plurality of sectors of the runway from one side of the runway, wherein the first field of view overlaps with the second field of view. A processor that communicates with the aforementioned multiple camera sets, The system includes a memory that communicates with the processor and stores instructions that can be executed by the processor, The aforementioned processor, Detecting thermal object images within multiple thermal images, Detecting visible light object images within multiple visible light images, In one of the plurality of thermal images and one of the plurality of visible light images, if the thermal object image and the visible light object image are detected, it is determined that the foreign object has been detected. The process involves converting the plurality of visible light images and the plurality of thermal images into a plurality of converted visible light images and a plurality of converted thermal images, wherein the plurality of converted visible light images and the plurality of converted thermal images are cockpit view images, and the cockpit view is a view of the runway from the cockpit of an aircraft. The process involves combining the multiple converted visible light images from the multiple camera sets to form a single visible light image. The process involves combining the multiple converted thermal images from the multiple camera sets to form a single thermal image. A system configured to transmit the single thermal image and the single visible light image to a display in the cockpit of the aircraft.

14. The system according to claim 13, wherein the first view includes a perspective view.

15. The system according to claim 13 or 14, wherein the processor is configured to rotate the visible light image in order to convert the visible light image.

16. The system according to any one of claims 13 to 15, wherein the processor is configured to rotate the thermal image in order to convert the thermal image.

17. The system according to any one of claims 14 to 16, wherein the processor is configured to distort the visible light image in order to convert the visible light image.

18. The system according to any one of claims 13 to 17, wherein the processor is configured to distort the thermal image in order to convert the thermal image.

19. The system according to any one of claims 13 to 18, wherein, in order to determine the foreign object, the processor is configured to generate at least one attribute of the foreign object in each of the thermal object image and the visible light object image, and to compare the at least one attribute of the foreign object in the thermal object image and the visible light object image, and the foreign object is detected if the at least one attribute of the foreign object in the thermal object image and the visible light object image are the same.

20. The system according to claim 19, wherein the at least one attribute of the foreign object includes the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image.

21. The system according to claim 20, wherein if the distance between the position of the thermal object image in the thermal image and the position of the visible light object image in the visible light image is within the position parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

22. The system according to any one of claims 19 to 21, wherein the at least one attribute of the foreign object includes the size of the thermal object image and the visible light object image.

23. The system according to claim 22, wherein if the difference between the size of the thermal object image in the thermal image and the size of the visible light object image in the visible light image is within the size parameter, then at least one attribute of the foreign object in the thermal object image and the visible light object image is the same.

Images