Foreign object detection method for airport runway based on multi-view image parallax

Abstract

The present invention discloses a method for detecting foreign objects on airport runways based on multi-view visual image parallax. The specific technical solution is: four high-pixel industrial cameras are used to observe the same detection area from different observation angles. When each camera observes the same detection area from different observation angles, when encountering foreign objects with three-dimensional characteristics, the foreign objects will present different shapes and produce different parallax images due to different observation angles. By processing this group of images to find the difference, the foreign objects can be detected. The present invention solves the shortcomings of the existing detection system based on millimeter wave radar technology, which has poor accuracy in detecting and identifying foreign objects and requires auxiliary detection means. The detection method of the present invention can achieve an identification accuracy of 5 mm, and the equipment cost and operation and maintenance cost are relatively low.

Description

Technical Field

[0001] The invention belongs to the technical field of airport safety protection and relates to an airport runway foreign body detection method based on multi-view visual image parallax. Background Art

[0002] Foreign Object Debris (FOD) on airport runways refers to any foreign matter that does not belong to the airport but appears in the airport area and may damage aviation instruments or aviation systems. According to statistics, FOD is mainly various scattered parts of aircraft, metal tools, stones, wood blocks and plastic products.

[0003] Manual detection is currently the most common FOD detection method. Its advantage is that it can detect millimeter-level FOD, but it requires a high level of comprehensive quality from the detection personnel. Since it is greatly affected by external conditions such as personal emotions, sense of responsibility, working environment, labor intensity, light intensity, etc., its overall reliability is not high.

[0004] Among the FAA-certified FOD systems in the world, three FOD detection systems (Tarsier, FODetect, and FODFinder) are based on millimeter-wave radar technology. Although the FOD detection system based on millimeter-wave radar is not affected by daytime, nighttime, and visibility, it still has the following limitations: ① The accuracy of detecting and identifying foreign objects is still poor, and it can only identify foreign objects of 5 cm and above, and it is still unable to identify foreign objects of millimeter level (such as lead bullets, etc.); ② Video detection is required as an auxiliary detection method; ③ The equipment cost is high, and the operation and maintenance cost is also high. Summary of the invention

[0005] The purpose of the present invention is to provide an airport runway foreign object detection method based on multi-view image parallax, which solves the three major problems existing in the prior art: the recognition accuracy is affected by the visibility during the day and night, the foreign objects at the millimeter level cannot be recognized, and the cost is high.

[0006] The specific technical solutions of the present invention are as follows:

[0007] A method for detecting foreign objects on airport runways based on multi-view visual images uses four high-pixel industrial cameras to observe the same detection area from different observation angles. When each camera observes the same detection area from a different observation angle, when encountering foreign objects with three-dimensional characteristics, the foreign objects will present different shapes and produce different parallax images due to different observation angles. By processing this group of images to find the difference, the foreign objects can be detected.

[0008] The specific steps include:

[0009] Step 1: Through four cameras a i, i∈(1,2,3,4) takes pictures of the detection area and records the pictures taken by each of them as I i ,i∈(1,2,3,4), each image is converted into a grayscale image and preprocessed with histogram averaging. After processing, each image is represented by f i (x,y) Indicates that three cameras α1, α2, and α3 are arranged in a horizontal equilateral triangle, and the fourth camera α4 is arranged at a bird's-eye view angle;

[0010] Step 2: Based on the linear spatial relationship between the four images, the perspective transformation between the images is performed. The matching between feature point pairs and the random consistent matching algorithm are combined to establish the perspective transformation matrix. The images f1(x, y), f2(x, y), and f3(x, y) taken at the equilateral triangle are transformed into the image ff observed from the top-down angle of the image f4(x, y). 1 (x,y),ff 2 (x,y),ff 3 (x,y);

[0011] Step 3: When there is a three-dimensional foreign object in the detection area, different shape information will be obtained by observing the same foreign object from different observation angles. The final detection result can be obtained by fusing the different information of each image, and the detection of the foreign object can be finally completed.

[0012] In step 1, the method of converting each image into a grayscale image and performing histogram averaging preprocessing is as follows: i , i∈(1,2,3,4) is Gaussian blurred to obtain a new set of images, and then each color image in this set of color images is decomposed into RGB three-channel brightness images denoted as R (x,y) ,G (x,y) ,B (x,y) , through f i (x,y) =0.299*R (x,y) +0.587*G (x,y) +0.114*B (x,y) Convert to grayscale image f i (x,y) .

[0013] In step 2, we first need to get the image f from step 1 i (x,y) The SIFT algorithm is used to extract feature points and each camera a i All extreme points in the captured image are described separately to form their own feature point set D i (i∈(1,2,3,4)).

[0014] In step 2, the method of perspective transformation between images is as follows:

[0015] <1> Feature point matching:

[0016] The image taken by the a4 camera is used as the sample image. The feature point descriptors are used to find the coordinates of the feature points that match the feature point set in the a4 image from the a1, a2, and a3 images respectively, and recorded as s i4 And i∈(1,2,3);

[0017] <2> Build an affine transformation matrix model:

[0018] ① Assume that the affine transformation model is as shown in Equation 3.1, and randomly select s i4 Extract 4 sample points and fit the model;

[0019]

[0020]

[0021] area(m1)-area(m2)<=ε (3.1)

[0022] In formula 3.1, N represents the number of pixels in the m1 region, M represents the number of pixels in the q1 region, f(x, y) represents the grayscale value of the pixel, and ε represents the threshold.

[0023] ② Assuming the tolerance range is: sigma, find the points within the tolerance range of the distance fitting curve and count the number of points;

[0024] ③ Randomly re-select from s i4 Select 4 points and repeat the steps from step 1 to step 2 until the iteration ends;

[0025] ④After each fitting, there are corresponding data points within the tolerance range. Find the case with the largest number of data points as the final model result;

[0026] Through the above steps to s i4 ,i∈(1,2,3) three sets of feature point pairs respectively establish three affine transformation models H i ; After obtaining the perspective transformation model of the corresponding camera image, a can be converted into i i∈(1,2,3) is the image f captured by the camera i (x,y) Transform to the image ff taken from the perspective of the a4 camera i (x,y) ,i∈(1,2,3);

[0027]

[0028] in fi (x,y) The position of the image pixel, is the position of the pixel after affine transformation, is the perspective transformation matrix H.

[0029] In step three, the specific method of fusing different information of each image is:

[0030] ① Through the fusion of the image brightness mean MoI disparity mean map, the canny edge detection operator is used to extract the outer and inner contours of the "virtual ring" through the gradient transformation degree between pixels, and the foreign body area is divided into two areas according to the contour, the solid area m1 and the virtual ring area q1. The average grayscale difference between the two areas is used to determine whether there is a foreign body here;

[0031] ②Introducing the image brightness deviation DoI fusion image to eliminate external interference factors, this feature can be amplified by dilation and then corrosion to achieve the effect of foreign body detection;

[0032] ③ Compare and combine the foreign objects detected in the two images to obtain the final detection result.

[0033] The technical solution of the present invention has the following beneficial effects:

[0034] During the detection process, four high-pixel industrial cameras are mainly used. When each camera observes the same detection area from different observation angles, when encountering foreign objects with three-dimensional characteristics, the foreign objects will present different shapes due to the different observation angles, thereby generating images with different parallaxes. By processing this group of images to find the difference, the foreign objects can be detected. The detection method of the present invention can achieve a recognition accuracy of 5 mm, and the equipment cost and operation and maintenance cost are low. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the perspective transformation principle of a method for detecting foreign objects on an airport runway based on multi-view image parallax according to the present invention;

[0036] Figure 2 This is an algorithm flow chart of an airport runway foreign object detection method based on multi-view image parallax of the present invention;

[0037] Figure 3 A schematic diagram of a model of a method for detecting foreign objects on an airport runway based on multi-view image parallax according to the present invention;

[0038] Figure 4 A Gaussian pyramid schematic diagram of a method for detecting foreign objects on an airport runway based on multi-view image parallax according to the present invention;

[0039] Figure 5A schematic diagram of a Gaussian difference pyramid structure of an airport runway foreign object detection method based on multi-view visual image parallax according to the present invention;

[0040] Figure 6 It is a schematic diagram of comparing local extreme value points of a method for detecting foreign objects on an airport runway based on multi-view image parallax according to the present invention;

[0041] Figure 7 It is a schematic diagram of an extreme point descriptor of an airport runway foreign object detection method based on multi-view image parallax of the present invention;

[0042] Figure 8 A schematic diagram of image brightness mean of a method for detecting foreign objects on an airport runway based on multi-view image parallax according to the present invention;

[0043] Fig. 9 A schematic diagram of image brightness deviation of a method for detecting foreign objects on an airport runway based on multi-view image parallax according to the present invention;

[0044] Fig.10 A schematic diagram of image brightness deviation angle of a method for detecting foreign objects on an airport runway based on multi-view image parallax according to the present invention;

[0045] Fig.11 The figure is a schematic diagram of a virtual ring of a foreign object in a method for detecting foreign objects on an airport runway based on multi-view image parallax according to the present invention.

[0046] Fig.12 This is a schematic diagram of the airport runway foreign object detection method based on multi-view image parallax after corrosion and expansion processing of the present invention.

[0047] Fig.13 The figure is a schematic diagram of camera shooting of a method for detecting foreign objects on an airport runway based on multi-view image parallax according to the present invention.

[0048] Fig.14 The figure is a schematic diagram of the detection results of a method for detecting foreign objects on an airport runway based on multi-view image parallax according to the present invention. DETAILED DESCRIPTION

[0049] In order to make the present invention feasible and the superiority of the technical solution more convincing, the important steps of the implementation of this invention will be explained in detail and clearly in combination with the drawings in the experimental process of this invention. The described example steps are part of this invention, not a complete description. It is usually to prove the feasibility and authenticity of the present invention to everyone. Therefore, the following explanation of the invention example drawings provided is conducive to a deep understanding of the example. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0050] Example

[0051] This embodiment uses 4 high-pixel industrial network port cameras, a steel frame, a gigabit switch, a portable 220V mobile power supply, a CPU processor and a display.

[0052] The specific construction steps are as follows:

[0053] <1> Use steel frame to build a Figure 3 The detection framework shown in the model fixes four industrial network port cameras at the three corners of an equilateral triangle and the center of gravity of the equilateral triangle. The camera at the center of gravity shoots vertically downward, and the camera at the triangle shoots at a 60-degree angle to the horizontal plane along the triangle angle bisector.

[0054] <2> Connect the 4 cameras to the switch and then to the CPU processor through the switch, and display the images processed by the CPU on the monitor. The whole system is powered by a 220V portable mobile power supply.

[0055] <3> After debugging the system, three universal wheels are installed at the bottom of the model to achieve the ultimate goal of real-time mobile detection.

[0056] Figure 3 This is the physical model that needs to be established for the detection method of the present invention, and the following foreign body detection process is implemented based on this model. Based on the above given model, the general process of the present invention is described as follows:

[0057] The steps of the present invention mainly include: preprocessing of image groups, registration and affine transformation between images, and fusion detection between images.

[0058] 1) Preprocessing of image groups:

[0059] ①According to Figure 1 As shown, when the model is used to detect foreign objects in a certain area, the images taken by the four cameras α1, α2, α3, and α4 are recorded as I1, I2, I3, and I4 and used as a group of data images.

[0060] ② Perform Gaussian blur on images I1, I2, I3, and I4 to obtain a new set of images, and then decompose each color image in this set of color images into RGB three-channel brightness images, denoted as R (x,y) ,G (x,y) ,B (x,y) , converted into a grayscale image f through formula 1.1 i (x,y) .

[0061] f i (x,y) =0.299*R (x,y) +0.587*G(x,y) +0.114*B (x,y) (1.1)

[0062] 2) Image registration and affine transformation

[0063] For the set of images after preprocessing i (x,y) The SIFT algorithm is used to extract feature points.

[0064] Specific steps for feature point extraction:

[0065] <1> The original image f i (x,y) Continuously downsampling, a series of images of different sizes are obtained, which form a tower model from top to bottom and from large to small, and the original image is used as the first layer of the tower model. The image of each layer is represented by f j i (x,y)(j∈(1,2,....,N)) represents.

[0066] <2> Through formula 2.1 Convolve G(x,y,σ) with different Gaussian convolution kernels to build a Gaussian pyramid such as Figure 4 shown.

[0067]

[0068] <3> The images generated by convolution of Gaussian kernels with different standard deviations are differentiated by equation 2.2 to obtain the Gaussian difference pyramid, as shown in Figure 5 shown.

[0069]

[0070] f j i (x, y) represents the j-th layer image of the ith camera, G(x, y, kσ) represents the Gaussian kernel with a standard deviation of kσ

[0071] <4> Gaussian Difference Pyramid Local Extrema Detection

[0072] The feature points are composed of local extreme points in the Gaussian difference pyramid space. To find the extreme points of the difference pyramid function, each pixel needs to be compared with all its adjacent points, such as Figure 6 shown.

[0073] <5> Extreme point direction assignment

[0074] The local features in the image are used to assign a reference direction to the corresponding extreme points, so that the descriptor is invariant to image rotation. The gradient and direction features of the pixels in the 3σ domain window of the Gaussian pyramid image are collected mainly through formula (2.3).

[0075]

[0076] After completing the gradient calculation of the extreme point, use the histogram to count the gradient and direction of the pixels in a certain range, take the maximum value of the histogram as the main direction of the key point, and retain the direction with a peak value greater than 80% of the main direction as the auxiliary direction of the key point, such as Figure 7 shown.

[0077] <6> Feature point description

[0078] Each extreme point has three pieces of information: position, scale, and direction. A descriptor is created for each extreme point, and a set of vectors is used to describe the extreme point. i All extreme points in the captured image are described separately to form their own feature point set D i (i∈(1,2,3,4)).

[0079] 3) Image registration and affine transformation

[0080] <1> Feature point matching

[0081] The image taken by the a4 camera is used as the sample image. The feature point descriptors are used to find the coordinates of the feature points that match the feature point set in the a4 image from the a1, a2, and a3 images respectively, and recorded as s i4 And i∈(1,2,3).

[0082] <2> Establishing the affine transformation matrix model Due to the shortcomings of the SIFT algorithm itself, the matching of feature points cannot be completely correct, so it is necessary to find suitable feature point pairs and establish three affine transformation models respectively. The specific steps are as follows:

[0083] ⑤ Assume that the affine transformation model is as shown in Equation 3.1, and randomly select s i4 Extract 4 sample points and fit the model;

[0084] ⑥ Assuming the tolerance range is: sigma, find the points within the tolerance range of the distance fitting curve and count the number of points;

[0085] ⑦ Randomly re-select from s i4 Select 4 points and repeat the steps from step 1 to step 2 until the iteration ends.

[0086] ⑧After each fitting, there are corresponding data points within the tolerance range. Find the case with the largest number of data points as the final model result.

[0087] Through the above steps, we can give s i4,i∈(1,2,3) three sets of feature point pairs respectively establish three affine transformation models H i .

[0088] After obtaining the perspective transformation model of the image captured by the camera, a can be transformed into i i∈(1,2,3) is the image f captured by the camera i (x,y) Transform to the image ff taken from the perspective of the a4 camera i (x,y) ,i∈(1,2,3).

[0089]

[0090] in f i (x,y) The position of the image pixel, is the position of the pixel after affine transformation, is the perspective transformation matrix H.

[0091] 3) Image fusion and foreign body detection

[0092] <1> Fusion between images

[0093] After processing i (x,y) ,i∈(1,2,3), according to the principle that people can identify three-dimensional foreign objects through different visual differences between the two eyes, the images are further fused. Since the three cameras do not observe the same target from the same observation point, even if the images taken by the three cameras are scale-invariantly transformed through the perspective transformation matrix, there will be certain differences between the three images. Therefore, the three fusion algorithms of image brightness mean, brightness deviation and brightness deviation angle can accurately detect the presence of foreign objects in the airport.

[0094] The specific fusion steps are as follows:

[0095] ① Mean image brightness (MoI)

[0096] The image brightness mean (Formula 4.2) is physically the brightness mean of the three images, which shows the degree of non-overlap of the basic images in the target area and provides a reference for the deviation degree and deviation angle, such as Figure 8 shown.

[0097]

[0098] ② Image brightness deviation (DoI)

[0099] The image brightness deviation (Formula 4.3) can describe the physical meaning of the foreign matter relative to the average image brightness of the three images. It reflects the difference between the three images for the normalized total light intensity, and can better reflect the edge situation of the specific volume foreign matter, such as Fig. 9 shown.

[0100]

[0101] ③Image brightness deviation angle (DoO)

[0102] The image brightness deviation angle (Formula 4.4) can describe the angle information of the edge of the foreign object in physical meaning. By calculating the arc tangent value of the interpolation ratio of the images collected by two different cameras, the angle of the edge of the foreign object in the DoI relative to the coordinate axis can be distinguished, proving that the edge of the foreign object is a whole, such as Fig.10 shown.

[0103]

[0104] By combining the three algorithms of image brightness mean, image brightness deviation and image brightness deviation angle, the presence of foreign objects in the area can be quickly detected.

[0105] <2> Foreign object detection in fused images

[0106] It can be seen from the MOI disparity mean map obtained by fusion that when there is a foreign object somewhere, due to the different observation angles, each image saves different shape information of the foreign object, so that the shape information of the foreign object cannot overlap after the final fusion, resulting in a "virtual ring", such as Fig.11 shown.

[0107] Based on this characteristic, this algorithm uses the canny edge detection operator to extract the outer and inner contours of the "virtual ring" through the gradient transformation between pixels, and divides the foreign body area into two areas m1 (solid area) and q1 (virtual ring area) according to the contour. The average grayscale difference between the two areas is used to determine whether there is a foreign body here, as shown in Formula 3.1.

[0108]

[0109] In formula 3.1, N represents the number of pixels in the m1 region, M represents the number of pixels in the q1 region, f(x, y) represents the grayscale value of the pixel, and ε represents the threshold.

[0110] Then, by introducing the fused image DoI, some external interference factors are eliminated, making the final detection result more accurate. From the fused image DoI, it can be seen that the fused image reflects the different edge shapes of foreign objects in images with different observation angles. This characteristic can be amplified by first dilating and then corroding to achieve the effect of foreign object detection. Fig.12shown.

[0111] Finally, the foreign objects detected by the two images are compared and combined to obtain the final detection result.

[0112] Figures 13 to 14 The real object image and the final detection result are given.

[0113] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting foreign objects on airport runways based on multi-view images, characterized in that: Use four high-pixel industrial cameras to observe the same inspection area from different observation angles. When each camera observes the same inspection area from different observation angles, when encountering foreign objects with three-dimensional characteristics, the foreign objects will present different shapes and produce different parallax images due to different observation angles. By processing this group of images to find the differences, foreign objects can be detected; The specific steps include: Step 1: Through four cameras a i , i∈(1,2,3,4) takes pictures of the detection area and records the pictures taken by each of them as I i ,i∈(1,2,3,4), each image is converted into a grayscale image and preprocessed with histogram averaging. After processing, each image is represented by f i (x,y) Indicates that three cameras α1, α2, and α3 are arranged in a horizontal equilateral triangle, and the fourth camera α4 is arranged at a bird's-eye view angle; Step 2: Based on the linear spatial relationship between the four images, the perspective transformation between the images is performed. The matching between feature point pairs and the random consistent matching algorithm are combined to establish the perspective transformation matrix. The images f1(x, y), f2(x, y), and f3(x, y) taken at the equilateral triangle are transformed into the image ff observed from the top-down angle of the image f4(x, y). 1 (x,y),ff 2 (x,y),ff 3 (x,y); The method of perspective transformation between the images is specifically as follows: <1> Feature point matching The image taken by the a4 camera is used as the sample image. The feature point descriptors are used to find the coordinates of the feature points that match the feature point set in the a4 image from the a1, a2, and a3 images respectively, and recorded as s i4 And i∈(1,2,3); <2> Building an affine transformation matrix model ① Assume that the affine transformation model is as shown in Equation 3.1, and randomly select s i4 Extract 4 sample points and fit the model; In formula 3.1, N represents the number of pixels in the m1 region, M represents the number of pixels in the q1 region, f(x, y) represents the grayscale value of the pixel, and ε represents the threshold; ② Assuming the tolerance range is: sigma, find the points within the tolerance range of the distance fitting curve and count the number of points; ③ Randomly re-select from s i4 Select 4 points and repeat the first step to the second step until the iteration ends; ④After each fitting, there are corresponding data points within the tolerance range. Find the case with the largest number of data points as the final model result; Through the above steps to s i4 ,i∈(1,2,3) three sets of feature point pairs respectively establish three affine transformation models H i ; After obtaining the perspective transformation model of the image captured by the camera, a can be transformed into i i∈(1,2,3) is the image f captured by the camera i (x,y) Transform to the image ff taken from the perspective of the a4 camera i (x,y) ,i∈(1,2,3); in f i (x,y) The position of the image pixel, is the position of the pixel after affine transformation, is the perspective transformation matrix H; Step 3: When there is a three-dimensional foreign object in the detection area, different shape information will be obtained by observing the same foreign object from different observation angles. The final detection result can be obtained by fusing the different information of each image, and the detection of the foreign object can be finally completed.

2. The method for detecting foreign objects on an airport runway based on multi-view images according to claim 1, characterized in that: In step 1, the method of converting each image into a grayscale image and performing histogram averaging preprocessing is as follows: i , i∈(1,2,3,4) is Gaussian blurred to obtain a new set of images, and then each color image in this set of color images is decomposed into RGB three-channel brightness images denoted as R (x,y) ,G (x,y) ,B (x,y) , through f i (x,y) =0.299*R (x,y) +0.587*G (x,y) +0.114*B (x,y) Convert to grayscale image f i (x,y) .

3. The method for detecting foreign objects on an airport runway based on multi-view images according to claim 1, characterized in that: In step 2, we first need to get the image f from step 1 i (x,y) The SIFT algorithm is used to extract feature points and each camera a i All extreme points in the captured image are described separately to form their own feature point set D i (i∈(1,2,3,4)).

4. The method for detecting foreign objects on an airport runway based on multi-view images according to claim 1, characterized in that: ① Through the fusion of the MOI disparity mean map, the canny edge detection operator is used to extract the outer and inner contours of the "virtual ring" through the gradient transformation degree between pixels, and the foreign body area is divided into two areas according to the contour, the solid area m1 and the regional virtual ring area q1. The average grayscale difference between the two areas is used to determine whether there is a foreign body here; ②Introducing the image brightness deviation DoI fusion image to eliminate external interference factors, this feature can be amplified by dilation and then corrosion to achieve the effect of foreign body detection; ③ Compare and combine the foreign objects detected in the two images to obtain the final detection result.

Images