An airport runway foreign matter detection method based on optical image analysis

Abstract

The application discloses an airport runway foreign matter detection method based on optical image analysis. The method is characterized in that: edge processing is performed on the photographed airport runway image, the distribution of the edge information of the image is calculated, and the image area where the runway is located is determined; then, based on the area where the runway is located, a straight line equation at the edge of the runway is calculated by using runway edge fitting, a runway edge fitting straight line is obtained, and the runway is segmented; the segmented runway image is processed by using AC saliency detection and OSTU threshold segmentation, and the position and size information of the foreign matter are saved; based on the above information, the corresponding area image is extracted after being enlarged in proportion, the extracted image is processed by using a K-Means algorithm, and image binarization processing is performed based on the image pixel information of the center position; finally, by comparing the detection information, combining the straight line equation of the runway edge, and eliminating the interference, the automatic detection of the airport runway foreign matter is realized.

Description

Technical Field

[0001] This invention relates to a method for detecting foreign objects on airport runways, and more particularly to a method for detecting foreign objects on airport runways based on optical image analysis. Background Technology

[0002] Foreign objects (FOD) on airport runways are foreign materials that can damage aircraft, such as scattered aircraft parts, metal tools, concrete blocks, and baggage components. FOD on airport runways can cause damage to aircraft and even lead to safety accidents with potentially disastrous consequences.

[0003] Currently, foreign object detection on airport runways mainly relies on manual patrols and close-range detection. This method increases the workload of staff, affects the efficiency of runway use, and is not timely enough, posing certain safety hazards. Therefore, more and more manufacturers are starting to use radar and optical imaging methods for foreign object detection on airport runways to ensure safe airport operations.

[0004] Currently, radar detection methods suffer from high equipment costs and maintenance difficulties. To reduce costs and provide more intuitive detection of foreign objects on runways, some manufacturers have begun to use optical imaging for foreign object detection on airport runways. However, most of these methods rely on fixed scenes combined with historical image information for foreign object detection, which is easily affected by weather, camera shake, and changes in lighting conditions. This results in poor robustness, missed detections, and a high false detection rate in poor weather or lighting conditions, failing to meet the engineering requirements of actual field operations. Summary of the Invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a method for detecting foreign objects on airport runways based on optical image analysis, which addresses the shortcomings of the existing technology.

[0006] To address the aforementioned technical problems, this invention discloses a method for detecting foreign objects on airport runways based on optical image analysis, comprising the following steps:

[0007] Step 1: Capture an image of the airport runway, perform edge processing on the airport runway image, calculate the distribution of its edge information, and determine the image region where the runway is located.

[0008] Step 2: Based on the image region where the runway is located, use runway edge fitting to calculate the equation of the straight line at the runway edge, obtain the fitted straight line at the runway edge, segment the runway, and obtain the segmented runway image;

[0009] Step 3, Initial detection of foreign objects: The segmented runway image is processed using AC saliency detection and OTU threshold segmentation to save the location and size information of the foreign objects;

[0010] Step 4, perform secondary detection of foreign objects: Based on the location and size information of the foreign object obtained in Step 3, after scaling up the image, extract the image of the corresponding area of ​​the foreign object, process the extracted image using the K-Means method, and perform threshold segmentation based on the image information of the center position;

[0011] Step 5: Compare the detection results of the two foreign objects, combine them with the straight line equation at the runway edge, eliminate interference, and complete the airport runway foreign object detection based on optical image analysis.

[0012] Furthermore, the edge processing of the airport runway image described in step 1 specifically includes:

[0013] Step 1-1: Use the Canny algorithm to perform edge detection on the airport runway image img_src and obtain the result image img_canny containing edge information. The size of the airport runway image img_src is M pixels wide and N pixels high.

[0014] Step 1-2: Perform morphological processing on the result image img_canny containing edge information obtained in step 1-1, performing opening and closing operations twice in sequence to generate the result binary image img_result.

[0015] Steps 1-3: Define the image region range of the binary image `img_result` from Steps 1-2. The upper region is taken from rows 0 to m of the image, and the lower region is taken from row n to the last row (row N). The number of columns is M. Count the number of pixels with a value of 255 in both the upper and lower regions, and calculate their respective proportions within the statistical regions. The specific method is as follows:

[0016]

[0017]

[0018] Where P(i,j) is the pixel value corresponding to the position of the i-th row and j-th column of the binary image img_result, which is a binary image, and the value is 0 or 255. rateup is the scaling factor of the upper half region and ratedown is the scaling factor of the lower half region.

[0019] Steps 1-4: Using the upper half region scaling factor (rateup) and lower half region scaling factor (ratedown) calculated in Steps 1-3, determine the area of ​​the runway in the image.

[0020] Furthermore, the specific determination method for identifying the area of ​​the runway in the image as described in steps 1-4 is as follows:

[0021]

[0022] Where rateval is the set scale coefficient threshold, and L is the region marker of the runway, represented as follows: UP means the runway is located in the entire upper half of the image, and there is an area outside the runway in the lower half of the image; MID means the runway is located in the middle of the image, and there is an area outside the runway in both the upper and lower halves of the image; DOWN means the runway is located in the entire lower half of the image, and there is an area outside the runway in the upper half of the image; ALL means the runway covers the entire image area, and there is no area outside the runway.

[0023] Furthermore, the segmentation of the runway described in step 2 specifically includes:

[0024] Step 2-1: Analyze the pixel distribution of the binary image img_result generated in Step 1-2, and generate the first position information table LocalUp and the second position information table LocalDown accordingly.

[0025] Step 2-2: Use the least squares method to perform linear fitting on the location point data in the first location information table LocalUp and the second location information table LocalDown generated in Step 2-1, respectively, to obtain two linear equations:

[0026] y = k up x+b up

[0027] y = k down x+b down

[0028] Where y is the column coordinate of the airport runway image, x is the row coordinate of the airport runway image, and k up k down b up and b down The coefficient values ​​of the equation obtained by fitting the data to a straight line;

[0029] Step 2-3: Combining the runway area markers obtained in Step 1-4, and using the straight line equations obtained in Step 2-2, solve for the straight line equations of the upper and lower edges of the runway in the airport runway image. The calculation method is as follows:

[0030] Equation of the straight line at the upper edge of the runway:

[0031]

[0032] Equation of the straight line at the lower edge of the runway:

[0033]

[0034] Step 2-4: Based on the equation of the runway edge line obtained in Step 2-3, solve for the four intersection points of the upper and lower edge lines of the runway with the left and right edges of the image, which are (0, y1), (M, y2), (0, y3), and (M, y4). Then, solve for the minimum value ymin and the maximum value ymax among y1, y2, y3, and y4.

[0035] Steps 2-5: Define a rectangular region, with the upper left position point as (0, ymin) and the lower right position point as (M, ymax). Extract the corresponding rectangular region from the airport runway image img_src in Step 1-1 to complete the segmentation of the runway region and generate the runway region segmentation image img_runway.

[0036] Furthermore, the specific method for generating the first location information table LocalUp and the second location information table LocalDown as described in step 2-1 is as follows:

[0037] When the pixel value at position (x,y) of the resulting binary image `img_result` is 255, where x is the row position of `img_result` and y is the column position of `img_result`, the number of pixels with values ​​of 255 and 0 in positions (x-1,y) to (x-index,y) of the resulting binary image `img_result` is counted as `num_up_white` and `num_up_black`, respectively. Simultaneously, the number of pixels with values ​​of 255 and 0 in positions (x+1,y) to (x+index,y) of the resulting binary image `img_result` is counted as `num_down_white` and `num_down_black`, respectively, where `index` is a preset statistical range. The percentage of pixels in each range is calculated using the following formula:

[0038] rate whiteup =num_up_white / index

[0039] rate blackup =num_up_black / index

[0040] rate whitedown =num_down_white / index

[0041] rate blackdown =num_down_black / index

[0042] Among them, rate whiteup and rate blackup These represent the percentages of pixels with values ​​of 255 and 0 in the `img_result` array at positions (x-1, y) to (x-index, y), respectively. (rate)whitedown and rate blackdown This represents the percentage of pixels with values ​​of 255 and 0 in the range (x+1,y) to (x+index,y) of the img_result.

[0043] When rate whiteup >val and rate blackdown When >val, place the position point (x,y) into the first position information table LocalUp;

[0044] When rate blackup >val and rate whitedown When >val, place the position point (x,y) into the second position information table LocalDown.

[0045] Furthermore, the initial detection of foreign objects described in step 3 specifically includes:

[0046] Step 3-1: Process the runway region segmentation image img_runway generated in Step 2-5 using the AC saliency detection algorithm to obtain the AC image img_ac;

[0047] Step 3-2: Perform OTU thresholding on the AC image img_ac from step 3-1 to obtain the OTU image img_ostu;

[0048] Step 3-3: Use median filtering to denoise the OTU image img_ostu from step 3-2 to obtain the denoised image img_blur;

[0049] Steps 3-4: Dilate the denoised image img_blur outward by n1 pixels to obtain the dilated image img_dilate, and then erode the image img_dilate inward by n1 pixels to obtain the eroded image img_result.

[0050] Steps 3-5 involve detecting and locating foreign objects in the erosion image img_result obtained in Step 3-4 through contour lookup, and obtaining preliminary scale and location information rect_ac of the foreign objects in the runway region segmentation image.

[0051] Furthermore, the secondary detection of foreign objects described in step 4 specifically includes:

[0052] Step 4-1: Based on the preliminary scale and location information rect_ac of the foreign object in the runway area segmentation image obtained in Step 3-5, set the target area range rect_roi, take the center position of the foreign object as the center of the rectangle, and set the width and height of the rectangle to be n2 times the width and height of the foreign object, respectively.

[0053] Step 4-2: Extract and segment the runway area segmentation image img_runway generated in Step 2-5 according to the target area rect_roi set in Step 4-1 to obtain the segmented image img_roi;

[0054] Step 4-3: Process the segmented image img_roi obtained in Step 4-2 using the K-Means algorithm to obtain the segmented image img_kmeans;

[0055] Step 4-4: Query the pixel value at the center position of the segmented image img_kmeans in Step 4-3, and set the pixel values at the positions with the same pixel value in this image img_kmeans to 255 and those with different pixel values to 0 to generate the binary image img_bina;

[0056] Step 4-5: Detect and locate foreign objects in the binary image img_bina obtained in Step 4-4 through contour finding, and obtain the secondary scale and position information rect_kmeans of the foreign objects in the image.

[0057] Further, the interference elimination described in Step 5 specifically includes:

[0058] Step 5-1: Calculate the IOU intersection over union rate of the two detection targets of the preliminary scale and position information rect_ac obtained in Step 3-5 and the secondary scale and position information rect_kmeans obtained in Step 4-5 iou ; and calculate its area ratio rate area = maxarea / minarea, where maxarea is the maximum area of the two detection targets and minarea is the minimum area. At the same time, calculate the distance between the center points of the two targets. When rate iou > iouval, rate area < areval, and distance < disval, it is determined that the detection result meets the requirements; otherwise, it is determined as interference and eliminated, where iouval, areval, and disval are preset thresholds;

[0059] Step 5-2: Determine the relationship between the position of the detection result that meets the requirements in Step 5-1 and the upper and lower runway edge lines solved in Step 2-3. When it is located between the two edge lines, it is determined as a real foreign object; otherwise, it is interference and eliminated.

[0060] Further, the proportional coefficient threshold rateval described in Step 1-4 takes a value of 0.15.

[0061] Further, the preset statistical range index described in Step 2-1 takes a value of 50.

[0062] Beneficial effects:

[0063] This invention solves the problems of high false detection rate, high missed detection rate, and low robustness in runway foreign object detection based on photoelectric images. First, edge processing is performed on the image to statistically analyze the distribution patterns of edge information and determine the image region where the runway is located. Based on the runway region, the least squares method is used to fit the runway edge line using the edge pixel distribution patterns, achieving runway region image segmentation. Next, AC saliency detection and OTU threshold segmentation are performed on the segmented runway image to complete the initial detection and localization of foreign objects. Then, based on the initial detection and localization information, the area where the foreign object is located is proportionally enlarged, and the image corresponding to the enlarged area is extracted. K-Means clustering is used to segment the image, and binarization is performed based on the center pixel information to achieve secondary detection of foreign objects. Finally, the foreign object information detected in the initial and secondary detections is compared, interference information is removed, and the runway edge line equation is combined to determine the area where the foreign object is located. False detection information is then eliminated, achieving automatic detection of runway foreign objects in optical images. The invention has undergone performance testing in various scenarios and can correctly detect and locate foreign objects in images under conditions such as darkness and strong light. The average detection time for each image is 100ms, which fully verifies the effectiveness of the invention. Attached Figure Description

[0064] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0065] Figure 1 This is a schematic diagram of the system flow of the present invention.

[0066] Figure 2 This is a schematic diagram of the image area where the runway of the present invention is located.

[0067] Figure 3 This is a diagram illustrating the runway area segmentation effect of the present invention.

[0068] Figure 4 This is a flowchart and effect diagram of the runway foreign object detection process of the present invention. Detailed Implementation

[0069] This invention discloses a method for detecting foreign objects on airport runways based on optical image analysis, comprising the following steps:

[0070] Step 1: Perform edge processing on the captured airport runway image, calculate the distribution of its edge information, and determine the image region where the runway is located.

[0071] Step 2: Based on the area where the runway is located, use runway edge fitting to calculate the equation of the straight line at the runway edge, obtain the fitted straight line at the runway edge, and divide the runway.

[0072] Step 3: The segmented runway image is processed using AC saliency detection (ACHANTA R, HEMAMI S, ESTRADAF, et al. Frequency-tuned salient region detection[C]. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, USA, 2010: 2376-2383) and OTU threshold (Qin Xiao, Yuan Chang'an, Deng Yulin et al., An improved OTU image segmentation algorithm[J]. Journal of Shanxi University (Natural Science Edition), 2013, 36(4): 530-534) to achieve the initial detection of foreign objects and save the location and size information of foreign objects.

[0073] Step 4: Based on the location and scale information of the foreign object detected in the first step, the image of the corresponding area is extracted after being enlarged proportionally. The extracted image is then processed using K-Mean (Li Man, Zhao Songlin. Research on the Analysis and Application of K-means Clustering Algorithm [J]. Charming China, 2011, 000(007): 243-243). Threshold segmentation is then performed based on the image information of the center position to achieve secondary detection of the foreign object.

[0074] Step 5: Compare the detection information of the two foreign objects, combine it with the straight line equation of the runway edge, eliminate interference, and realize the automatic detection of foreign objects on the airport runway.

[0075] Step 1 includes:

[0076] Step 1-1: Use the Canny algorithm (Liu Chao, Zhou Jiliu, He Kun. Adaptive edge detection method based on Canny algorithm [J]. Computer Engineering and Design, 2010(18): 4-5) to perform edge detection on the 1920*1080 image img_src and obtain the result image img_canny containing edge information.

[0077] Step 1-2: Perform morphological processing on img_canny from step 1-1 (Lu Zongqi, Zhu Yu. Fast algorithm for digital morphological erosion and dilation operations [J]. National Conference on Image and Graphics, 2006(13):306-311), perform erosion and dilation operations twice in sequence, and generate the resulting binary image img_result.

[0078] Steps 1-3: Set the image region range of img_result in Steps 1-2. The upper region is taken from rows 0 to m, where m is generally 300. The lower region is taken from rows n to 1920, where n is generally 1620. The columns are taken as full columns, which are columns 0 to 1080. Count the number of pixels with a value of 255 in the upper and lower regions, and calculate the proportion coefficient of each pixel in the statistical region.

[0079] The formula for calculating the percentage is as follows:

[0080]

[0081]

[0082] Where P(i,j) is the pixel value of img_result at the position of the i-th row and the j-th column. It is a binary image with a value of 0 or 255. rateup is the scaling factor of the upper half region and ratedown is the scaling factor of the lower half region.

[0083] Steps 1-4: Calculate the scale factor rateup for the upper half of the region and the scale factor ratedown for the lower half of the region using the scale factor calculation formula from Steps 1-3, and determine the region of the runway in the image.

[0084] The method for determining the runway area is as follows:

[0085]

[0086] Where rateval is the set scale threshold, typically 0.15, and L is the indicator of the area where the runway is located. UP means the runway is located in the entire upper half of the image, and there is an area outside the runway in the lower half of the image; MID means the runway is located in the middle of the image, and there is an area outside the runway in both the upper and lower halves of the image; DOWN means the runway is located in the entire lower half of the image, and there is an area outside the runway in the upper half of the image; ALL means the runway covers the entire image area, and there is no area outside the runway.

[0087] Step 2 includes:

[0088] Step 2-1: Analyze the pixel distribution of the `img_result` generated in Step 1-2. When the pixel value at position (x, y) of `img_result` is 255, where x is the row position of `img_result` and y is the column position of `img_result`, count the number of pixels with values ​​of 255 and 0 in positions (x-1, y) to (x-index, y) of `img_result`, num_up_white and num_up_black, respectively. Simultaneously, count the number of pixels with values ​​of 255 and 0 in positions (x+1, n) to (x+index, y), num_down_white and num_down_black, respectively. Here, `index` is typically set to 50. Calculate the percentage of pixels in each position using the following formula:

[0089] rate whiteup =num_up_white / index

[0090] rate blackup =num_up_black / index

[0091] rate whitedown =num_down_white / index

[0092] rate blackdown =num_down_black / index

[0093] Where rate whiteup and rate blackup These represent the percentages of pixels with values ​​of 255 and 0 in the range (x-1, y) to (x-index, y) of `img_result`, respectively. whitedown and rate blackdown The percentage of pixels with values ​​of 255 and 0 in the range (x+1,y) to (x+index,y) of `img_result`; when rate whiteup >val and rate blackdown When >val, the position point (x,y) is placed in the position information table LocalUp. When rate blackup >val and rate whitedown When >val, place the position point (x,y) into the position information table LocalDown;

[0094] Step 2-2: Use the least squares method to perform linear fitting on the location point data in the LocalUp and LocalDown location information tables from Step 2-1, and solve for the linear equations y = k. up x+bup y = k down x+b down Where y is the column coordinate of the image, x is the row coordinate of the image, and k is the number of columns. up k down b up b down The coefficient values ​​of the equation obtained by fitting the data to a straight line.

[0095] Step 2-3: Combining the runway area markers obtained in Step 1-4, and using the straight line equations obtained in Step 2-2, solve for the straight line equations at the upper and lower edges of the runway in the image. The calculation formulas are as follows:

[0096] Equation of the straight line at the upper edge of the runway:

[0097]

[0098] Equation of the straight line at the lower edge of the runway:

[0099]

[0100] Step 2-4: Based on the equation of the runway edge line obtained in Step 2-3, solve for the four intersection points of the upper and lower edge lines of the runway with the left and right edges of the image, which are (0, y1), (1920, y2), (0, y3), and (1920, y4). Then, solve for the minimum value ymin and the maximum value ymax among y1, y2, y3, and y4.

[0101] Step 2-5: Based on the intersection coordinates obtained in Step 2-4, define a rectangular region, with the upper left position point being (0, ymin) and the lower right position point being (1920, ymax). Extract the corresponding rectangular region from the image img_src in Step 1-1 to complete the segmentation of the runway region and generate the runway region segmentation image img_runway.

[0102] Step 3 includes:

[0103] Step 3-1: Process the runway region image img_runway generated in step 2-5 using the AC saliency detection algorithm (ACHANTA R, HEMAMI S, ESTRADA F, et al. Frequency-tuned salient region detection[C]. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, USA, 2010:2376-2383) to obtain the image img_ac.

[0104] Step 3-2: Perform OTU thresholding on the image img_ac from step 3-1 (Qin Xiao, Yuan Changan, Deng Yulin et al., An improved OTU image segmentation algorithm [J]. Journal of Shanxi University (Natural Science Edition), 2013, 36(4): 530-534) to obtain the image img_ostu.

[0105] Step 3-3: Use median filtering (Gao Haojun, Du Yuren. Application of median filtering in image processing [J]. Information Research, 2004, 30(008): 35-36) to denoise the image img_ostu in step 3-2 to obtain the image img_blur.

[0106] Steps 3-4 involve dilating the image img_blur outward by n pixels (Lu Zongqi, Zhu Yu. Fast algorithm for digital morphological erosion and dilation operations [J]. National Conference on Image and Graphics, 2006(13):306-311), where n is typically 15, to obtain the image img_dilate. Then, the image img_dilate is eroded inward by n pixels to obtain the image img_result.

[0107] Steps 3-5 involve contour lookup to detect and locate foreign objects in the image `img_result` obtained in step 3-4, and to obtain the scale and location information of the foreign objects in the image, `rect_ac`.

[0108] Step 4 includes:

[0109] Step 4-1: Based on the foreign object location and scale information rect_ac from Step 3-5, set the region of interest range rect_roi of the image, with the center of the foreign object as the center of the rectangle, and set the width and height of the rectangle to be n times the width and height of the foreign object, respectively. n is generally taken as 2 or 4.

[0110] Step 4-2: Extract and segment the runway region segmentation image img_runway generated in Step 2-5 according to the region of interest rect_roi set in Step 4-1 to obtain the image img_roi;

[0111] Step 4-3: Use the K-Means algorithm (Li Man, Zhao Songlin. Research on the analysis and application of K-means clustering algorithm [J]. Charming China, 2011, 000(007): 243-243) to process the image img_roi obtained in step 4-2 to obtain the image img_kmeans;

[0112] Step 4-4: Query the pixel values at the center positions of the images img_kmeans in Step 4-3, and set the pixel values at the positions in img_kmeans with the same pixel values as those at the center positions to 255, and set those with different pixel values to 0, generating a binary image img_bina;

[0113] Step 4-5: Through contour search, detect and locate foreign objects in the image img_bina obtained in Step 4-4, and re-obtain the scale and position information rect_kmeans of the foreign objects in the image.

[0114] Step 5 includes:

[0115] Step 5-1: Calculate the IOU (Intersection over Union) rate of the two detected objects, rect_ac in Step 3-5 and rect_kmeans in Step 4-5; iou ; and calculate their area ratio rate area = maxarea / minarea, where maxarea is the maximum area of the two detected objects and minarea is the minimum area. At the same time, calculate the distance between the center points of the two objects, distance. When rate iou > iouval, rate area < areval, and distance < disval, it is determined that the detection result meets the requirements. If the requirements are not met, it is considered interference and excluded. Generally, iouval is taken as 0.8, areval is taken as 4, and disval is taken as 20;

[0116] Step 5-2: Determine the relationship between the position of the detection result that meets the requirements in Step 5-1 and the upper and lower runway edge lines solved in Step 2-3. When it is located between the two edge lines, it is considered a real foreign object; otherwise, it is considered interference, completing the final detection and location of foreign objects in the airport runway image.

[0117] Example:

[0118] The following further describes the present invention with reference to the accompanying drawings and examples.

[0119] The present invention discloses an airport runway foreign object detection method based on optical image analysis. As Figure 1 shown, it includes the following steps:

[0120] S1: Determine the area of the runway in the image:

[0121] Determining the location of the runway within an image is crucial for runway image region segmentation. Only by automatically identifying the runway's location can the accuracy of runway segmentation be guaranteed. First, edge detection is performed on the image to statistically analyze the distribution patterns of image edge information. Then, the proportion of pixels with a value of 255 in the upper and lower halves is calculated. The core formula is:

[0122]

[0123]

[0124] Where P(i,j) is the pixel value of img_result at the i-th row and j-th column position, which is a binary image with a value of 0 or 255; rateup is the scaling factor for the upper half of the region; ratedown is the scaling factor for the lower half of the region; and m and n are the set row values ​​for the image's endpoint and start point. Then, based on the calculated scaling factors, the region of the runway in the image is determined, resulting in four possible outcomes (corresponding to...). Figure 2 a, Figure 2 b, Figure 2 c, Figure 2 d) These are UP, MID, DOWN, and ALL, respectively. The diagram shows the area where the runway is located and its corresponding results. Figure 2 As shown.

[0125] S2: Runway area division:

[0126] Runway region segmentation is a prerequisite for foreign object detection. The process involves finding the positions of the upper and lower edges of the runway in each column of the image. Specifically, this involves acquiring the pixel values ​​of each position in the input image after edge processing and morphological processing. Following a set logic, the proportion coefficients of 0-pixel and 255-pixel values ​​are calculated. If a threshold requirement is met, the coordinates of that position are added to a designated location information table. Then, linear fitting is performed on the data in the location information table to obtain the linear equations of the upper and lower edges. Combined with the runway's position in the image, the linear equations of the upper and lower edges of the runway in the image are obtained. The core determination formula is as follows:

[0127] Equation of the straight line at the upper edge of the runway:

[0128]

[0129] Equation of the straight line at the lower edge of the runway:

[0130]

[0131] Where y is the column coordinate of the image, x is the row coordinate of the image, and k is the column coordinate of the image. up k down b up b downThe coefficients of the equation obtained by fitting the data to a straight line are given, and L is the image region marker where the runway is located. Then, based on the equations of the upper and lower edges of the runway, the runway region in the image is segmented, and the segmentation effect is as follows. Figure 3 As shown.

[0132] S3: Initial foreign object detection:

[0133] Initial foreign object detection is the first step in discovering foreign objects. The segmented runway image is processed using the AC saliency detection algorithm; the foreground target in the foreign object image is segmented using OTU thresholding; the image is denoised using median filtering; then, morphological operations are used to connect the broken parts of the image and remove minor noise interference; finally, contour finding is used to detect and locate foreign objects in the image.

[0134] S4: Secondary foreign object detection:

[0135] Secondary foreign object detection is a supplement to primary foreign object detection. Based on the information from the first detection, a secondary detection region is defined with the target center as the center and its width and height enlarged proportionally. The corresponding image region is extracted, and K-Means is used to cluster and segment the extracted image. Then, based on the center pixel of the segmented image, image binarization is performed. Finally, contour lookup is used to achieve secondary detection and localization of foreign objects in the image.

[0136] S5: Foreign object confirmation:

[0137] By comparing the location information of two foreign object detections, primarily through comparisons of Intersection over Union (IOU), area ratio, and center distance, it is determined whether the detected target is a real foreign object or interference. Finally, the location of the confirmed foreign object is compared with the straight-line equations of the upper and lower edges of the runway to determine whether the target is on the runway. This achieves the final detection and localization of foreign objects in airport runway images. The specific detection process and results are as follows: Figure 4 As shown.

[0138] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding an airport runway foreign object detection method based on optical image analysis, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0139] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0140] This invention provides a concept and method for foreign object detection on airport runways based on optical image analysis. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for detecting foreign objects on airport runways based on optical image analysis, characterized in that, Includes the following steps: Step 1: Capture an image of the airport runway, perform edge processing on the airport runway image, calculate the distribution of its edge information, and determine the image region where the runway is located. Step 2: Based on the image region where the runway is located, use runway edge fitting to calculate the equation of the straight line at the runway edge, obtain the fitted straight line at the runway edge, segment the runway, and obtain the segmented runway image; Step 3, Initial detection of foreign objects: The segmented runway image is processed using AC saliency detection and OTU threshold segmentation to save the location and size information of the foreign objects; Step 4, perform secondary detection of foreign objects: Based on the location and size information of the foreign object obtained in Step 3, after scaling up the image, extract the image of the corresponding area of ​​the foreign object, process the extracted image using the K-Means method, and perform threshold segmentation based on the image information of the center position; Step 5: Compare the detection results of the two foreign objects, combine them with the straight line equation at the runway edge, eliminate interference, and complete the airport runway foreign object detection based on optical image analysis. The segmentation of the runway mentioned in step 2 specifically includes: Step 2-1: Analyze the pixel distribution of the binary image img_result generated in Step 1-2, and generate the first position information table LocalUp and the second position information table LocalDown accordingly. Step 2-2: Use the least squares method to perform linear fitting on the location point data in the first location information table LocalUp and the second location information table LocalDown generated in Step 2-1, respectively, to obtain two linear equations: ； ； Where y is the column coordinate of the airport runway image, and x is the row coordinate of the airport runway image. , , and The coefficient values ​​of the equation obtained by fitting the data to a straight line; Step 2-3: Combining the runway area markers obtained in Step 1-4, and using the straight line equations obtained in Step 2-2, solve for the straight line equations of the upper and lower edges of the runway in the airport runway image. The calculation method is as follows: Equation of the straight line at the upper edge of the runway: ； Equation of the straight line at the lower edge of the runway: ； Step 2-4: Based on the equation of the runway edge line obtained in Step 2-3, solve for the four intersection points of the upper and lower edge lines of the runway with the left and right edges of the image, which are (0, y1), (M, y2), (0, y3), and (M, y4). Then, solve for the minimum value ymin and the maximum value ymax among y1, y2, y3, and y4. Steps 2-5: Define a rectangular region, with the upper left position point as (0, ymin) and the lower right position point as (M, ymax). Extract the corresponding rectangular region from the airport runway image img_src in Step 1-1 to complete the segmentation of the runway region and generate the runway region segmentation image img_runway. The edge processing of the airport runway image described in step 1 specifically includes: Step 1-1: Use the Canny algorithm to perform edge detection on the airport runway image img_src and obtain the result image img_canny containing edge information. The size of the airport runway image img_src is M pixels wide and N pixels high. Step 1-2: Perform morphological processing on the result image img_canny containing edge information obtained in step 1-1, performing opening and closing operations twice in sequence to generate the result binary image img_result. Steps 1-3: Define the image region range of the binary image `img_result` from Steps 1-2. The upper region is taken from rows 0 to m of the image, and the lower region is taken from row n to the last row (row N). The number of columns is M. Count the number of pixels with a value of 255 in both the upper and lower regions, and calculate their respective proportions within the statistical regions. The specific method is as follows: ； ； Where P(i,j) is the pixel value corresponding to the position of the i-th row and j-th column of the binary image img_result, which is a binary image, and the value is 0 or 255. rateup is the scaling factor of the upper half region and ratedown is the scaling factor of the lower half region. Steps 1-4: Using the upper half region scaling factor rateup and lower half region scaling factor ratedown calculated in steps 1-3, determine the area of ​​the runway in the image. The specific determination method for identifying the area of ​​the runway in the image as described in steps 1-4 is as follows: ； Where rateval is the set scale coefficient threshold, and L is the region marker of the runway, represented as follows: UP means the runway is located in the entire upper half of the image, and there is an area outside the runway in the lower half of the image; MID means the runway is located in the middle of the image, and there is an area outside the runway in both the upper and lower halves of the image; DOWN means the runway is located in the entire lower half of the image, and there is an area outside the runway in the upper half of the image; ALL means the runway covers the entire image area, and there is no area outside the runway.

2. The method for detecting foreign objects on airport runways based on optical image analysis according to claim 1, characterized in that, The specific method for generating the first location information table LocalUp and the second location information table LocalDown as described in step 2-1 is as follows: When the pixel value at position (x, y) in the resulting binary image `img_result` is 255, where x is the row position of `img_result` and y is the column position of `img_result`, the number of pixels with values ​​of 255 and 0 in positions (x-1, y) to (x-index, y) of the resulting binary image `img_result` is counted as `num_up_white` and `num_up_black`, respectively. Simultaneously, the number of pixels with values ​​of 255 and 0 in positions (x+1, y) to (x+index, y) of the resulting binary image `img_result` is counted as `num_down_white` and `num_down_black`, respectively, where `index` is a preset statistical range. The percentage of pixels in each range is calculated using the following formula: ； ； ； ； in, and These represent the percentages of pixels with values ​​of 255 and 0 in the `img_result` array at positions (x-1, y) to (x-index, y), respectively. and The percentage of pixels with values ​​of 255 and 0 in the range (x+1,y) to (x+index,y) in the img_result. when and When the position (x, y) is entered into the first position information table LocalUp; when and When the position (x, y) is entered into the second position information table LocalDown.

3. The method for detecting foreign objects on airport runways based on optical image analysis according to claim 2, characterized in that, The initial detection of foreign objects described in step 3 specifically includes: Step 3-1: Process the runway region segmentation image img_runway generated in Step 2-5 using the AC saliency detection algorithm to obtain the AC image img_ac; Step 3-2: Perform OTU thresholding on the AC image img_ac from step 3-1 to obtain the OTU image img_ostu; Step 3-3: Use median filtering to denoise the OTU image img_ostu from step 3-2 to obtain the denoised image img_blur; Steps 3-4: Dilate the denoised image img_blur outward by n1 pixels to obtain the dilated image img_dilate, and then erode the image img_dilate inward by n1 pixels to obtain the eroded image img_result. Steps 3-5 involve detecting and locating foreign objects in the erosion image img_result obtained in Step 3-4 through contour lookup, and obtaining preliminary scale and location information rect_ac of the foreign objects in the runway region segmentation image.

4. The method for detecting foreign objects on airport runways based on optical image analysis according to claim 3, characterized in that, The secondary detection of foreign objects described in step 4 specifically includes: Step 4-1: Based on the preliminary scale and location information rect_ac of the foreign object in the runway area segmentation image obtained in Step 3-5, set the target area range rect_roi, take the center position of the foreign object as the center of the rectangle, and set the width and height of the rectangle to be n2 times the width and height of the foreign object, respectively. Step 4-2: Extract and segment the runway region segmentation image img_runway generated in Step 2-5 according to the target region rect_roi set in Step 4-1 to obtain the segmented image img_roi; Step 4-3: Process the segmented image img_roi obtained in Step 4-2 using the K-Means algorithm to obtain the segmented image img_kmeans; Step 4-4: Query the pixel value at the center position of the segmented image img_kmeans in step 4-3, and set the pixel value of the position in img_kmeans that is the same as the pixel value to 255, and set the pixel value of the position that is different to 0, and generate the binarized image img_bina; Steps 4-5: By contour lookup, foreign objects in the binarized image img_bina obtained in step 4-4 are detected and located, and the secondary scale and position information of the foreign objects in the image rect_kmeans are obtained.

5. The method for detecting foreign objects on airport runways based on optical image analysis according to claim 4, characterized in that, The interference removal described in step 5 specifically includes: Step 5-1: Calculate the Intersection over Union (IOU) of the two detected objects for the preliminary scale and position information rect_ac obtained in Step 3-5 and the secondary scale and position information rect_kmeans obtained in Step 4-5 ; and calculate their area ratio , where maxarea is the maximum area of the two detected objects, minarea is the minimum area, and at the same time calculate the distance distance between the center points of the two objects. When , , when distance < disval, it is determined that the detection result meets the requirements; otherwise, it is determined as interference and is excluded, where iouval, areval, and disval are preset thresholds; Step 5-2: Determine the relationship between the location of the detection result that meets the requirements in Step 5-1 and the upper and lower runway edge lines solved in Step 2-3. If it is located between the two edge lines, it is determined to be a real foreign object; otherwise, it is interference and is removed.

6. The method for detecting foreign objects on airport runways based on optical image analysis according to claim 5, characterized in that, The threshold value of the proportional coefficient, rateval, mentioned in steps 1-4, is 0.

15.

7. The method for detecting foreign objects on airport runways based on optical image analysis according to claim 6, characterized in that, The preset statistical range index is 50.

Images