An aerial vision airport runway detection and positioning method

Abstract

The application discloses an aviation visual airport runway detection and positioning method and belongs to the field of image processing; specifically, first, a runway to be detected is actually photographed and a mask image is labeled, then an improved residual network learning module is adopted on the basis of an original FCN image segmentation network, and improved components such as a hollow convolution and a transposed convolution are introduced to an up-sampling structure, a CRF post-processing module is added, and the actually photographed image is segmented into a binary mask image. Next, the binary mask image is subjected to runway corner point detection, whether the upper and lower boundary corner points in the runway mask image can be completely seen is determined, a runway center line detection method and an improved center line detection method are respectively adopted for processing, finally, a runway center line equation is obtained through a transformation equation, and the runway lines and the center line are represented in a result image in combination with two-side runway line equations. The application has strong robustness and fast processing speed, can process video information in real time, and has a wide application range.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to an aerial vision method for airport runway detection and localization. Background Technology

[0002] For a long time, pilots have been able to judge whether an airport runway meets the standards for a successful landing by visual inspection during aircraft approach and landing. If a pilot cannot visually identify the visual reference required for descent at the decision altitude (DA) or minimum descent altitude (MDA), he has no choice but to go around the runway. Going around the runway will affect the interests of airlines and passengers and cause unnecessary economic losses.

[0003] Enhanced flight vision system technology has emerged to address this need. This technology provides pilots with clearer, richer, and more reliable safety guidance information by fusing and displaying enhanced information in real time within a visual navigation vision system. It enhances pilots' situational awareness outside the cockpit, improves operational efficiency during the approach and landing phase of civil aviation, and ultimately achieves high-precision, high-reliability, and high-availability real-time situational awareness in complex environments, providing technical support for safe and precise guidance during the approach and landing phase.

[0004] During civil aviation approach and landing, different landing attitudes at different altitudes and angles need to be considered, which makes the shape of the runway target complex and varied. At the same time, the complexity of the airport scene increases the difficulty of image segmentation, making it difficult for the accuracy of runway corner detection to meet industry standards.

[0005] Existing runway corner detection technologies are mainly divided into two types: traditional methods based on line detection and detection algorithms based on deep learning;

[0006] The first category of traditional methods mainly includes contour line detection, Harris corner detection, Canny operator edge detection, and sliding window template detection. These methods mostly rely on prior knowledge and specific thresholds, making it difficult to correctly detect corners in complex scenes and failing to meet the real-time requirements of corner detection. At the same time, they are difficult to escape the influence of human experience and have significant limitations in complex scenes with multiple angles and heights.

[0007] The second category of deep learning-based detection algorithms is further divided into two types: target detection methods for runways and pixel-level image segmentation methods. Although they generally outperform traditional methods in terms of processing performance and speed, they still have the following problems:

[0008] In multi-target scenarios, the former determines the position and category of the target based on the feature values ​​of the runway target, and continuously improves the target box based on the loss value, and finally successfully identifies the target runway. Although the target detection method can accurately regress the runway position, it is difficult to regress the accurate coordinates of the runway corner points.

[0009] The latter method determines the category of each pixel, achieving pixel-level classification through methods such as masking. However, the accuracy of pixel segmentation is not ideal; in complex scenes, segmenting incorrect pixels using masks can significantly affect the detection of corner coordinates. Summary of the Invention

[0010] To address the shortcomings of existing technologies, such as the inability to detect corner points in real time, the need for manual parameter tuning, susceptibility to prior knowledge, vulnerability to image noise in complex scenes, and poor fault tolerance, this invention proposes an aviation vision-based airport runway detection and localization method. This method combines deep learning with traditional approaches, improving the system's real-time performance and accuracy, enhancing its fault tolerance, and improving its detection performance and noise immunity in complex environments. Without relying on complex prior assumptions, this invention enables real-time output of the location coordinates of airport runway corner points and corresponding images of the runway lines and centerline, based solely on a complex airport scene image.

[0011] The specific implementation method is as follows:

[0012] Step 1: For the airport runway to be tested, select key points of the runway, take real-time photos with a drone to obtain the original images, and annotate them to obtain the corresponding mask images; combine each original image and its corresponding mask image into an image pair.

[0013] Step 2: Construct a pixel-level image segmentation model that includes a residual network learning module, a transposed convolutional upsampling structure, and a CRF post-processing module;

[0014] The residual network learning module is an improvement on the original ResNet101 network, consisting of five downsampling layers: an initial convolutional layer, Layer 1, Layer 2, Layer 3, and Layer 4.

[0015] Specifically, improved residual learning units are used in Layer 3 and Layer 4. The original feature extraction process is improved by adding dilated convolution operations with different dilation rates and adjusting the stride value. The residual learning units include Bottleneck1 and Bottleneck2 structures. Bottleneck1 corresponds to the residual learning between two layers during the downsampling convolution process in shallow networks, while Bottleneck2 corresponds to the residual learning between three layers during the downsampling convolution process in deep networks.

[0016] The transposed convolutional upsampling structure includes, from top to bottom, a transposed convolutional layer, a Dropout layer, and an image restoration layer.

[0017] The transposed convolutional layer uses a 3×3 convolutional layer to amplify key feature details of the feature map and suppress information in non-feature regions. The Dropout layer changes the number of channels in the input image, facilitating the stitching of subsequent pixel classification results. The image restoration layer uses bilinear interpolation to restore the image to its original size.

[0018] The transposed convolutional upsampling structure was copied into three identical structures and connected to Layer 2, Layer 3, and Layer 4 of the residual network learning module, respectively, to perform upsampling operations of 16x, 8x, and 4x on the feature maps.

[0019] Finally, the upsampled results are concatenated to obtain the initial mask image, ensuring that it is the same size as the input image.

[0020] The CRF post-processing module mainly includes the fully connected conditional random field (CRF) method, i.e., pixel-level dense connections; the optimized mask image is obtained after passing through this module.

[0021] Step 3: Using the original image of the airport runway and the corresponding mask image, train the pixel-level image segmentation model and continuously optimize the model's weight parameters through the loss function.

[0022] Step 4: Convert the weight file of the trained pixel-level image segmentation model from PyTorch to ONNX format and save it on the ONNXRuntime inference framework for accelerated inference. The mask image of the runway is obtained by inference for each original image of the runway.

[0023] Step 5: For the runway mask image used in inference, construct an adaptive runway corner detection module to detect the corners of each inference mask image.

[0024] The specific process is as follows:

[0025] Step 501: Detect the edge information of the current inference runway mask image using the Canny operator, detect the position of the straight line using the Hough transform, cluster the angle values ​​formed by the straight line and the polar axis in the parameter space, save the parameter values ​​of the runway lines on both sides after clustering, and outline the runway lines on both sides in the runway mask image.

[0026] Step 502: Mark the pixel value of the corresponding pixel points of the runway lines on both sides as 1, and save the result image as a runway line mask image.

[0027] Step 503: For the outlined runway contour, use the convex hull algorithm to draw the smallest quadrilateral that best matches the runway contour, and mark the pixel value of the boundary point on the smallest quadrilateral as 1. Save the resulting image as a quadrilateral mask image.

[0028] Step 504: Sum the pixel values ​​of pixels at the same position in the runway line mask image and the quadrilateral mask image, filter out the pixels with a pixel value of 2, which are the corner points of the airport runway, return the corresponding coordinates of the corner points, and mark the corner points in the current runway mask image.

[0029] Step 6: Determine whether the corner points of the upper and lower boundaries in the runway mask image can be seen completely. If so, proceed to Step 7 and process them using the runway centerline detection method; otherwise, proceed to Step 8.

[0030] When the drone lands, the runway end furthest from the drone's landing position is defined as the upper boundary.

[0031] Step 7: Construct a runway line and centerline detection module, and mark the runway lines and centerline on the runway mask image where corner points have been detected;

[0032] The specific testing steps are as follows:

[0033] 1) Define a rectangle in the new view plane, and define the two midpoints A and B of the upper and lower boundaries of the rectangle as centerline feature points.

[0034] 2) Using the perspective transformation algorithm, the perspective correction matrix M is obtained based on the spatial correspondence between the coordinates of the four vertices of the rectangle and the coordinates of the four corner points detected on the runway mask image;

[0035] The perspective correction matrix M is a 3x3 homogeneous invertible matrix:

[0036] Where a 11 a 12 a 13 The element controls the affine transformation of elements on the x-axis; a 21 a 22 a 23 The element controls the affine transformation of elements on the y-axis; a 31 a 32 Element controls perspective transformation; a 33 The value is 1.

[0037] 3) Transform the centerline feature points A and B to their corresponding points A' and B' on the runway mask image using the perspective correction matrix M;

[0038] 4) Connect the corresponding points A' and B' to obtain the equation of the runway centerline. Similarly, connect the coordinates of the four corner points of the runway mask image in pairs to obtain the equations of the runway lines on both sides of the airport runway.

[0039] 5) Using the drawing function, based on the runway centerline equation and runway line equation, visualize the runway line and centerline, and finally label them on the result graph.

[0040] Step 8: When the corner points of the upper boundary of the runway can be seen completely, but the corner points of the lower boundary are not displayed completely, take the improved centerline detection measures.

[0041] Specifically:

[0042] Step 801: For the four corner points (a1,b1), (a2,b2), (c1,d1), and (c2,d2) detected in the runway mask image, sort the corner points according to their y values, select the two points with larger y values ​​as (c1,d1) and (c2,d2), and determine the coordinates of the corner point with the second largest y value as (c1,d1).

[0043] Step 802: Connect the two corner points (a1, b1) and (a2, b2) of the upper boundary of the runway with a straight line, calculate the slope k of the line, and draw a line parallel to the upper boundary of the runway through point (c1, d1). Determine the equation of the parallel line using the following formula:

[0044] y-d1=k·(x-c1)

[0045] Step 803: Connect the corner points (a1,b1) and (c1,d1), and (a2,b2) and (c2,d2) respectively to determine the equations of the straight lines on both sides of the runway.

[0046] Step 804: The line parallel to corner point (c1, d1) intersects the right boundary line of the runway to obtain a new corner point (c2). ′ ,d ′ 2).

[0047] Step 805, combine (a1,b1)(a2,b2)(c1,d1)(c2) ′ ,d ′ 2) As the new corresponding corner point in the original mask image.

[0048] Step 806: Input the four new corresponding corner points and the four endpoints of the custom view plane rectangle into the perspective transformation algorithm to obtain the perspective correction matrix M'.

[0049] Step 807: The two centerline feature points A and B of the custom view plane rectangle are transformed to the corresponding points in the current runway mask image through the perspective correction matrix M'. Connecting the two corresponding points will yield the centerline of the runway, which can then be extended to obtain the final result image.

[0050] The advantages of this invention are:

[0051] (1) The present invention provides an aviation vision airport runway detection and localization method, which uses a large number of real airport runway images in complex scenes with multiple angles and heights for training. Compared with the original traditional detection method, it is more robust, more realistic, and more applicable to engineering deployment. The present invention is more in line with reality.

[0052] (2) The present invention provides an aviation vision airport runway detection and positioning method. After training the runway corner detection model and the runway centerline detection model, the real runway image can be input to output real-time runway corner information and runway centerline information without the need for other prior information.

[0053] (3) The present invention provides an aviation vision airport runway detection and positioning method, which is robust and fast in processing, can process video information in real time, and has a wide range of applications. Attached Figure Description

[0054] Figure 1 This is an overall flowchart of an aviation vision-based airport runway detection and positioning method according to the present invention;

[0055] Figure 2 This is a flowchart of a pixel-level image segmentation model for an aviation vision airport runway detection and localization method according to the present invention;

[0056] Figure 3 This invention provides a pixel-level image segmentation runway mask image.

[0057] Figure 4 This is a flowchart of the adaptive runway corner detection module of the aviation vision airport runway detection and positioning method of the present invention;

[0058] Figure 5 This is a flowchart of the detection process for the runway lines and centerline after the corner points of the upper and lower boundaries of the runway are fully visible in this invention.

[0059] Figure 6 This is a flowchart of the process for detecting the runway line and centerline when the corner points of the upper and lower boundaries of the runway cannot be fully seen in this invention.

[0060] Figure 7 This is an example diagram of the runway corner detection results of the present invention;

[0061] Figure 8 This is an example diagram showing the detection results of the runway lines and center line on both sides of the present invention;

[0062] Figure 9 This is a diagram showing the processing results of the runway lines and centerline in the first runway condition according to the present invention;

[0063] Figure 10 This is a diagram showing the processing results of the runway lines and centerline for the second runway scenario according to the present invention; Detailed Implementation

[0064] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some, not all, embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should fall within the scope of protection of the present invention.

[0065] This invention discloses an aviation vision-based method for airport runway detection and localization. The pixel-level image segmentation model improves the performance of the image segmentation network based on the FCN network, completes the segmentation of the original runway image and obtains the corresponding mask image, then detects the runway lines to obtain the corner points of the runway, and finally detects and draws the center line and runway lines of the runway through a perspective transformation method.

[0066] In the pixel-level image segmentation model, to address the issues of imprecise segmentation results and lack of spatial consistency between pixels in the original FCN network, this invention modifies the backbone network by adding a transposed convolutional upsampling structure and a CRF post-processing method. This refines the downsampling and upsampling processes, preserving both abstract and detailed image features. In the adaptive runway corner detection module, an improved convex hull algorithm and adaptive runway line clustering algorithm are integrated. The corresponding runway line equation is detected based on the runway line's features in the parameter space, and the final runway corner coordinates are obtained using the detected minimum envelope quadrilateral. In the runway line and centerline detection module, to address the issue of complex runway angles affecting accurate centerline detection, perspective transformation is used to accurately identify centerline feature points, thereby correctly locating the runway centerline and improving the robustness of the detection algorithm.

[0067] In the absence of publicly available airport runway datasets, this invention establishes an airport runway image dataset in VOC2007 format containing 5000 images; to meet the real-time requirements of corner detection, this invention deploys the trained model on ONNXRuntime for accelerated inference.

[0068] like Figure 1 As shown, the specific implementation method is as follows:

[0069] Step 1: For the airport runway to be tested, select key points of the runway, take real-time photos with a drone to obtain the original images, and annotate them to obtain the corresponding mask images; combine each original image and its corresponding mask image into an image pair.

[0070] Step 2: Construct a pixel-level image segmentation model that includes a residual network learning module, a transposed convolutional upsampling structure, and a CRF post-processing module;

[0071] The original ResNet101 residual network's learning units suffer from high computational cost, slow processing speed, and poor performance. This invention employs an improved residual network learning module based on the original ResNet101 network, such as... Figure 2 As shown, it consists of five downsampling layers: the initial convolutional layer, Layer 1, Layer 2, Layer 3, and Layer 4.

[0072] Specifically, improved residual learning units are used in Layer 3 and Layer 4. The original feature extraction process is improved by adding dilated convolution operations with different dilation rates and adjusting the stride value. This increases the receptive field of the convolution operation and effectively reduces the computational cost in the downsampling process. Adjusting the stride value of the convolution calculation makes the downsampling process more effective.

[0073] The improved ResNet101 network contains 33 improved residual learning units. Based on the presence or absence of shortcut connections, the residual learning units are divided into Bottleneck1 and Bottleneck2 structures. Bottleneck1 corresponds to the residual learning between two layers in the downsampling convolution process of shallow networks, and Bottleneck2 corresponds to the residual learning between three layers in the downsampling convolution process of deep networks.

[0074] The transposed convolutional upsampling structure includes, from top to bottom, a transposed convolutional layer, a Dropout layer, and an image restoration layer.

[0075] The transposed convolutional layer uses a 3×3 convolutional layer to amplify key feature details of the feature map and suppress information in non-feature regions. The Dropout layer mainly prevents overfitting by changing the number of channels in the input image, facilitating the stitching of subsequent pixel classification results. The image restoration layer uses bilinear interpolation to restore the image to its original size.

[0076] The transposed convolutional upsampling structure was copied into three identical structures and connected to Layer 2, Layer 3 and Layer 4 of the residual network learning module, respectively. The transposed convolutional layer followed the structure of the decoder in the Deeplab series and performed upsampling operations of 16x, 8x and 4x on the feature map, respectively.

[0077] Finally, the upsampled results are concatenated to obtain the initial mask image, ensuring that it is the same size as the input image.

[0078] This invention adds the CRF post-processing module after processing the existing segmentation model. The module mainly includes the fully connected conditional random field (CRF) method, i.e., pixel-level dense connections. After processing by this module, an optimized mask image is obtained, which significantly improves the accuracy of image segmentation.

[0079] The result of semantic segmentation by the pixel-level image segmentation model is as follows Figure 3 As shown.

[0080] Step 3: Using the original image of the airport runway and the corresponding mask image, train the pixel-level image segmentation model and continuously optimize the model's weight parameters through the loss function.

[0081] Training includes the following process:

[0082] 1) Data augmentation was performed on the original runway images captured by drones using vertical and horizontal flipping and random cropping.

[0083] 2) Input the data-augmented runway image into the residual network learning module of the pixel-level image segmentation model, and continuously extract abstract features from the image using the downsampling structure;

[0084] 3) The segmentation results of Layer 2, Layer 3 and Layer 4 are concatenated with the transposed convolutional upsampling structure respectively. Finally, the segmentation prediction results obtained by each layer are concatted to obtain the prediction mask image.

[0085] 4) Compare the predicted mask image with the ground truth mask image corresponding to the original image, and feed the loss back to the pixel-level image segmentation model to update the model weights;

[0086] 5) Repeat the above steps until the preset number of iterations is reached. By continuously adjusting the hyperparameters of the network, the final weight file is obtained and saved as the optimal parameters.

[0087] Step 4: Convert the weight file of the trained pixel-level image segmentation model from PyTorch to ONNX format and save it on the ONNXRuntime inference framework for accelerated inference. The mask image of the runway is obtained by inference for each original image of the runway.

[0088] Accelerate reasoning to meet real-time requirements.

[0089] Step 5: For the runway mask image used in inference, construct an adaptive runway corner detection module to detect the corners of each inference mask image.

[0090] The adaptive runway corner detection module consists of an adaptive runway line detection method, a minimum envelope quadrilateral method, and a mask summation detection module. To address the issue of the original method being susceptible to image noise, the last module improves upon this by integrating an adaptive runway detection clustering algorithm and a convex hull algorithm, thereby enhancing detection accuracy.

[0091] The specific structure of the module is as follows: Figure 4 As shown, the specific process is as follows:

[0092] Step 501: Detect the edge information of the current inference runway mask image using the Canny operator, detect the position of the straight line using the Hough transform, describe the information of the straight line in the parameter space, cluster the angle values ​​formed by the straight line and the polar axis in the parameter space, save the parameter values ​​of the runway lines on both sides after clustering, and outline the runway lines on both sides in the runway mask image.

[0093] The improved adaptive runway line detection method clusters the angle values ​​formed by the straight line and the polar axis in the parameter space, effectively reducing the adverse effects of noise on the runway line detection process.

[0094] Step 502: Mark the pixel value of the corresponding pixel points of the runway lines on both sides as 1, and save the result image as a runway line mask image.

[0095] Step 503: For the outlined runway contour, use the convex hull algorithm to draw the smallest quadrilateral that best matches the runway contour, and mark the pixel value of the boundary point on the smallest quadrilateral as 1. Save the resulting image as a quadrilateral mask image.

[0096] Step 504: Sum the pixel values ​​of pixels at the same position in the runway line mask image and the quadrilateral mask image, and filter out pixels with a value of 2. Pixels with a value of 2 indicate that they are on both the runway line and the smallest quadrilateral. These pixels are the corner points of the airport runway. Return the corresponding coordinates of the corner points and mark them in the current runway mask image.

[0097] Step 6: Determine whether the corner points of the upper and lower boundaries in the runway mask image can be seen completely. If so, proceed to Step 7 and process them using the runway centerline detection method; otherwise, proceed to Step 8.

[0098] When the drone lands, the runway end furthest from the drone's landing position is defined as the upper boundary.

[0099] Step 7: Construct a runway line and centerline detection module, and mark the runway lines and centerline on the runway mask image where corner points have been detected;

[0100] To address the problem of inaccurate centerline calculations caused by distortions in the planar position of airport runway lines due to the tilt angle between the viewpoint and the ground, this invention improves the algorithm's universality and robustness in complex scenarios.

[0101] like Figure 5 As shown, the specific detection steps are as follows:

[0102] 1) Define a rectangle in the new view plane, and define the two midpoints A and B of the upper and lower boundaries of the rectangle as centerline feature points.

[0103] 2) Using the perspective transformation algorithm, the perspective correction matrix M is obtained based on the spatial correspondence between the coordinates of the four vertices of the rectangle and the coordinates of the four corner points detected on the runway mask image;

[0104] The perspective correction matrix M is a 3x3 homogeneous invertible matrix:

[0105] Where a 11 a 12 a 13 The element controls the affine transformation of elements on the x-axis; a 21 a 22 a 23 The element controls the affine transformation of elements on the y-axis; a 31 a 32 Element controls perspective transformation; a 33 The value is 1.

[0106] 3) Transform the centerline feature points A and B to their corresponding points A' and B' on the runway mask image using the perspective correction matrix M;

[0107] 4) Connect the corresponding points A' and B' to obtain the equation of the runway centerline. Similarly, connect the coordinates of the four corner points of the runway mask image in pairs to obtain the equations of the runway lines on both sides of the airport runway.

[0108] 5) Using the drawing function, based on the runway centerline equation and runway line equation, visualize the runway line and centerline, and finally label them on the result graph.

[0109] Step 8: When the corner points of the upper boundary of the runway can be seen completely, but the corner points of the lower boundary are not displayed completely, take the improved centerline detection measures.

[0110] like Figure 6 As shown, specifically:

[0111] Step 801: For the four corner points (a1,b1), (a2,b2), (c1,d1), and (c2,d2) detected in the runway mask image, sort the corner points according to their y values, select the two points with larger y values ​​as (c1,d1) and (c2,d2), and determine the coordinates of the corner point with the second largest y value as (c1,d1).

[0112] Step 802: Connect the two corner points (a1, b1) and (a2, b2) of the upper boundary of the runway with a straight line, calculate the slope k of the line, and draw a line parallel to the upper boundary of the runway through point (c1, d1). Determine the equation of the parallel line using the following formula:

[0113] y-d1=k·(x-c1)

[0114] Step 803: Connect the corner points (a1,b1) and (c1,d1), and (a2,b2) and (c2,d2) respectively to determine the equations of the straight lines on both sides of the runway.

[0115] Step 804: The line parallel to corner point (c1, d1) intersects the right boundary line of the runway to obtain a new corner point (c2). ′ ,d ′ 2).

[0116] Step 805, combine (a1,b1)(a2,b2)(c1,d1)(c2) ′ ,d ′ 2) As the new corresponding corner point in the original mask image.

[0117] Step 806: Input the four new corresponding corner points and the four endpoints of the custom view plane rectangle into the perspective transformation algorithm to obtain the perspective correction matrix M'.

[0118] Step 807: The two centerline feature points A and B of the custom view plane rectangle are transformed to the corresponding points in the current runway mask image through the perspective correction matrix M'. Connecting the two corresponding points will yield the centerline of the runway, which can then be extended to obtain the final result image.

[0119] Example:

[0120] This invention uses labelme software to annotate key points of the runway under test to obtain mask label files. The runway dataset is then organized according to PASCALVOC 2007 version to obtain 5,000 real-world UAV-captured airport runway images and their mask image annotation files. Each original image and its corresponding mask image are combined into an image pair.

[0121] The runway's angle ranges from 0° to 360°, its vertical distance ranges from 100 meters to 5000 meters, and its horizontal distance ranges from 500 meters to 2500 meters.

[0122] Step 1: Divide the airport runway dataset captured by drones into training, validation, and test sets according to proportions. Then, add fog to the runway images based on the original images to better simulate the complex situations in real-world scenarios.

[0123] Step 2: Construct a pixel-level image segmentation model

[0124] The residual network innovative learning module uses the ResNet101 network as the backbone structure to obtain runway feature information; the transposed convolution upsampling structure uses transposed convolution to complete upsampling and eliminate the influence of non-feature positions, and finally restores the image size; the CRF post-processing module ensures the consistency of image segmentation and performs correction and classification on each pixel after segmentation.

[0125] This invention adds dilated convolution operations with different dilation rates to the residual learning units, thereby increasing the receptive field of the convolution operation and encompassing a wider range of image information. This makes the runway feature information captured by the residual network more abstract. To ensure that the size of the resulting image remains consistent and to ensure that more details are captured, this invention also adjusts the stride parameter value of the network.

[0126] The difference between dilated convolution and regular convolution lies in the fact that, with the same kernel size, dilated convolution results in a larger receptive field. This means that a single pixel output by dilated convolution contains richer contextual information. Another characteristic of dilated convolution is that it does not affect the size of the input feature map while increasing the receptive field, effectively overcoming the shortcomings of pooling layers. This embodiment introduces a dilation coefficient `rate` corresponding to dilated convolution. The larger the dilation coefficient, the larger the corresponding receptive field. Let the side length of the receptive field be `r`, and the formula for calculating the receptive field can be obtained as follows:

[0127]

[0128] As can be seen from the formula for calculating the receptive field, the change in the receptive field increases exponentially with the expansion coefficient.

[0129] The backbone network of this invention adopts an improved ResNet101 residual network, which includes five downsampling layers: Conv1 layer, Layer1 layer, Layer2 layer, Layer3 layer, and Layer4 layer.

[0130] The Conv1 layer contains a 7×7 convolution operation, the Layer1 layer contains 3 innovative learning units, the Layer2 layer contains 4 innovative residual learning units, the Layer3 layer contains 23 innovative residual learning units, and the Layer4 layer contains 3 innovative residual learning units.

[0131] The innovative residual learning units can be divided into Bottleneck1 and Bottleneck2 based on the presence or absence of shortcut connections. Layer 1 contains 3 Bottleneck1 units, Layer 2 contains 4 Bottleneck1 units, Layer 3 contains 1 Bottleneck1 unit and 22 Bottleneck2 units, and Layer 4 contains 1 Bottleneck1 unit and 2 Bottleneck2 units.

[0132] 2) The transposed convolution upsampling structure consists of a transposed convolution layer, a Dropout layer, and an image restoration layer.

[0133] Following the decoder structure of the Deeplab series, this invention refines the upsampling structure by performing upsampling operations of 16x, 8x, and 4x on the feature maps, respectively. Finally, the processed images obtained from the upsampling are concatenated using `concat`. A 3×3 convolutional kernel is used in the transposed convolutional layer. The 3×3 convolutional layer amplifies the details of key features in the feature map and suppresses information in non-feature regions.

[0134] Transposed convolution, also known as deconvolution, is a technique used to resize images. Multiple convolution operations often reduce the image size, while in many cases it's necessary to restore the feature map to its original input image size. First, the convolution operation can be understood as matrix multiplication of the input feature matrix and the convolution kernel matrix.

[0135] The input feature matrix X of size 3×3 is represented as follows:

[0136]

[0137]

[0138] The convolution kernel W can be represented as a sparse matrix C.

[0139]

[0140] Expand the feature matrix X into column vectors.

[0141] X = [x 11 x 12 x 13 x 21 x 22 x 23 x 31 x 32 x 33 ] T

[0142] The convolution operation can then be expressed by the formula:

[0143] Y = C·X

[0144] Transposed convolution, on the other hand, interchanges the input and output, which can be expressed by the formula:

[0145] X = C T ·Y

[0146] Here, we assume a parameter environment where the input X = [abcd], the convolution kernel k = [xyz], and the output Y = [pq]. T Let stride = 1; then

[0147]

[0148] The result of the transposed convolution can be expressed as: It can be obtained through matrix calculation, or by trying to devise a new ordinary convolution to complete the matrix calculation process:

[0149] Suppose the input matrix X in this ordinary convolution process T =[0 0 pq 0 0], the convolution kernel k is represented as k = [zyx], and similarly through Y T =k·X T The formula completes the output matrix Y T The calculation also uses a stride value of 1.

[0150] After calculation, Y at this time T The matrix calculation result is the same as that of the transposed convolution mentioned above. Therefore, it can be seen that the relevant calculation process of the transposed convolution is consistent with the assumed calculation process of the ordinary convolution. So, the calculation process of the transposed convolution is summarized.

[0151] The main idea is to add extra padding values ​​around the output matrix Y, with different padding values ​​corresponding to different upsampling factors. Then, the original convolution kernel is swapped top, bottom, left, and right to obtain a new convolution kernel. Finally, the transformed output matrix Y is multiplied with the new convolution kernel to obtain the corresponding upsampling result.

[0152] The specific formulas for calculating the parameters corresponding to the output image are as follows:

[0153] H out =(H in -1)×stride[0]-2×padding[0]+kernel_size[0]

[0154] W out =(W in-1)×stride[1]-2×padding[1]+kernel_size[1]

[0155] Where H out W is represented as the height value of the output image. out H represents the width of the output image. in W represents the height value of the input image. in The input image width is represented by `stride`, the spacing value is represented by `padding`, the size of the padding is represented by `padding`, and the kernel size is represented by `kernel_size`.

[0156] The Dropout layer primarily prevents overfitting and is connected to a 1×1 convolution to change the number of channels in the input image, setting the number of channels to the number of target categories. This invention cleverly uses pixel classification results corresponding to different upsampling structures to stitch together, thereby continuously approaching the optimal classification result.

[0157] Finally, the method of bilinear interpolation for images that do not conform to the original image size is corrected after each upsampling structure is processed.

[0158] This invention replicates the above-mentioned upsampling structure three times, splices it with Layer 2, Layer 3, and Layer 4 respectively, and processes the results obtained from these three layers respectively.

[0159] The pixel-level segmentation results obtained by the original network algorithm are not satisfactory. A lack of spatial consistency and insufficient consideration of the contextual relationships between pixels are also problems faced by most current image segmentation models. To overcome this deficiency, this invention designs a post-processing algorithm to improve the network structure.

[0160] In the CRF method, two variables are first defined: pixel I and the label X corresponding to this pixel. Both range from the entire image. j X represents the color vector of pixel j. j Let I represent the label of pixel j, and let pixel I and label X satisfy a Gibbs distribution; the formula can be expressed as follows:

[0161]

[0162] In this formula, E(x|I) represents the energy function:

[0163] E(x|I)=∑ i ψ u (x i )+∑ i<j ψ p (x i ,xj )

[0164] The univariate potential function ∑ i ψ u (x i This represents the output from the front-end FCN network. The binary potential function ψ in the formula... p (x i ,x j This can be represented as:

[0165]

[0166] The binary potential function describes the relationship between pixels, ensuring that pixels with similar features are labeled with the same name, while pixels that are "far apart" are labeled differently. This "distance" is related to both the pixel's color and its actual position. The fully connected conditional random field (CRF) calculates the probability of associating a given pixel with all pixels in the image, thus ensuring spatial consistency in image segmentation. This invention employs the fully connected CRF algorithm.

[0167] Step 3: Construct an adaptive runway corner detection module

[0168] 1) The adaptive runway line detection method mainly relies on the traditional line detection method. This process first converts the color RGB image to grayscale, then uses the Canny operator to extract the image edges, then uses the Hough transform to mark the lines, selects the corresponding lines that exceed the point threshold, saves the feature values ​​of the lines in the parameter space, and finally represents the lines in the runway line mask image using the line drawing function.

[0169] The Canny operator edge extraction method calculates the gradient Gx in the X direction and the gradient G in the Y direction of the image using the finite difference method. y Finally, the gradient value at a certain point is calculated. And the gradient direction θ = arctan(G) y / G x Then, non-primary edge lines are filtered out using non-maximum suppression and hysteresis thresholding methods, finally resulting in clear edge lines.

[0170] The Hough transform converts a straight line in a Cartesian coordinate system into data points in a parameter space. It determines whether a line is a principal line within a specified threshold range by counting the number of coordinate points it passes through. The coordinates of the data points include (rho, theta), where rho represents the perpendicular distance between the line and the origin, and theta represents the angle between the line and the epipolar line. The line with rho and theta parameter values ​​is represented on a mask image, and a and b are defined, where a = cos(theta) and b = sin(theta).

[0171] Suppose that three points on a straight line are represented in a Cartesian coordinate system as (x0, y0), (x1, y1), and (x2, y2).

[0172] Where x0 = a*rho, y0 = b*rho;

[0173] x1=x0+1000*(-b), y1=y0+1000*a;

[0174] x2=x0-1000*(-b), y2=y0-1000*a;

[0175] The line is drawn on the mask image based on the values ​​of (x1, y1) and (x2, y2).

[0176] Next, the K-means clustering method is used to cluster based on the angle between the line and the polar axis. K-means clustering is an unsupervised method. First, k cluster centroids are randomly selected, denoted as μ1, μ2...μ... k ∈R n

[0177] For each sample i, calculate the class to which the sample should belong.

[0178] Recalculate the centroid of each class.

[0179] In the method for extracting the runway lines on both sides, the clustering method selects the feature value of the sample points as the angle between the line and the polar axis. Based on the size of the angle, the centroids of the two clusters are determined and used as the feature values ​​of the runway lines on both sides. The pixels corresponding to the runway lines are then outlined on the runway line mask image. Pixels that meet the relevant conditions are marked with a pixel value of 1.

[0180] 2) The minimum envelope quadrilateral method, based on the contour obtained in the previous stage, uses the convex hull algorithm to find a quadrilateral that can contain all the pixels in the segmentation result image, and then draws the minimum quadrilateral that is closest to the airport runway, and draws the minimum quadrilateral in the quadrilateral mask for use in subsequent stages.

[0181] The minimum envelope quadrilateral method employs the convex hull algorithm, where the convex hull is a convex polygon formed by connecting the outermost points of the image. It contains all pixels in the segmentation mask image, and the algorithm stores the boundary points of the convex polygon in an array. Based on the information of the outermost points, it finds the minimum quadrilateral to enclose this convex polygon. Using the four vertices of the minimum quadrilateral, it outlines the minimum quadrilateral within the quadrilateral mask, and the pixel values ​​of points on the quadrilateral's border are marked as 1.

[0182] 3) The mask summation module assigns a pixel value of 2 to pixels that lie both on the runway line and on the smallest quadrilateral. Pixels with a value of 2 are the runway corner points to be found. The coordinates of these corner points are then derived, resulting in the runway corner detection results as shown below. Figure 7 As shown.

[0183] Step 4: Construct a runway line and centerline detection module

[0184] First, a new view plane is defined. Experiments show that the size of the rectangle defined within the newly constructed view plane does not affect the predicted regression of the centerline feature point coordinates on the original image plane, because there is a corresponding planar positional ratio in the perspective transformation. This module defines the size of the rectangle within the new view plane as 200*200.

[0185] This module defines the centerline feature points as the corresponding midpoints of the upper and lower boundaries of the runway in the new view plane.

[0186] Next, four easily identifiable points are selected in the new view plane. In this invention, the four vertices of a rectangle in the new view plane are selected. The coordinates of the four vertices of the rectangle constructed by the new view plane and the coordinates of the four corner points obtained from the track detection in the original image are input into the perspective transformation algorithm to calculate the spatial correspondence between the two planes. The new view plane is then projected onto the original track plane, and the perspective correction matrix M, which completes the projection process, is calculated.

[0187] Finally, based on the definition, two centerline feature points in the new viewing plane are found, and coordinate transformation is performed using the perspective correction matrix M to transform them to the corresponding points in the original image plane, and then marked in the result image.

[0188] Connecting the corresponding points of two centerline feature points in the original plane image yields the runway centerline equation. Similarly, connecting the coordinates of the four corner points obtained from the runway corner detection module in pairs yields the runway line equations on both sides of the airport runway. Finally, the runway lines and centerlines are labeled in the result image, as shown below. Figure 8 As shown.

[0189] Based on the integrity of the corner points of the upper and lower boundaries of the runway in the image to be detected, the processing method is divided into two cases:

[0190] During the descent and landing of an aircraft, the camera is pointed at the runway, and the shape of the runway in the image generally falls into two categories.

[0191] In the first scenario, the corners of both the upper and lower boundaries of the runway can be fully visible, which can be handled using the runway centerline detection method described above.

[0192] The second scenario is that the corner points of the runway's upper boundary are observable, but the corner points of the runway's lower boundary are not fully displayed. The centerline detection measures need improvement.

[0193] First, the corner detection module normally detects the four corner points of the runway (a1,b1), (a2,b2), (c1,d1), and (c2,d2). The corner points are sorted according to their y-values, and the two points with the larger y-values ​​are determined as (c1,d1) and (c2,d2). The corner point with the second largest y-value is determined as (c1,d1).

[0194] Draw a straight line connecting points (a1, b1) and (a2, b2) (i.e., the upper boundary of the runway), and calculate the slope k of the line. Draw a line parallel to the upper boundary of the runway through (c1, d1), and determine the equation of the parallel line using the following formula:

[0195] y-d1=k·(x-c1)

[0196] Next, connect (a1,b1) and (c1,d1), and (a2,b2) and (c2,d2) respectively to determine the equations of the straight lines on both sides of the runway. The line parallel to point (c1,d1) intersects the right (left) boundary line of the runway to obtain the new endpoint (c2). ′ ,d ′ 2). (a1,b1)(a2,b2)(c1,d1)(c2) ′ ,d ′ 2) As the new corresponding endpoints of the trapezoid in the original plane. Then, the new endpoints of the quadrilateral in the original plane and the four endpoints of the rectangle defined in the new view plane are input into the perspective transformation algorithm to obtain the perspective correction matrix M. The two centerline feature points defined in the new view plane are transformed to the corresponding points in the original plane through perspective transformation. The corresponding points are connected to obtain the runway centerline and extended to obtain the final result.

[0197] Step 5: Train the image segmentation network using an actual airport scene image with runways, and continuously optimize the relevant parameters of the model through the loss function.

[0198] Step 6: Convert the weight file of the trained model from PyTorch to ONNX format, and place the ONNX format file on the ONNXRuntime inference framework for accelerated inference to meet real-time requirements.

[0199] To meet real-time requirements, this experiment uses the GPU version of ONNXRuntime for inference, with CUDA version 11.3. In addition, this invention employs model compression, pruning, and quantization techniques on the original model design. Model compression reduces the size and computational requirements of the deep learning model. Pruning removes unnecessary connections or neurons from the neural network to reduce the model size. These unnecessary connections are typically learned during training but have little impact on the final model's inference performance. Quantization adjusts parameters that are not critical during training, simplifying floating-point parameters into integers; these measures improve the practicality of the model approach.

[0200] Step 7: Input the inference results into the runway corner detection module. This module first binarizes the image obtained after semantic segmentation, then uses a clustering algorithm combined with traditional methods to find the runway lines on both sides, and uses a convex hull algorithm to find the minimum envelope quadrilateral. Next, it uses masking and summation to finally obtain the corresponding corner points of the runway and their coordinates. Finally, it calculates the spatial correspondence between the new view plane and the original runway plane to obtain a new perspective transformation matrix. Using the perspective transformation, it obtains the equations of the runway lines and centerline, and labels the runway centerline and runway lines in the result image. The runway centerline detection process is divided into two cases based on the integrity of the corner points at the upper and lower boundaries of the runway. The final detection result image for the first case is shown below. Figure 9 As shown in the figure, the final detection result in the second case is as follows. Figure 10 As shown.

[0201] Experimental verification

[0202] The training data uses a self-built VOC2007 version of the UAV real-world dataset of complex airport runways. The training set, validation set, and test set ratio in this dataset are 75%, 10%, and 15%, respectively, with a total of 10 sets of images and 5000 images in total. The images have been fogged up, so there are 3750 training images, 500 validation images, and 750 test images.

[0203] The training images are input into the improved semantic segmentation network. The optimizer used is the Adam optimizer, the initial learning rate is set to 0.001, the learning rate adjustment strategy is cosine annealing, and the maximum number of iterations is set to 50 epochs.

[0204] The network performance is validated using a validation set at each epoch, and the model weights are saved in a weight file.

[0205] On the validation set, the model achieved an IOU of 0.9171 for the runway target and an average accuracy of 0.9343 for the runway target. The pixel-level image segmentation model processed a single image in as little as 0.02 seconds, the adaptive runway corner detection module processed a single mask image in as little as 0.021 seconds, and the runway line and centerline detection module processed a single image in as little as 0.025 seconds.

Claims

1. A method for detecting and locating airport runways using aerial vision, characterized in that, Specifically, it includes: Step 1: For the airport runway to be tested, select key points of the runway, take real-time photos with a drone to obtain the original images, and annotate them to obtain the corresponding mask images; combine each original image and its corresponding mask image into an image pair. Step 2: Construct a pixel-level image segmentation model that includes a residual network learning module, a transposed convolutional upsampling structure, and a CRF post-processing module; The residual network learning module is an improvement on the original ResNet101 network, consisting of five downsampling layers: an initial convolutional layer, Layer 1, Layer 2, Layer 3, and Layer 4. In particular, the improved residual learning unit is used in Layer 3 and Layer 4 respectively. The original features are extracted by adding dilated convolution operations with different dilation rates and adjusting the stride value. The improved residual learning unit includes Bottleneck1 and Bottleneck2 structures. Bottleneck1 corresponds to the residual learning between two layers during the downsampling convolution process of a shallow network, and Bottleneck2 corresponds to the residual learning between three layers during the downsampling convolution process of a deep network. The transposed convolutional upsampling structure includes, from top to bottom, a transposed convolutional layer, a Dropout layer, and an image restoration layer; The transposed convolutional layer uses a 3×3 convolutional layer to amplify the key feature details of the feature map and suppress information in non-feature regions; the Dropout layer changes the number of channels in the input image to facilitate the stitching of subsequent pixel classification results; the image restoration layer restores the image to its original size using bilinear interpolation. The transposed convolutional upsampling structure was copied into three identical structures and connected to Layer 2, Layer 3, and Layer 4 of the residual network learning module, respectively, to perform upsampling operations of 16x, 8x, and 4x on the feature maps. Finally, the upsampled results are concatenated to obtain the initial mask image, ensuring that it is the same size as the input image; The CRF post-processing module mainly includes a fully connected conditional random field method, i.e., pixel-level dense connections; the optimized mask image is obtained after processing by this module. Step 3: Using the original image of the airport runway and the corresponding mask image, train the pixel-level image segmentation model and continuously optimize the model's weight parameters through the loss function. Step 4: Convert the weight file of the trained pixel-level image segmentation model from PyTorch to ONNX format and save it on the ONNXRuntime inference framework for accelerated inference. The mask image of the runway is obtained by inference for each original image of the runway. Step 5: For the runway mask image used in inference, construct an adaptive runway corner detection module to detect the corners of each inference mask image. Step 6: Determine whether the corner points of the upper and lower boundaries in the runway mask image can be seen completely. If so, proceed to Step 7 and process them using the runway centerline detection method; otherwise, proceed to Step 8. When the drone lands, the runway end furthest from the drone's landing position is defined as the upper boundary. Step 7: Construct a runway line and centerline detection module, and mark the runway lines and centerline on the runway mask image where corner points have been detected; Step 8: When the corner points of the upper boundary of the runway can be seen completely, but the corner points of the lower boundary are not displayed completely, the improved centerline detection measures shall be adopted.

2. The method for detecting and locating airport runways using aerial vision as described in claim 1, characterized in that, The specific process of step five is as follows: Step 501: Detect the edge information of the current inference runway mask image using the Canny operator, detect the position of the straight line using the Hough transform, cluster the angle values ​​formed by the straight line and the polar axis in the parameter space, save the parameter values ​​of the runway lines on both sides after clustering, and outline the runway lines on both sides in the runway mask image. Step 502: Mark the pixel value of the corresponding pixel points of the two runway lines as 1, and save the result image as a runway line mask image; Step 503: For the outlined runway contour, use the convex hull algorithm to draw the smallest quadrilateral that is closest to the runway contour, and mark the pixel value of the boundary point on the smallest quadrilateral as 1. Save the resulting image as a quadrilateral mask image. Step 504: Sum the pixel values ​​of pixels at the same position in the runway line mask image and the quadrilateral mask image, filter out the pixels with a pixel value of 2, which are the corner points of the airport runway, return the corresponding coordinates of the corner points, and mark the corner points in the current runway mask image.

3. The method for detecting and locating airport runways using aerial vision as described in claim 1, characterized in that, The specific detection steps in step seven are as follows: 1) Define a rectangle in the new view plane, and define the two midpoints A and B of the upper and lower boundaries of the rectangle as centerline feature points; 2) Using the perspective transformation algorithm, the perspective correction matrix M is obtained based on the spatial correspondence between the coordinates of the four vertices of the rectangle and the coordinates of the four corner points detected on the runway mask image; The perspective correction matrix M is a 3x3 homogeneous invertible matrix: Where a 11 a 12 a 13 The element controls the affine transformation of elements on the x-axis; a 21 a 22 a 23 The element controls the affine transformation of elements on the y-axis; a 31 a 32 Element controls perspective transformation; a 33 The value is 1; 3) Transform the centerline feature points A and B to their corresponding points A' and B' on the runway mask image using the perspective correction matrix M; 4) Connect the corresponding points A' and B' to obtain the equation of the runway centerline. Similarly, connect the coordinates of the four corner points of the runway mask image in pairs to obtain the equations of the runway lines on both sides of the airport runway. 5) Using the drawing function, based on the runway centerline equation and runway line equation, visualize the runway line and centerline, and finally label them on the result graph.

4. The method for detecting and locating airport runways using aerial vision as described in claim 1, characterized in that, The specific detection steps for step eight are as follows: Step 801: For the four corner points (a1,b1)(a2,b2)(c1,d1)(c2,d2) detected in the runway mask image, sort the corner points according to their y values, select the two points with larger y values ​​as (c1,d1)(c2,d2), and determine the coordinates of the corner point with the second largest y value as (c1,d1). Step 802: Connect the two corner points (a1, b1) and (a2, b2) of the upper boundary of the runway with a straight line, calculate the slope k of the line, and draw a line parallel to the upper boundary of the runway through point (c1, d1). Determine the equation of the parallel line using the following formula: y-d1=k·(x-c1) Step 803: Connect the corner points (a1,b1) and (c1,d1), and (a2,b2) and (c2,d2) respectively to determine the equations of the straight lines on both sides of the track; Step 804: The line parallel to the corner point (c1,d1) intersects the right boundary line of the runway to obtain a new corner point (c'2,d'2). Step 805: Use (a1,b1)(a2,b2)(c1,d1)(c'2,d'2) as the new corresponding corner points in the original mask image; Step 806: Input the four new corresponding corner points and the four endpoints of the custom view plane rectangle into the perspective transformation algorithm to obtain the perspective correction matrix M'. Step 807: The two centerline feature points A and B of the custom view plane rectangle are transformed to the corresponding points in the current runway mask image through the perspective correction matrix M'. Connecting the two corresponding points will yield the centerline of the runway, which can then be extended to obtain the final result image.

Images