Airport runway small target segmentation method based on double-path fusion context network

Abstract

The application belongs to the technical field of image processing, and discloses an airport runway small target segmentation method based on a dual-path fusion context network, which comprises the following steps: an airport runway scene video is shot by using a camera to obtain an airport runway original image and is divided into training data and test data; a small target segmentation network model is constructed by using a neural network with a dual-branch structure; the training set is input into the model for training, and the parameters of the network model are optimized through a loss function; the originally collected airport runway image is input into the trained network model to obtain a mask image after image segmentation is completed. The application adopts the above method, adopts the dual-branch network structure, enhances the global detail perception ability and long-distance dependence modeling ability of the network, improves the extraction efficiency and receptive field of multi-dimensional feature semantic information, effectively optimizes the output result of the semantic segmentation head in combination with two loss functions, and realizes the effect of real-time semantic segmentation on a long-distance airport runway.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for small target segmentation of airport runways based on a dual-path fusion context network. Background Technology

[0002] With the improvement of people's living standards, airplanes have become an indispensable means of transportation for travel. While bringing great convenience to people's lives, airplanes also pose significant hidden dangers. The variability and instability of weather conditions greatly affect aircraft flight safety, resulting in countless accidents. Statistics provided by the International Aircraft Safety Council show that most civil aircraft flight accidents occur during the approach and landing phase. During the approach and landing process, pilots need to endure both physical and psychological stress, making it the most challenging phase for aircraft.

[0003] Developing advanced intelligent navigation systems for aircraft has become a key focus in the modern aviation industry, aiming to improve safety, efficiency, and situational awareness. The rapid segmentation and identification method for small targets on airport runways aims to address the safety hazards arising from the difficulty in accurately identifying airport runway targets during precise approach and landing, due to rapid changes in the size and drastic changes in viewing angle of airport targets, and the high real-time requirements.

[0004] Currently, commonly used image processing network algorithms are mainly based on deep learning and traditional algorithms. Traditional algorithms primarily include edge detection and template matching methods, utilizing Canny and Harris operators for corner detection. However, these methods cannot quickly detect runway and corner information, failing to meet real-time requirements. Furthermore, traditional methods yield poor results and are unsuitable for the detailed environments of real-world airport runways.

[0005] Deep learning methods for real-time segmentation of small targets on airport runways are still in their early stages. The main deep learning approaches include those based on YOLO-V5, Mask R-CNN, and DeepLabv3 semantic segmentation networks. However, the existing network models struggle to meet the demands of real-time runway segmentation and accurate corner detection, especially for small targets on distant runways where pixel segmentation remains unclear. Furthermore, the segmentation results are highly dependent on the scene and do not account for interference from complex environments.

[0006] Therefore, designing an accurate and fast image segmentation algorithm for airport runway scenarios has become an urgent problem to be solved. Summary of the Invention

[0007] The purpose of this invention is to provide a method for segmenting small targets on airport runways based on a dual-path fusion context network. It employs a dual-branch network structure: the detail branch introduces a Transformer-based long-range attention module to enhance the network's global detail perception and long-range dependency modeling capabilities; the semantic branch embeds a hybrid convolutional context to improve the extraction efficiency and receptive field of multi-dimensional feature semantic information; during training, two loss functions are combined to effectively optimize the output of the semantic segmentation head, achieving real-time semantic segmentation of distant airport runways.

[0008] To achieve the above objectives, this invention provides a method for small target segmentation on airport runways based on a dual-path fusion context network, comprising the following steps: Step S1: Use a camera to capture video of the airport runway scene, and control the pilot's flight altitude within the altitude range of the approach and landing phase; extract frames from the captured video to obtain the original image of the airport runway, and randomly divide it into training data and test data. Step S2: A small target segmentation network model is constructed using a neural network with a dual-branch structure. The structure includes detail branches, semantic branches, guidance optimization branches, and a fusion dual-branch module. Step S3: Input the long-distance airport runway images from the training set into the small target segmentation network model for training, and optimize the parameters of the airport runway small target segmentation network model through the loss function; Step S4: Input the original acquired airport runway image into the trained airport runway small target segmentation network model to obtain the mask image after image segmentation.

[0009] Preferably, in step S2, the detail branch consists of four depthwise separable convolutional layers and a long-range attention module based on the Transformer structure; First, the image to be processed is input into the detail branch, and detail information is learned and downsampled through the first four depthwise separable convolutional layers. Among them, the depthwise separable convolutional layers introduce DW convolution and PW convolution operations on the basis of using 3*3 convolution operation with stride of 2. Then, the merged result is fed into the long-range attention module GCT based on the Transformer structure through a skip connection, so that the detail branch has the ability to capture long-range global context information.

[0010] The preferred implementation process of the long-range attention module based on the Transformer structure is as follows: For an image of size H*W*3, it is divided into image blocks of size 4*4. The dimensions of Q, K, and V in the Transformer module are N*C, where N is of size H*W, which is the sequence length. The efficient multi-head self-attention mechanism allows the model to focus on information of different dimensions in the sequence and can capture multiple feature information such as local and global features, object relationships, and texture in parallel, thereby improving the model's ability to understand images, as shown below: ; in, This refers to the query matrix; This refers to the key matrix; Refers to a value matrix; This indicates the dimension of the query, key, and value vectors, and acts as a scaling factor. Normalize each query dimension; H is the image height; W is the image width; C is the number of channels; To reduce the amount of computation, The size is transformed, first by reshaping to become ( , The size of each attention head sequence becomes... As shown below: ; in, This refers to the result of the transformation; This refers to resizing and reshaping the vector result; This refers to the shape scaling factor; This refers to the number of vector sequences; Then, concat is used to integrate context information from different dimensions, and finally, the number of channels is changed to C through the Linear layer, as shown below: ; in, This refers to the operation of changing the number of channels.

[0011] Preferably, the semantic branch consists of one Stem layer and three GEL layers, which are connected to a hybrid convolutional context module for multi-dimensional and multi-scale feature semantic extraction. The Stem layer is used for downsampling and feature learning of images. It is placed in the first layer of the semantic branch and is divided into two paths for downsampling and feature learning. Effective downsampling is achieved by using a 3*3 convolutional layer and a max pooling layer, respectively. The GEL layer is an aggregation expansion layer, which includes a skip connection structure that is divided into two paths for processing. One path first uses a 3*3 convolution to map the features to a higher dimension, with a stride of 2, and then uses two consecutive 3*3 depthwise convolutions on each individual output channel to increase the receptive field of the semantic branch. Finally, a 1*1 convolution is used to project the feature map into the low-channel space, and it is connected to the residual structure composed of the other path's DW convolution and 1*1 convolution. The hybrid convolutional context module, consisting of depthwise separable convolutional layers, two dilated convolutional layers, and an asymmetric convolutional layer, is used to improve the computational efficiency and receptive field size of the network.

[0012] Preferably, the hybrid convolutional context module adopts depthwise separable convolution and asymmetric convolution proposed by the E-Net network to improve computational efficiency; the depthwise separable convolution decomposes the standard convolution into DW depthwise convolution and PW pointwise convolution. When the number of channels is large, the DW convolution and PW convolution are combined, and the Spatial Dropout strategy is used to reduce the amount of computation. Multi-scale depth-separable convolutional layers utilize three DW convolutional modules with kernel sizes of 1*1, 3*3, and 5*5 to calculate features in different receptive fields, thereby improving the receptive field. After concatenating the feature maps along the channel dimension, PW convolution is used to reduce the dimensionality to the original number of channels. After passing through the ReLU activation function, the feature maps are added to the original input features to obtain the feature maps.

[0013] Preferably, the hybrid convolutional context module also employs dilated convolution and asymmetric convolution; Among them, dilated convolution improves the receptive field of semantic branches without losing feature map resolution, and uses two dilation rates of 2 and 4 to capture contextual information of different scales of the input image. After the first dilated convolutional layer, another dilated convolutional layer is added, and an alternating dilation rate sequence is used so that the subsequent dilated convolutional layers can sample the pixels skipped by the previous layer. Asymmetric convolution helps semantic branches capture features at different directions and scales, with kernels of 1*7 and 7*1 size.

[0014] The preferred optimization branch consists of multiple loss functions, specifically including the median frequency-balanced cross-entropy loss function and the log-scale invariant loss function; The median frequency-balanced cross-entropy loss function is as follows: ; in, Indicates the number of species; 0 or 1 indicates that... Is the pixel at position 1? kind; Indicates the pixel as the first The probability of a class; Indicates assignment to Median frequency weights for classes; The log-scale invariant loss function is as follows: ; in, This represents the true value of the i-th sample; This represents the predicted value of the i-th sample; Indicates the total number of samples; Avoid square roots being zero; The total loss function is as follows: ; in, and The weights are a mixture of the two loss functions.

[0015] Preferably, a dual-branch system is used to receive and process information from detail branches and semantic branches; Processing the detail branch yields two feature images: one obtained through depthwise separable convolutional layers. One feature map is reduced by a factor of 4 in both width and height after convolutional and average pooling layers, and then used as weights to perform dot product with the semantic branch feature map. Processing the semantic branch also yields two feature images, one of which is processed in the same way as the detail branch. The feature map, and the other is the feature image size restored by upsampling. Used to perform a dot product with the result of the detail branch feature map; Finally, the two pieces of information are directly added together to obtain the final fusion result.

[0016] Preferably, in step S3, the long-distance airport runway images from the training set are input into the small target segmentation network model for training. The parameters of the real-time small target segmentation network model for airport runways are optimized using a loss function. The specific process is as follows: Step S31: Based on the PyTorch 2.0 framework, build a small target segmentation network model, input the long-distance airport runway images of the training set into the network, and train it on the GPU. Step S32: During training, randomly crop the image to 256*256 and simultaneously add data augmentation operations such as vertical flipping and horizontal flipping. The training batch size is set to 16, the Adam optimizer is used, and the initial learning rate is set to 2×10. -4 A cosine learning rate decay strategy is used, where the learning rate gradually decays to 0.01 times the initial learning rate. Step S33: Train the airport runway small target segmentation network model, and continuously optimize the model by using the loss function of the guided optimization branch to obtain the final weight parameters of the model.

[0017] Therefore, the present invention employs the above-mentioned airport runway small target segmentation method based on dual-path fusion context network, and the beneficial effects are as follows: (1) The real-time segmentation method for small targets on airport runways proposed in this invention is well adapted to the problem of inaccurate segmentation of small targets at long distances encountered by aircraft during the approach and landing phase. This architecture can capture the details and background information of the target more comprehensively by simultaneously utilizing local features and global context information, thereby improving the segmentation accuracy of small targets. It also provides more reliable technical support for aviation safety and has broad application prospects and important practical significance.

[0018] (2) In the detail branch, the results are input to the long-distance attention module based on the Transformer structure through skip connections, which enhances the network’s global detail-aware context extraction capability; in the semantic branch, semantic information is captured step by step through the stem module and the GEL module, and then multi-dimensional feature semantic information in the image is captured through the hybrid convolutional context module.

[0019] (3) In view of the complex characteristics of airport runway scene, the present invention designs a loss function in the guiding optimization branch, optimizes the segmentation result of small target on airport runway, and effectively utilizes the feature information of detail branch and semantic branch by using the fusion dual-branch structure, thereby improving the global receptive field of the network.

[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall network structure of the airport runway small target segmentation method based on dual-path fusion context network of the present invention; Figure 2 This is a schematic diagram of the long-distance attention module in the detailed branch of this invention; Figure 3 This is a schematic diagram of the stem module in the semantic branch of this invention; Figure 4 This is a schematic diagram of the structure of the GEL module in the semantic branch of this invention; Figure 5 This is a schematic diagram of the hybrid context convolution module in the semantic branch of this invention; Figure 6 This is a schematic diagram of the structure of the multi-scale depth-separable convolutional layer in the semantic branch of this invention; Figure 7 This is a schematic diagram of the dual-branch structure of the present invention; Figure 8 This is a comparison chart of the processing results of this invention on a self-built airport runway scene dataset. Each row corresponds to the output effect of different network models. Among them, (a) is the segmentation result of BiSeNetV3 algorithm; (b) is the segmentation result of DNLNet algorithm; (c) is the segmentation result of SETR algorithm; (d) is the segmentation result of SegFormer algorithm; (e) is the segmentation result of the algorithm (Ours) proposed in this invention; and (f) is the segmentation result of ground truth (GT) image annotation. Figure 9 The diagram shows a comparison of the processing results of this invention on the Cityscapes dataset; where (a) the original input images corresponding to each group of scenes; (b) the semantic segmentation results of the BiSeNetV2 algorithm; (c) the semantic segmentation results of the STDC1 algorithm; (d) the semantic segmentation results of the STDC2 algorithm; (e) the semantic segmentation results of the CGNet algorithm; (f) the semantic segmentation results of the method proposed in this paper; and (h) the ground truth (GT) segmentation results of the corresponding scenes. Detailed Implementation

[0022] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] This invention proposes a fast low-light image sharpening algorithm based on depthwise separable convolution. It improves upon the zero-reference Zero-DCE algorithm by replacing the Zero-DCE feature extraction network with a U-Net structure. The network uses multi-scale convolutional modules and attention-like convolutional modules to enhance its feature extraction capabilities. Then, the pixel values ​​of the low-light image are effectively adjusted by adjusting the curve. Finally, various loss functions are used to optimize the network model parameters.

[0024] To address the issue of lost feature information for small targets on airport runways at long distances, a detail branch with the ability to capture global context information over long distances is proposed. To address the rapidly changing and irregular size of airport runway targets, a semantic branch with a hybrid convolutional context module is designed. For the complex characteristics of airport runway scenes, a guided optimization branch and loss function are designed. A fused dual-branch structure effectively utilizes the feature information from both the detail and semantic branches. Finally, an edge-aware post-processing module is designed to obtain runway edge-aware information, including corner coordinates and centerline information.

[0025] like Figure 1 As shown, the present invention provides a method for small target segmentation on airport runways based on a dual-path fusion context network, comprising the following steps: Step S1: Use a camera to capture video of the airport runway scene, and control the pilot's flight altitude within the altitude range of the approach and landing phase; extract frames from the captured video to obtain the original image of the airport runway, and randomly divide it into training data and test data.

[0026] Step S2: A small target segmentation network model is constructed using a neural network with a dual-branch structure. The structure includes detail branches, semantic branches, guidance optimization branches, and a fusion dual-branch module.

[0027] Step S21, the detail branch, consists of four depthwise separable convolutional layers and a long-range attention module based on the Transformer structure.

[0028] First, the image to be processed is input into the detail branch, where detail information is learned and downsampled through the first four depthwise separable convolutional layers. In addition to using 3x3 convolution operations with a stride of 2, the depthwise separable convolutional layers also introduce DW and PW convolution operations.

[0029] Then, the merged result is fed into a Transformer-based long-range attention module (GCT) via skip connections. Figure 2 As shown, this enables detailed branches to capture global context information over long distances.

[0030] For an image of size H*W*3, it is divided into image blocks of size 4*4. The dimensions of Q, K, and V in the Transformer module are N*C, where N is of size H*W, which is the sequence length. The efficient multi-head self-attention mechanism allows the model to focus on information of different dimensions in the sequence and can capture multiple feature information such as local and global features, object relationships, and texture in parallel, thereby improving the model's ability to understand images, as shown below: ; in, This refers to the query matrix; This refers to the key matrix; Refers to a value matrix; This indicates the dimension of the query, key, and value vectors, and acts as a scaling factor. Normalize each query dimension; H is the image height; W is the image width; C is the number of channels.

[0031] To reduce the amount of computation, The size is transformed, first by reshaping to become ( , The size of each attention head sequence becomes... As shown below: ; in, This refers to the result of the transformation; This refers to resizing and reshaping the vector result; This refers to the shape scaling factor; This refers to the number of vector sequences.

[0032] Then, concat is used to integrate context information from different dimensions, and finally, the number of channels is changed to C through the Linear layer, as shown below: ; in, This refers to the operation of changing the number of channels.

[0033] Setting R size to 16 will cause non-overlapping patch merging to... The feature image of size is transformed by merging 2*2 non-overlapping patch blocks. The feature map is large, but it loses local continuity information, which may contain texture information.

[0034] Therefore, by merging overlapping blocks using K=3, S=2, and P=1, the local information between adjacent blocks is preserved, thus solving the problem of loss of original local continuity. At the same time, the size of the feature image is reduced, ensuring that the segmentation speed of the semantic segmentation network is improved.

[0035] The GCT module employs an efficient multi-head attention mechanism and an overlapping patch merging method, which reduces computational cost while ensuring the model's powerful representational capabilities.

[0036] Step S22, the semantic branch, consists of a Stem layer and three GEL layers, and is connected to the Hybrid Convolutional Context Module (HCCM) for multi-dimensional and multi-scale feature semantic extraction.

[0037] The Stem layer, located in the first layer of the semantic branch, is used for downsampling and feature learning of images. It is divided into two paths for downsampling and feature learning. Figure 3 As shown, 3*3 convolutional layers and max pooling layers are used for effective downsampling, which can improve the network's computational efficiency and feature extraction capabilities.

[0038] The GEL layer is a clustered extension layer, and its structure is as follows: Figure 4As shown, it mainly includes a skip connection structure, which is divided into two paths for processing. One path first uses a 3*3 convolution to map the features to a higher dimension, sets the stride to 2, and then uses two consecutive 3*3 depth convolutions on each individual output channel to increase the receptive field of the semantic branch. Finally, a 1*1 convolution is used to project the feature map to the low channel space, and it is connected with the residual structure composed of the other path DW convolution and 1*1 convolution.

[0039] Semantic branches require an increased receptive field to effectively embed global contextual information. A hybrid convolutional context module, consisting of depthwise separable convolutional layers, two dilated convolutional layers, and an asymmetric convolutional layer, is used to improve the network's computational efficiency and receptive field size. Its structure is as follows: Figure 5 As shown.

[0040] The hybrid convolutional context module employs depthwise separable convolution and the asymmetric convolution proposed by the E-Net network to improve computational efficiency. Depthwise separable convolution decomposes standard convolution into DW depthwise convolution (channel-wise spatial filtering) and PW pointwise convolution (cross-channel combination), significantly reducing the number of parameters and computational cost (FLOPs). This is particularly effective when dealing with a large number of channels, where the combination of DW and PW convolutions can significantly improve computational efficiency. Spatial Dropout is also employed to further reduce computational cost.

[0041] This invention modifies the original single-scale feature extraction structure of depthwise separable convolutions and designs a multi-scale depthwise separable convolutional layer to improve the receptive field. The specific structure is as follows: Figure 6 As shown, three DW convolutional modules with kernel sizes of 1*1, 3*3, and 5*5 are used to calculate features for different receptive fields. Different kernel sizes can enhance the network's contextual understanding, thus designing a multi-scale depth-separable convolutional layer structure. After concatenating the feature maps along the channel dimension, PW convolution is used to reduce the dimensionality to the original number of channels. After passing through the ReLU activation function, the feature maps are added to the original input features to obtain the final feature map.

[0042] Building upon this, dilated convolution and asymmetric convolution are also employed. Dilated convolution can effectively improve the receptive field of semantic branches without sacrificing feature map resolution, using two dilation rates of 2 and 4 to capture contextual information at different scales in the input image. Asymmetric convolution can help semantic branches flexibly capture features at different directions and scales. In the semantic segmentation task of airport runways, the shape of the runway is diverse and irregular. Asymmetric convolution helps to extract semantic information from feature images from different angles. The original 7*7 convolution kernel is improved to 1*7 and 7*1 kernels.

[0043] If the hybrid convolutional context module uses dilated convolutional layers with an inappropriate combination of dilation rates, a grid effect will occur. That is, each weight element of the convolutional kernel will only touch a portion of the discrete pixels in the input feature map, while skipping other pixels. Furthermore, sparse sampling will result in uncovered regions between the receptive fields corresponding to different weights.

[0044] Therefore, to avoid the grid effect, a dilated convolutional layer is added after the first dilated convolutional layer, and an alternating dilation rate sequence is used so that subsequent dilated convolutional layers can sample pixels skipped by the previous layer, thereby ensuring that the receptive field of the overall semantic branch meets the requirement of dense sampling.

[0045] Step S23: The guided optimization branch consists of multiple loss functions, specifically including the median frequency-balanced cross-entropy loss function and the log-scale invariant loss function. This guided optimization branch is added only during network training and not during network inference.

[0046] The median frequency-balanced cross-entropy loss function is as follows: ; in, Indicates the number of species; 0 or 1 indicates that... Is the pixel at position 1? kind; Indicates the pixel as the first The probability of a class; Indicates assignment to Median frequency weights for classes.

[0047] Median frequency balancing refers to the practice of weighting the loss differently based on the true class when the number of pixels for each class varies significantly in the training set (e.g., road, sky, and building pixels dominate in the CamVid dataset). This promotes frequency balance by assigning weights to each class based on the median class frequency calculated across the entire training set, divided by the average frequency of each class. This means that larger classes in the training set have weights less than 1, while the smallest class has the highest weight. This allows for increased segmentation weights for smaller targets, enhancing the network's attention to them.

[0048] The log-scale invariant loss function is as follows: ; in, This represents the true value of the i-th sample; This represents the predicted value of the i-th sample; Indicates the total number of samples; Avoid square roots that are zero.

[0049] Log-scale invariant loss can improve the model's performance in complex scenarios, enabling the model to better adapt to changes in airport runway size and minimize pixel position deviations.

[0050] The total loss function is as follows: ; in, and The weights are a mixture of two loss functions; this invention sets... It is 0.75. It is 0.25.

[0051] Step S24: Merge the two branches to receive and process information from the detail branch and the semantic branch.

[0052] The specific structure of the fusion dual-part is as follows Figure 7 As shown, two feature images are obtained by processing the detail branch, one of which is obtained through a depthwise separable convolutional layer. One approach is to reduce the width and height of the feature map by a factor of 4 after passing through convolutional and average pooling layers, and then use these reductions as weights to perform dot product with the semantic branch feature map.

[0053] Processing the semantic branch also yields two feature images, one of which is processed in the same way as the detail branch. The feature map, and the other is the feature image size restored by upsampling. Used for dot product with the feature map results of detail branches.

[0054] Finally, the two parts of information are directly added together to obtain the final fusion result, which effectively utilizes the information from the detail branch and the semantic branch.

[0055] Step S3: Input the long-distance airport runway images from the training set into the small target segmentation network model for training, and optimize the parameters of the airport runway small target segmentation network model through the loss function.

[0056] Step S31: The proposed network model is built on the PyTorch 2.0 framework. The long-distance airport runway images from the training set are input into the network, and training is performed on a GPU. The GPU used is an RTX 3080ti with 12GB of VRAM. The experimental equipment uses Ubuntu 18.04 64-bit as the operating system, has 64GB of RAM, and uses CUDA version 11.7.

[0057] Step S32: During training, randomly crop the images to 256*256 and simultaneously perform a series of data augmentation operations such as vertical and horizontal flipping. Set the batch size for training the network model to 16, use the Adam optimizer, and set the initial learning rate to 2×10. -4 During training, a cosine learning rate decay strategy is used, and the learning rate is gradually decayed to 0.01 times the initial learning rate.

[0058] Step S33: Train the airport runway small target segmentation network model, and continuously optimize the model by using the loss function of the guided optimization branch to obtain the final weight parameters of the model.

[0059] Step S4: Input the original acquired airport runway image into the trained airport runway small target segmentation network model to obtain the mask image after image segmentation.

[0060] Example This embodiment uses the Cityscapes and Camvid image datasets, as well as a real-world airport runway image dataset, for experiments.

[0061] The self-built airport runway dataset contains a total of 4032 images, with 3144 images in the training set, 564 images in the test set, and 324 images in the validation set. The Cityscapes dataset is a public dataset for urban street scene semantic segmentation tasks, focusing on semantic understanding of high-resolution images in urban environments. This dataset contains 5000 high-resolution images (2048*1024 pixels) from 50 different cities, divided into training, validation, and test sets of 2975, 500, and 1525 images respectively. The Cityscapes dataset covers 19 semantic labels, such as roads, buildings, vehicles, pedestrians, and traffic signs, suitable for object detection, semantic segmentation, and scene understanding in urban environments. The Camvid dataset contains 701 high-resolution images, covering 11 common categories of urban scenes, including roads and pedestrians, divided into a training set (468 images), a validation set (100 images), and a test set (133 images).

[0062] This embodiment compares the algorithm with 10 other algorithms, including CGNet, STDC, and DDRNet. The experimental dataset is a self-built airport runway scene dataset. The comparison results are as follows: Figure 8 As shown, the experimental results revealed that in small target recognition, other models had many false detections and incorrectly segmented pixels, and the runway edges were also distorted and unclear.

[0063] This invention also conducted comparative experiments on different small reference objects such as cars, streetlights, and motorcycles on the Cityscapes dataset. Specific experimental comparison results are shown below. Figure 9 As shown in the figure. Experiments on various datasets have demonstrated that the algorithm proposed in this invention can perform semantic segmentation well for small targets on airport runways at long distances.

[0064] Therefore, this invention adopts the above-mentioned airport runway small target segmentation method based on dual-path fusion context network. It adopts a dual-branch network structure. The detail branch introduces a Transformer-based long-distance attention module to enhance the network's global detail perception ability and long-distance dependency modeling ability. The semantic branch embeds a hybrid convolutional context to improve the extraction efficiency and receptive field of multi-dimensional feature semantic information. During training, the output results of the semantic segmentation head are effectively optimized by combining two loss functions to achieve the effect of real-time semantic segmentation of distant airport runways.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for small target segmentation on airport runways based on a dual-path fusion context network, characterized in that, Includes the following steps: Step S1: Use a camera to capture video of the airport runway scene, and control the pilot's flight altitude within the altitude range of the approach and landing phase; extract frames from the captured video to obtain the original image of the airport runway, and randomly divide it into training data and test data. Step S2: A small target segmentation network model is constructed using a neural network with a dual-branch structure. The structure includes detail branches, semantic branches, guidance optimization branches, and a fusion dual-branch module. Step S3: Input the long-distance airport runway images from the training set into the small target segmentation network model for training, and optimize the parameters of the airport runway small target segmentation network model through the loss function; Step S4: Input the original acquired airport runway image into the trained airport runway small target segmentation network model to obtain the mask image after image segmentation.

2. The airport runway small target segmentation method based on dual-path fusion context network according to claim 1, characterized in that, In step S2, the detail branch consists of four depthwise separable convolutional layers and a long-range attention module based on the Transformer structure; First, the image to be processed is input into the detail branch, and detail information is learned and downsampled through the first four depthwise separable convolutional layers. Among them, the depthwise separable convolutional layers introduce DW convolution and PW convolution operations on the basis of using 3*3 convolution operation with stride of 2. Then, the merged result is fed into the long-range attention module GCT based on the Transformer structure through a skip connection, so that the detail branch has the ability to capture long-range global context information.

3. The airport runway small target segmentation method based on dual-path fusion context network according to claim 2, characterized in that, The specific implementation process of the long-range attention module based on the Transformer architecture is as follows: For an image of size H*W*3, it is divided into image blocks of size 4*4. The dimensions of Q, K, and V in the Transformer module are N*C, where N is of size H*W, which is the sequence length. The efficient multi-head self-attention mechanism allows the model to focus on information of different dimensions in the sequence and can capture multiple feature information such as local and global features, object relationships, and texture in parallel, thereby improving the model's ability to understand images, as shown below: ； in, This refers to the query matrix; This refers to the key matrix; Refers to a value matrix; This indicates the dimension of the query, key, and value vectors, and acts as a scaling factor. Normalize each query dimension; H is the image height; W is the image width; C is the number of channels; To reduce the amount of computation, The size is transformed, first by reshaping to become ( , The size of each attention head sequence becomes... As shown below: ； in, This refers to the result of the transformation; This refers to resizing and reshaping the vector result; This refers to the shape scaling factor; This refers to the number of vector sequences; Then, concat is used to integrate context information from different dimensions, and finally, the number of channels is changed to C through the Linear layer, as shown below: ； in, This refers to the operation of changing the number of channels.

4. The airport runway small target segmentation method based on dual-path fusion context network according to claim 3, characterized in that, The semantic branch consists of one Stem layer and three GEL layers, which are connected to a hybrid convolutional context module to extract multi-dimensional and multi-scale feature semantics. The Stem layer is used for downsampling and feature learning of images. It is placed in the first layer of the semantic branch and is divided into two paths for downsampling and feature learning. Effective downsampling is achieved by using 3x3 convolutional layers and max pooling layers respectively; The GEL layer is an aggregation expansion layer, which includes a skip connection structure that is divided into two paths for processing. One path first uses a 3*3 convolution to map the features to a higher dimension, with a stride of 2, and then uses two consecutive 3*3 depthwise convolutions on each individual output channel to increase the receptive field of the semantic branch. Finally, a 1*1 convolution is used to project the feature map into the low-channel space, and it is connected to the residual structure composed of the other path's DW convolution and 1*1 convolution. The hybrid convolutional context module, consisting of depthwise separable convolutional layers, two dilated convolutional layers, and an asymmetric convolutional layer, is used to improve the computational efficiency and receptive field size of the network.

5. The airport runway small target segmentation method based on dual-path fusion context network according to claim 4, characterized in that, The hybrid convolutional context module employs depthwise separable convolution and the asymmetric convolution proposed by the E-Net network to improve computational efficiency. The depthwise separable convolution decomposes the standard convolution into DW depthwise convolution and PW pointwise convolution. When the number of channels is large, the DW convolution and PW convolution are combined, and the Spatial Dropout strategy is used to reduce the amount of computation. Multi-scale depth-separable convolutional layers utilize three DW convolutional modules with kernel sizes of 1*1, 3*3, and 5*5 to calculate features in different receptive fields, thereby improving the receptive field. After concatenating the feature maps along the channel dimension, PW convolution is used to reduce the dimensionality to the original number of channels. After passing through the ReLU activation function, the feature maps are added to the original input features to obtain the feature maps.

6. The airport runway small target segmentation method based on dual-path fusion context network according to claim 5, characterized in that, The hybrid convolutional context module also employs dilated convolution and asymmetric convolution; Among them, dilated convolution improves the receptive field of semantic branches without losing feature map resolution, and uses two dilation rates of 2 and 4 to capture contextual information of different scales of the input image. After the first dilated convolutional layer, another dilated convolutional layer is added, and an alternating dilation rate sequence is used so that the subsequent dilated convolutional layers can sample the pixels skipped by the previous layer. Asymmetric convolution helps semantic branches capture features at different directions and scales, with kernels of 1*7 and 7*1 size.

7. The airport runway small target segmentation method based on a dual-path fusion context network according to claim 6, characterized in that, The optimization branch is guided by a variety of loss functions, including the median frequency-balanced cross-entropy loss function and the log-scale invariant loss function. The median frequency-balanced cross-entropy loss function is as follows: ； in, Indicates the number of species; 0 or 1 indicates that... Is the pixel at position 1? kind; Indicates the pixel as the first The probability of a class; Indicates assignment to Median frequency weights for classes; The log-scale invariant loss function is as follows: ； in, This represents the true value of the i-th sample; This represents the predicted value of the i-th sample; Indicates the total number of samples; Avoid square roots being zero; The total loss function is as follows: ； in, and The weights are a mixture of the two loss functions.

8. The airport runway small target segmentation method based on dual-path fusion context network according to claim 7, characterized in that, The dual-branch fusion is used to receive and process information from detail branches and semantic branches; Processing the detail branch yields two feature images: one obtained through depthwise separable convolutional layers. One feature map is reduced by a factor of 4 in both width and height after convolutional and average pooling layers, and then used as weights to perform dot product with the semantic branch feature map. Processing the semantic branch also yields two feature images, one of which is processed in the same way as the detail branch. The feature map, and the other is the feature image size restored by upsampling. Used to perform a dot product with the result of the detail branch feature map; Finally, the two pieces of information are directly added together to obtain the final fusion result.

9. The airport runway small target segmentation method based on dual-path fusion context network according to claim 7, characterized in that, In step S3, the long-distance airport runway images from the training set are input into the small target segmentation network model for training. The parameters of the real-time small target segmentation network model for airport runways are optimized using a loss function. The specific process is as follows: Step S31: Based on the PyTorch 2.0 framework, build a small target segmentation network model, input the long-distance airport runway images of the training set into the network, and train it on the GPU. Step S32: During training, randomly crop the image to 256*256 and simultaneously add data augmentation operations such as vertical flipping and horizontal flipping. The training batch size is set to 16, the Adam optimizer is used, and the initial learning rate is set to 2×10. -4 A cosine learning rate decay strategy is used, where the learning rate gradually decays to 0.01 times the initial learning rate. Step S33: Train the airport runway small target segmentation network model, and continuously optimize the model by using the loss function of the guided optimization branch to obtain the final weight parameters of the model.

Images