A semantic segmentation method for drone-viewed videos based on deep learning

Abstract

The present invention belongs to the field of drone vision, and relates to a method for semantic segmentation of drone-perspective video based on deep learning. Aiming at the problem of image semantic segmentation, the present invention designs an "encoder-decoder" asymmetric network structure, wherein the encoder stage fuses channel splitting (Channel Split) and channel reorganization (Channel Shuffle) to improve the Bottleneck structure for downsampling and feature extraction, and the decoder stage is based on a spatial pyramid multi-feature fusion module to extract rich features and fuse them, and finally upsampling is performed to obtain the segmentation result. Subsequently, in view of the problem of video semantic segmentation, the image segmentation model designed by the present invention is used as a segmentation module for video semantic segmentation, and the optical flow method is combined to improve the key frame selection strategy and perform feature transfer, thereby reducing redundancy and speeding up video segmentation.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle vision and relates to a method for semantic segmentation of unmanned aerial vehicle perspective video based on deep learning. Background Art

[0002] In the field of computer vision, the main applications of neural networks are image recognition, target localization and detection, and semantic segmentation. Semantic segmentation is a typical computer vision problem that involves taking some raw data (such as two-dimensional images) as input and converting them into masks with highlighted regions of interest. In other words, in semantic segmentation, we need to divide the visual input into different semantically interpretable categories. For example, we may need to classify all pixels in the image that belong to cars and paint these pixels a certain color. Today, semantic segmentation is one of the key issues in computer vision, and more and more application scenarios require accurate and efficient segmentation technology, such as autonomous driving, indoor navigation, medical image analysis, etc. In recent years, the rise of deep learning technology has brought about earth-shaking changes in related fields. Therefore, many computer vision problems have begun to be solved using deep architectures.

[0003] In intelligent scenarios such as autonomous driving and drones, semantic segmentation can assist basic functions such as obstacle avoidance, tracking, and path planning. For example, in autonomous driving, it is necessary to perceive and analyze the driving environment in detail, and at the same time focus on pedestrians, vehicles, roads, traffic signs and other areas. Semantic segmentation is essential. Compared with sensors such as lidar, cameras have low power consumption, small size, low cost, wide adaptability, and can obtain rich information. Therefore, image and video semantic segmentation based on vision has important application value and practical significance for intelligent environmental perception.

[0004] Video is essentially a continuous single image. The purpose of video semantic segmentation is to use spatial and temporal features to perform image semantic segmentation on each video frame. However, if each frame of the video is fed into the image segmentation model, it will bring huge computational overhead. At the same time, due to the rapid changes in the scene in the video, the uncertainty of the segmentation result will also occur. Therefore, video semantic segmentation needs to be further developed on the basis of image semantic segmentation. At present, the main research directions are to use the temporal information of the video to ensure the segmentation accuracy and to reduce video redundancy and improve the segmentation speed.

[0005] Regarding direction 1, Fayyaz et al. proposed using the classic LSTM module to learn the temporal features of the video and assist in the propagation of spatial features, but this will seriously affect the running speed. The Netwarp structure proposed by Gadde et al. uses optical flow to fuse the features of the previous frame with the features of the current frame to enhance the features. The STGRU module proposed by Nilsson et al. uses the previous and next frames of the current frame as input for training based on the image semantic segmentation model, and uses optical flow to transfer the segmentation results of the previous and next frames to the current frame. This multi-frame information fusion method is conducive to strengthening feature representation and effectively improving segmentation accuracy.

[0006] For the second direction, Zhu et al. proposed the classic DFF structure combined with a fixed-interval keyframe selection strategy. In the feature extraction process of the current frame, the features of the previous keyframe are directly converted using the optical flow method. The computational complexity of the optical flow method is much smaller than that of feature extraction, so the model efficiency is greatly improved. Based on the DFF structure, Xu et al. proposed DVSNet, which adds a shallow neural network structure to determine whether the current frame is a keyframe. At the same time, the model accuracy can be adjusted by changing the confidence threshold of the discriminant network. Summary of the invention

[0007] The present invention provides a video semantic segmentation algorithm based on drone vision. At present, the main application field of semantic segmentation is still in the field of driverless cars, and there are very few applications in drones. In recent years, drones have become more and more popular due to their advantages such as strong maneuverability and economic affordability. Starting from the needs of real application scenarios, the present invention considers the accuracy and speed of the model at the same time, and studies the problem of real-time semantic segmentation of images and videos under limited computing and storage conditions. By designing a lightweight deep learning model, the focus is on studying the simplification of network structure, reduction of model parameters, fusion of global and local feature information, association of spatiotemporal features of video frames, selection of key frames, etc., and the superiority of the proposed method is verified through theoretical analysis, simulation experiments, data comparison, etc.

[0008] The technical solution of the present invention:

[0009] Aiming at the problem of image semantic segmentation, an "encoder-decoder" asymmetric network structure is designed, in which the encoder stage fuses channel splitting (Channel Split) and channel reorganization (Channel Shuffle) to improve the Bottleneck structure for downsampling and feature extraction. The decoder stage is based on a spatial pyramid multi-feature fusion module to extract rich features and fuse them, and finally upsampling is performed to obtain the segmentation result. Subsequently, for the problem of video semantic segmentation, the image segmentation model designed by the present invention is used as a segmentation module for video semantic segmentation, and the optical flow method is combined to improve the key frame selection strategy and perform feature transfer, reduce redundancy, and speed up video segmentation. The specific steps are as follows:

[0010] (1) UAV image data acquisition: The code can directly obtain the image stream of the camera mounted on the drone in real time by calling the VideoCapture method of the OpenCV library, and use the image stream as the input of the video semantic segmentation framework.

[0011] (2) Send the image data into the video semantic segmentation framework: Figure 4 The figure shows the overall framework of the video semantic segmentation model based on the optical flow method proposed in the present invention. Some common mathematical symbols used in this framework are shown in Table 1.

[0012] Table 1 Summary of mathematical symbols in this framework

[0013]

[0014] This framework has two main steps and consists of two branches: optical flow branch and segmentation branch. Let I i and I i+1 are two current frames to be processed, I k is the key frame closest to them (the first frame is taken as the key frame by default), assuming I i with I k The difference is small, I i+1 with I k There is a big difference.

[0015] 1) Step 1: Current frame I i or I i+1 With keyframe I kAfter preprocessing, the optical flow calculation network is input simultaneously to obtain the optical flow field between the two frames. The optical flow field is then input to the discriminant network (Decision Network, DN). The discriminant network begins to analyze the similarity between the two input video frames and calculates the prediction value confidence between the two frames. The discriminant network compares the prediction value confidence with the set confidence threshold t. If the prediction value confidence is greater than t, the current frame is sent to the optical flow branch for further processing. If the prediction value confidence is less than t, the current frame is sent to the segmentation branch for further processing. So in Figure 1 In the current frame I i Processed by the optical flow branch (red flowchart), the current frame I i+1 Processing is done through the segmentation branch (blue flowchart). The greater the confidence of the predicted value, the more similar the current frame is to the key frame, and the better the segmentation result obtained through the optical flow branch. At the same time, the t value determines the frequency of use of the two branches and also affects the final segmentation speed and accuracy.

[0016] 2) Step 2: The discriminant network sends each video frame to two subsequent different branches in turn according to the similarity between the current frame and the key frame to obtain the segmentation result of the current frame. The segmentation branch directly sends the current frame to the semantic segmentation network for processing, which is the same as the general image semantic segmentation processing flow. The optical flow branch uses the optical flow field of the current frame and the key frame in step 1 as input, and converts the previously processed key frame segmentation map to the segmentation result of the current frame through the propagation function W, and no longer needs to be processed through the segmentation network. It should be noted that the optical flow branch cannot obtain the segmentation result by relying solely on the optical flow calculation network, and must use the segmentation map and propagation function of the previous nearest neighbor key frame.

[0017] (3) Output results: Visualize the semantic segmentation results, display the current fps and whether the optical flow branch or the original semantic segmentation branch is executed in real time, and save the segmentation results in a folder.

[0018] In the method of the present invention, a series of data processing and network prediction are performed on the camera image carried by the unmanned aerial vehicle to finally obtain a visualization result after semantic segmentation.

[0019] Beneficial effects of the present invention:

[0020] The present invention has no restrictions on the operating platform, is flexible and convenient to use, has a small program size (only a few MB), and has strong portability. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a structural diagram of the video semantic segmentation framework based on the optical flow method;

[0022] Figure 2 This is the overall schematic diagram of the QNet model;

[0023] Figure 3 (a) Bottleneck structure designed for the present invention;

[0024] Figure 3 (b) is the initialization module;

[0025] Figure 4 The spatial pyramid structure designed for the present invention;

[0026] Figure 5 This is the structure diagram of the feature propagation strategy based on the optical flow method;

[0027] Figure 6 Schematic diagram of the discriminant network structure and its training strategy;

[0028] Figure 7 Comparison chart of the results of different video semantic segmentation frameworks on the Cityscapes dataset;

[0029] Figure 8 Comparison chart of image semantic segmentation and video semantic segmentation results on the Cityscapes dataset;

[0030] Fig. 9 This is the actual scene segmentation effect from the low-altitude perspective of a drone. DETAILED DESCRIPTION

[0031] The specific embodiments of the present invention are described in detail below in conjunction with the content of the invention:

[0032] The following describes in detail the process of using drones to perform real-time semantic segmentation. The present invention further illustrates the purpose and method of use of the present invention through the following cases, but the present invention is not limited thereto.

[0033] 1. Experimental equipment and environment configuration

[0034] Experimental equipment: DJI Matrice 210RTK V2 drone, high-performance airborne computer Manifold2, Zenmuse X7 gimbal camera

[0035] Software system: LinuxUbuntu 16.04LTS Server system

[0036] Programming language: Python 3.6

[0037] Deep learning framework: Tensorflow1.14

[0038] 2. Experimental Methods

[0039] (1) UAV image data acquisition: In the Ubuntu system, the code can directly obtain the image stream of the camera mounted on the drone in real time by calling the VideoCapture method of the OpenCV library, and use the image stream as the input of the video semantic segmentation framework.

[0040] (2) Video semantic segmentation network prediction: Execute the inference.py file in the main folder directory (that is, locate the terminal in the main folder directory, enter pythoninference.py and press Enter). The terminal will display whether the current image frame uses the optical flow branch or the original semantic segmentation branch, and display the real-time indicator fps and segmentation results.

[0041] Among them, the semantic segmentation network adopted by the original semantic segmentation branch is a lightweight network QNet designed and named by the present invention. In view of the accuracy and speed trade-off problem of image semantic segmentation, the semantic segmentation network designed by the present invention takes lightweight and high performance as the core goals, and adopts autoencoders as the basic network structure. Autoencoders can play the role of data denoising and nonlinear feature dimensionality reduction, but traditional autoencoders with symmetrical structures generally have poor generalization ability. Therefore, the present invention adopts an encoding-decoding asymmetric structure to design the entire image semantic segmentation network. Among them, the encoding end mainly simplifies the model structure as much as possible while performing feature extraction, reduces the number of parameters, and improves computational efficiency. In order to ensure the final segmentation accuracy, the decoding end fully integrates feature maps of different scales in the process of restoring feature information. The lightweight image semantic segmentation model designed by the present invention is named Quick Network (QNet). The network structure is shown in Table 2, and the schematic diagram of the model structure is shown in Figure 2 shown.

[0042] In terms of improving real-time performance, the present invention mainly designs a lightweight feature extraction module at the encoding end, and does not use a pre-trained model with high computational complexity. A new feature extraction unit is designed based on the basic unit of ResNet, which integrates strategies such as asymmetric convolution and dilated convolution, which can greatly reduce the number of parameters and calculations. In addition, the decoding end further improves the calculation efficiency by reducing the number of feature map channels.

[0043] In terms of accuracy, the present invention mainly uses an improved spatial pyramid structure at the decoding end, obtains a larger receptive field through 6 branches, and fuses feature maps of different scales, which greatly improves the feature restoration performance of the decoding end. In addition, the encoding end introduces channel slicing and channel reorganization strategies to communicate and fuse information between different channels, further improving the effectiveness of the extracted features.

[0044] Table 2 QNet network structure composition table (input size is set to 360×480, C represents the number of semantic categories)

[0045]

[0046] In the initialization stage, in order to speed up the model inference speed, the present invention draws on the ENet network and designs an initialization module to reduce the input image size and increase the number of channels. Figure 3 As shown in (b), the maximum pooling is performed under non-overlapping 2×2 windows, the convolution layer has 13 convolution kernels, the feature map channels after Concat are 16, and the size is 1 / 2 of the input image.

[0047] The encoding stage mainly uses the feature extraction unit based on channel split and channel reorganization designed by the present invention. Block 1 consists of five basic units, of which the first unit is downsampled, and Block 2 consists of nine basic units, of which the first unit is downsampled. The decoding stage mainly uses the spatial pyramid multi-feature fusion module designed by the present invention, which improves the ability to understand the encoded feature map, fully utilizes the spatial information through the multi-scale feature map, and takes into account the calculation cost and efficiency.

[0048] In order to further improve the network performance, the network of the present invention uses asymmetric convolution and dilated convolution in some modules. The specific settings and parameters are shown in Table 2.

[0049] 1) Asymmetric Convolution: Decompose the n×n convolution into a series of 1×n and n×1 convolutions, first perform a 1×n convolution, then perform an n×1 convolution, the result is the same as directly performing n×n convolution. However, the amount of multiplication calculations has changed from n×n to 2×n, so asymmetric convolution can greatly reduce the number of parameters, reduce the redundancy of convolution parameters, and improve the reasoning speed while keeping the receptive field unchanged. The larger n is, the more obvious the effect of asymmetric convolution in reducing the amount of calculation. For example, if n=3, 33% of the calculation can be saved after decomposition. At the same time, asymmetric convolution integrates multiple nonlinear activation layers instead of a single nonlinear activation layer, which increases the discrimination ability of the network. Through testing, it is also found that asymmetric convolution has better effects when used in the middle layer of the network.

[0050] 2) Dilated Convolution: In the traditional field of deep learning-based image processing, convolution is usually used first and then pooling is used for downsampling to reduce the image size. Pooling can increase the receptive field to obtain more feature information, and finally upsampling is performed to restore the size. However, in the process of reducing the image size first and then increasing the image size, some information is lost, which affects the final classification accuracy. In order to avoid excessive downsampling and reducing the resolution of the feature map and losing information, dilated convolution can expand the receptive field by adding holes without using pooling operations. With the same number of parameters and computational complexity, a larger receptive field is obtained, more global information is obtained, and multi-scale context information is captured, while retaining the relative spatial position information of the feature map. For example, the original 3×3 convolution kernel can have a receptive field of 5×5 (Dilated rate = 2) or larger. In order to improve the performance of dilated convolution, the Dilated rate of the stacked convolution cannot have a common divisor greater than 1, and should be designed as a zigzag structure, and it needs to satisfy formula (1).

[0051] M i =max[M i+1 -2r i ,M i+1 -2(M i+1 -r i ),r i ] (1)

[0052] where r i is the Dilated rate of layer i, and M i It refers to the maximum Dilated rate at layer i. Assuming there are n layers in total, the default is m n =r n The present invention designs the Dilated rate into an irregular non-cyclic structure according to these rules, namely [1, 2, 5, 1, 9, 17].

[0053] Among them, channel split can effectively improve the network reasoning speed, and channel reorganization (Channel shuffle) can help the flow of information in different feature channels. After adopting more groups, more channels can be used under the same computing constraints, or the number of feature maps increases, the network's feature extraction ability is enhanced, and the network performance is improved. Therefore, through Channel split and Channel shuffle, a high-performance lightweight network model can be built.

[0054] Based on the Bottleneck structure and Channel split and Channel shuffle operations, the present invention designs a new feature extraction unit using the basic structure of ResNet. Figure 3(a). The basic Bottleneck structure is a residual unit containing 3 layers, first a 1×1 convolution, then a 3×3 convolution to reduce the amount of calculation, followed by a 1×1 convolution, and finally a short-circuit connection to add the input directly to the output. The unit designed by the present invention is composed of two branches that perform pixel-level addition, wherein the main branch first performs slices of the input image channel. The present invention is divided into two groups of branches with equal number of channels, and then the step-by-step convolution in the Bottleneck is performed simultaneously. The first 1×1 convolution unit reduces the number of channels to 1 / 4 of the input. The convolution size of the main convolution unit (i.e., Conv in the figure) is 3×3, and the types include ordinary convolution, void convolution, and asymmetric convolution. The last 1×1 convolution unit expands the number of channels, and the channels are overlapped (Concat) after the convolution operation. Finally, the channels are reorganized after short-circuit pixel-by-pixel addition with the secondary branch to output the feature map. If the unit needs to be downsampled to reduce the size of the feature map, the secondary branch needs to add a Maxpooling layer and perform a Padding operation at the same time to match the output sizes of the two branches. The main branch replaces the first 1×1 convolution unit with a 2×2 convolution, and the convolution step size is also changed to 2, so that the output size of the two branches is 1 / 2 of the input size. Finally, the feature maps of the two branches are added and then the channels are shuffled to promote information fusion. In order to speed up the gradient descent speed during network training and reduce training time, the present invention adds Batch Normalization between all convolutions. At the same time, in order to reduce the risk of overfitting of the network and improve the generalization ability of the network, the present invention uses the PRelu activation function and Dropout operation.

[0055] Among them, the present invention draws on the spatial pyramid multi-feature fusion structure (SPP) to design an improved multi-feature fusion module, such as Figure 4As shown, the present invention adopts 1×1, 2×2, 4×4, 8×8, 16×16, 32×32 pooling kernels and step sizes, and obtains 6 feature maps of different sizes after the feature map obtained by the encoding end is subjected to average pooling operation, and obtains several different sub-regions. In order to maintain the weight of the global feature, the channel of each feature map is then reduced to 1 / 2 of the original using 1×1 convolution, and then these low-dimensional feature maps are upsampled by bilinear interpolation to make them the same scale as the original feature map. At the same time, the 6 feature maps are added in two groups, and finally concat together with the original input feature map, and the feature maps of different levels are superimposed as the final pyramid pooling global feature. This module aggregates the information of different regions of the input feature map using receptive fields of different sizes, reduces the information loss between different regions, and fully integrates the global information and local information of different scales, obtains information between different scales and different sub-regions, and improves the ability of the network to use global context information and the overall reasoning ability of the decoding end. At the same time, reducing the number of channels and the final direct concat operation also fully guarantees a low amount of calculation.

[0056] In order to further improve the computational efficiency and speed up the inference, the present invention will not perform feature propagation in the feature extraction stage, but directly propagate the segmentation map of the key frame to the current frame through the optical flow method to obtain the segmentation result of the current frame.

[0057] Current frame I i After input, the key frame closest to the previous one I k The optical flow field F between the two frames is calculated through the optical flow network k→i , current frame I i The position of the pixel point p in is projected back to the key frame I through the optical flow field k corresponds to p+δp, where δp=F k→i (p), since δp is generally a non-integer, feature transformation can be achieved through bilinear interpolation, as shown in equations (2) and (3):

[0058] S i (p) = ∑ q G(q,p+δp)S k (q) (2)

[0059] G(q,p+δp)=g(q x ,p x +δp x )·g(q y ,p y +δp y ) (3)

[0060] Where g(a,b) = max(0,1-|ab|). Finally, the propagation function between the current frame segmentation map and the key frame segmentation map can be simplified as formula (4):

[0061] S i =W(S k ,F k→i ) (4)

[0062] The present invention uses FlowNet2-s as the optical flow calculation network, which has significant improvements in data training and model structure compared to FlowNet, and the overall performance of the model is also the best at present. The semantic segmentation network uses the lightweight image semantic segmentation network QNet designed by the present invention, and the overall segmentation performance, especially the real-time performance, is significantly improved compared to other mainstream methods. The schematic diagram of the feature propagation strategy based on the optical flow method is shown in Figure 2. Figure 5 As shown, the optical flow field between the current frame and the previous key frame is first obtained through the optical flow calculation network FlowNet2-s, where the key frame has been previously segmented by the semantic segmentation network QNet. Then, the segmentation map of the key frame and the optical flow field between the two frames are fused and calculated to obtain the segmentation map of the current frame.

[0063] The feature propagation strategy based on the optical flow method proposed in this invention integrates the semantic segmentation network and the optical flow calculation network to build a new model. Both networks can directly use the previous pre-trained model, and the overall model does not need to be newly trained and tested, which greatly reduces the computational cost. The video semantic segmentation network constructed by this strategy can avoid semantic segmentation of each video frame, effectively reducing the amount of calculation, and the overall network will be fully accelerated. At the same time, the advanced optical flow calculation network ensures the accuracy of feature propagation and meets the real-time and accuracy requirements of video tasks.

[0064] The present invention adopts a quality-based key frame selection method and uses optical flow method to assist in improving the overall quality and efficiency. Figure 6 This is the overall structure diagram of the key frame scheduling strategy based on the discriminant network and the training strategy of the discriminant network. The discriminant network is a lightweight convolutional neural network consisting of only a single convolutional layer and three fully connected layers, and its input is the output of the optical flow network.

[0065] In the training phase of the discriminant network, the current frame I i and the nearest neighbor keyframe I k The training goal is to obtain the prediction confidence score to indicate the similarity of the two input images. After the current frame and the key frame are input together, the optical flow network FlowNet2-s calculates the optical flow field F of the two. k→i And Warp function W, then the semantic segmentation result of the key frame is calculated by the Wrap function to obtain the semantic segmentation output of the current frame Oi The other branch directly calculates the semantic segmentation output S through the image semantic segmentation network QNet proposed by the present invention. i , define the label value confidence score as O i and S i The expected similarity of is calculated as shown in formula (5):

[0066]

[0067] Where P is the total number of pixels in the current frame, p is the index of P, and O i (p),S i (p) represents the semantic category label of pixel p calculated by the two branches respectively. C(u,v) is an indicative function that outputs 1 only when u is equal to v, otherwise it is 0.

[0068] The discriminant network branch outputs the predicted value confidence score, and the segmentation branch outputs the label value confidence score. The two are calculated in different ways, which are equivalent to representing the model's predicted value and true value respectively. Based on the two, a regression model can be built for training, with mean square error as the loss function. Therefore, the label value confidence score is only used in the training process of the discriminant network. During the test process, the discriminant network outputs only the predicted value confidence score.

[0069] Set a confidence threshold t, compare the confidence score of the prediction value of the current frame with t. If it is greater than t, it indicates that the current frame has a high similarity with the previous key frame, and the video frame can be set as a non-key frame. Otherwise, if it is less than t, it indicates that the current frame has a low similarity with the previous key frame, and the current frame can be extracted as a new key frame. The confidence threshold t is set artificially. If it is too high, the extracted key frame accuracy will be higher, but the number of extracted key frames will be relatively large. The subsequent segmentation branch needs to process more video frames, and the overall segmentation accuracy will be improved, but at the same time it will bring a greater amount of calculation, affecting the real-time performance. On the contrary, if the t value is too small, the model runs faster, but it will reduce the subsequent segmentation accuracy. Therefore, it is necessary to make a good balance according to the specific data set or application scenario requirements and reasonably adjust the confidence threshold t. The key frame scheduling strategy based on the discriminant network proposed by the present invention can adaptively extract key frames, flexibly adjust according to the video content, effectively improve the key frame scheduling performance while limiting the amount of calculation, and help improve the efficiency and accuracy of subsequent segmentation tasks.

[0070] 3. Experimental results

[0071] (1) Comparative experiment:

[0072] In order to display the comparative experimental results more intuitively and comprehensively, in addition to comparing the results of each video segmentation framework, the segmentation effects of the basic image segmentation network and the corresponding video semantic segmentation framework are also compared.

[0073] Table 3 Comparison results of image and video segmentation accuracy and real-time performance of Cityscapes dataset (t=90)

[0074]

[0075] The individual image semantic segmentation networks QNet, PSPNet, ICNet and the corresponding video semantic segmentation frameworks QNet+FlowNet2-s, PSPNet+FlowNet2-s, and ICNet+FlowNet2-s are comprehensively compared in terms of accuracy and speed, using indicators such as Class IoU, Category IoU, and Frame commonly used in the Cityscapes dataset. The experimental results are shown in Table 3. The QNet+FlowNet2-s video semantic segmentation framework proposed in the present invention has no advantage over other frameworks in terms of accuracy indicators such as IoU, but has more advanced performance in terms of indicators such as FLOPs, number of parameters, and model size. The number of video frames processed per second reaches 23.4FPS, which is significantly higher than other frameworks, indicating that the framework proposed in the present invention has more advantages under conditions of limited computing power and storage conditions, and is more suitable for mobile devices in actual scenarios. (In the data table, italics indicate the optimal values ​​of the same group for the image semantic segmentation model, and bold indicates the optimal values ​​of the same group for the video semantic segmentation framework.)

[0076] Figure 7 By comparing the segmentation result graphs of various video semantic segmentation frameworks, it can be found that the video semantic segmentation framework based on the optical flow method is effective, each object is clearly segmented, the edges are clear, and it can accurately reflect the semantic information of the scene. Among them, ICNet+FlowNet2-s has a better overall effect, fewer error points, and clear object contours. The QNet+FlowNet2-s proposed in the present invention has a certain guarantee of accuracy, and can also segment small objects such as inconspicuous telephone poles, and the segmented objects can correspond to the actual picture.

[0077] Figure 8 By comparing the segmentation results of the image semantic segmentation network QNet proposed in this paper and the video semantic segmentation framework QNet+FlowNet2-s, it can be found that the pure image semantic segmentation is better than the video semantic segmentation after adding the optical flow method. The edges of the segmented objects are neater and there are fewer noise points, which also verifies the theoretical analysis of this chapter. In order to speed up the segmentation speed, using the same segmentation method under the same conditions, the video semantic segmentation will lose a certain degree of accuracy.

[0078] (2) Actual scenario test results:

[0079] In order to further verify the effectiveness of the semantic segmentation framework proposed in this chapter, experiments were also conducted in actual scenarios. This paper uses a DJI M210 RTK V2 drone equipped with a ZENMUSE X5 camera to fly low over Beichen Road, the main campus of Dalian University of Technology, to obtain video data similar to the Cityscapes street view dataset, and then uses QNet+FlowNet2-s to directly perform semantic segmentation. The overall effect is as follows: Fig. 9 As shown in the figure, it can be found that the overall segmentation effect is good, and the main objects such as roads, pedestrians, cars, and trees are clearly segmented, but some images have more noise and some mis-segmentation, especially when the UAV's flight speed is unstable or the scene is too complex.

Claims

1. A method for semantic segmentation of drone-view video based on deep learning, characterized in that: Here are the steps: The first step is to obtain drone image data: the code can directly obtain the image stream of the camera mounted on the drone in real time by calling the VideoCapture method of the OpenCV library, and use the image stream as the input of the video semantic segmentation framework; The second step is to feed the image data into the video semantic segmentation framework; The video semantic segmentation framework consists of two branches: optical flow branch and segmentation branch. The data processing process includes: 2.1 Current Frame I i or I i+1 With keyframe I k After preprocessing, the optical flow calculation network is input simultaneously to obtain the optical flow field between the two frames, and then the optical flow field is input to the discriminant network. The discriminant network begins to analyze the similarity between the two input video frames and calculates the prediction value confidence between the two frames. The discriminant network compares the prediction value confidence with the set confidence threshold t. If the prediction value confidence is greater than t, the current frame is sent to the optical flow branch for further processing. If the prediction value confidence is less than t, the current frame is sent to the segmentation branch for further processing. The larger the prediction value confidence, the more similar the current frame is to the key frame, and the better the segmentation result obtained by converting the optical flow branch. At the same time, the t value determines the usage frequency of the two branches and also affects the final segmentation speed and accuracy. 2.2 The discriminant network sends each video frame to two subsequent different branches in turn according to the similarity between the current frame and the key frame to obtain the segmentation result of the current frame; the segmentation branch directly sends the current frame to the semantic segmentation network for processing; the semantic segmentation network adopts an "encoder-decoder" asymmetric network structure, in which the encoder stage integrates channel slicing and channel reorganization to improve the Bottleneck structure for downsampling and feature extraction, and the decoder stage is based on the spatial pyramid multi-feature fusion module to extract rich features and fuse them, and finally upsample to obtain the segmentation result; the optical flow branch takes the optical flow field of the current frame and the key frame in step 1 as input, and converts the previously processed key frame segmentation map to the segmentation result of the current frame through the propagation function W, and no longer needs to be processed through the segmentation network; 2.3 Output results: Visualize the semantic segmentation results, display the current fps and whether the optical flow branch or the original semantic segmentation branch is executed in real time, and save the segmentation results in a folder.

Images