A UAV monitoring target classification method based on 3D feature fusion and semantic segmentation
Through the semantic segmentation method of three-dimensional feature fusion, combined with the three-dimensional attention module and the overlapping fusion module, the problems of slow detection speed and inaccurate classification in UAV monitoring are solved, and more efficient target classification and real-time performance are achieved.
Patent Information
- Application Number
- CN202210747559.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-28
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2042-06-28
AI Technical Summary
The existing drone monitoring target classification methods in high-resolution aerial images detect inaccurate classification problems caused by slow speed and lightweight networks, and lack effective feature extraction and fusion methods.
Using the semantic segmentation method of three-dimensional feature fusion, the body attention module and the overlapping fusion module are constructed, combined with the MobilenetV2 network, feature extraction and context information fusion are performed, and vertical and horizontal attention weight weighting and adaptive channel weight pooling are used to improve the model running rate and classification accuracy.
It improves the real-time and accuracy of drone monitoring target classification, solves the problems of too slow high-resolution image segmentation rate and low identification classification accuracy, and enhances the detection accuracy of small targets.
Smart Images
Figure CN115063704B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of artificial intelligence semantic segmentation, and more particularly to the classification of target objects in drone aerial photography scenarios, and more particularly to, but not limited to, a drone monitoring target classification method that combines three-dimensional feature fusion with semantic segmentation. Background Art
[0002] With the development of the times, drones are increasingly used in production and life. Therefore, exploring drone-based monitoring target classification methods with the help of machine learning and deep learning technologies has become a hot topic in industry research.
[0003] Most of the existing methods for target classification are through traditional machine learning and adding manual feature variables. Different feature variables will lead to different classification results. Manually adding feature variables often relies on experience and has a certain degree of blindness. Deep learning is one of the important advances in artificial intelligence. It subverts the design ideas of traditional algorithms and can automatically learn features from data, avoiding the complexity and limitations of manual design and feature extraction in traditional algorithms. In the absence of human intervention, it has great feature learning potential. In the field of remote sensing, deep learning, especially convolutional neural networks (CNN), has been widely used in the classification, extraction, recognition and segmentation of remote sensing images, surpassing traditional methods in all aspects.
[0004] At present, the deep learning algorithms used for target classification are mainly semantic segmentation algorithms with codec structures. The main idea is to complete feature extraction through standard deep networks such as Alexnet, VGG-16, Resnet, etc. in the encoder part, and then the decoder part projects the recognition features (low resolution) learned by the encoder semantically to the pixel space (high resolution) to obtain dense classification. There are mainly three methods: region-based semantic segmentation, full convolutional network semantic segmentation, and weakly supervised semantic segmentation. Region-based semantic segmentation will perform semantic segmentation based on the target detection results. First, free-form regions are extracted from the image and described, and then region-based classification is performed; full convolutional network semantic segmentation is a pixel-to-pixel mapping, and no regional target detection is performed. The advantage is that it can accept inputs of any size for prediction; weakly supervised semantic segmentation mainly uses annotated bounding boxes to achieve semantic segmentation.
[0005] In view of this, it is necessary to provide a new method to solve at least part of the above problems. Summary of the invention
[0006] In response to one or more problems in the prior art, the present invention proposes a UAV monitoring target classification method based on stereo feature fusion semantic segmentation. By training on a UAV dataset and extracting feature representations of generalized target objects, the distinguishability between features is highlighted, and the problem of slower and less detection due to high resolution of aerial images and inaccurate target classification due to the use of lightweight networks is solved, thereby improving the model operation rate and classification accuracy.
[0007] The technical solution to achieve the purpose of the present invention is:
[0008] A method for classifying drone monitoring targets by fusion of stereo features and semantic segmentation, comprising:
[0009] Step 1: Construct a stereo attention module to magnify the importance of feature information according to the horizontal position and vertical position of the pixel, respectively. The stereo attention module includes a horizontal channel branch and a vertical channel branch; sequentially perform pooling, downsampling, depth compression, depth enhancement, depth adjustment and upsampling on the input feature map obtained by the drone to obtain a vertical attention weight map and a horizontal attention weight map; use the vertical attention weight map and the horizontal attention weight map to weight the input feature map;
[0010] Step 2: Construct an iterative fusion module to obtain the context information of the features proposed by the backbone network; the iterative fusion module includes three branches, and the adaptive channel weight pooling module in the iterative fusion module is used to obtain the global feature information of the feature maps input by the two branches of the current step, and the outputs of the two branches are fused with the feature map input by a branch of the previous step for output;
[0011] Step 3: Construct a convolutional neural network with a codec structure, in which the encoder uses the MobilenetV2 network as the backbone network for feature extraction, and inserts a stereo attention module into the MobilenetV2 backbone network, then obtains context information through the iterative fusion module, and fuses context information of different levels and steps through a three-step iterative fusion network; the decoder adjusts the channel and size of the feature map through channel splicing and convolution layers, restores the feature map to the original size through upsampling, and obtains the classification result through the Softmax layer; the convolutional neural network finally constructed includes convolution layers, stereo attention modules, activation function layers, pooling layers, BN layers, iterative fusion modules, Softmax layers and iterative fusion networks;
[0012] Step 4: Use the trained convolutional neural network to detect and classify objects in drone aerial images.
[0013] Furthermore, in the drone monitoring target classification method of the present invention using 3D feature fusion semantic segmentation, the step 1 of constructing a 3D attention module to obtain a longitudinal attention weight map specifically includes:
[0014] 1) Input feature map Perform feature extraction through a convolution layer to obtain the input feature map after feature extraction in Represents the three-dimensional features of the feature map, h represents the feature map height, w represents the feature map width, and c represents the feature map depth;
[0015] 2) Perform average pooling on the input feature map F horizontally to obtain the feature map Z after horizontal pooling:
[0016] Z=CrossPool(F)
[0017] Among them, CrossPool represents horizontal average pooling, and the feature map after horizontal pooling
[0018] 3) Downsample the feature map Z after horizontal pooling vertically to obtain the feature map Z' after vertical downsampling:
[0019] Z'=dw(Z)
[0020] Among them, dw represents downsampling, the feature map after vertical downsampling
[0021] 4) Use three 1×1 convolutional layers to extract the features of the feature map Z' after vertical downsampling:
[0022] 4-1) Compress the depth of the feature map Z' after vertical downsampling to obtain the feature map Z1 after vertical compression:
[0023] Z1=kenerl 1×1 ·Z'
[0024] Among them, the scaling ratio is r, kenerl 1×1 is a 1×1 convolution kernel, the compressed feature map
[0025] 4-2) The feature map Z1 after longitudinal compression is depth-enhanced and activated using the Relu activation function to obtain the feature map Z2 after longitudinal depth enhancement:
[0026] Z2=δ(kenerl 1×1 ·Z1)
[0027] Among them, δ represents the Relu activation function, and the feature map after the vertical depth is improved
[0028] 4-3) Adjust the depth of the feature map Z2 after the longitudinal depth enhancement to make it consistent with the depth of the input feature map F' after feature extraction, and use the Sigmoid function to map the result to (0,1) to obtain the longitudinal feature weight vector Z3:
[0029] Z3=σ(kenerl 1×1 ·Z2)
[0030] Among them, σ represents the Sigmoid activation function, and the vertical feature weight vector
[0031] 5) Upsample the vertical feature weight vector Z3 vertically to restore it to the size of the input feature map vertically, and obtain the vertical attention weight map A:
[0032] A=up(Z3)
[0033] Among them, the vertical attention weight map
[0034] Furthermore, in the drone monitoring target classification method of the present invention using stereo feature fusion semantic segmentation, the step 1 of constructing a stereo attention module to obtain a lateral attention weight map specifically includes:
[0035] 1) Input feature map Perform average pooling in the vertical direction to obtain the feature map U after vertical pooling:
[0036] U=VerticalPool(F)
[0037] Among them, VerticalPool represents the vertical average pooling, the feature map after vertical pooling;
[0038] 2) The feature map U after vertical pooling is horizontally downsampled to obtain the feature map U' after horizontal downsampling:
[0039] U'=dw(U)
[0040] Among them, dw represents downsampling, the feature map after horizontal downsampling
[0041] 3) Use three 1×1 convolutional layers to extract the features of the feature map U' after horizontal downsampling:
[0042] 3-1) Compress the depth of the feature map U' after horizontal downsampling to obtain the horizontally compressed feature map U1:
[0043] U1=kenerl 1×1 ·U'
[0044] Among them, the scaling ratio is r, kenerl 1×1 is a 1×1 convolution kernel, the compressed feature map
[0045] 3-2) The feature map U1 after horizontal compression is depth-enhanced and activated using the Relu activation function to obtain the feature map U2 after horizontal depth enhancement:
[0046] U2=δ(kenerl 1×1 U1)
[0047] Among them, δ represents the Relu activation function, and the feature map after horizontal depth enhancement
[0048] 3-3) Adjust the depth of the feature map U2 after horizontal depth enhancement to make it consistent with the depth of the input feature map F' after feature extraction, and use the Sigmoid function to map the result to (0,1) to obtain the horizontal feature weight vector U3:
[0049] U3=σ(kenerl 1×1 U2)
[0050] Among them, σ represents the Sigmoid activation function, and the horizontal feature weight vector
[0051] 4) Upsample the horizontal feature weight vector U3 horizontally to restore it to the size of the input feature map horizontally, and obtain the horizontal attention weight map B:
[0052] B=up(U3)
[0053] Among them, the horizontal attention weight map
[0054] Furthermore, in the drone monitoring target classification method of the present invention using stereo feature fusion semantic segmentation, the step 1 of weighting the input feature map using the longitudinal attention weight map and the lateral attention weight map specifically includes:
[0055] 1) The number of vertically overlapping pixels between the two adjacent frames of images on the UAV’s motion trajectory is recorded as λ A , the number of horizontally overlapping pixels is denoted as λ B , the direction adaptive weight W is obtained according to the number of vertical and horizontal overlapping pixels d :
[0056]
[0057] Among them, h represents the height of the feature map, and w represents the width of the feature map;
[0058] 2) Perform weighted operations based on the longitudinal attention weight map, the lateral attention weight map, and the direction adaptive weight to obtain the weighted feature map
[0059]
[0060] Among them, A represents the vertical attention weight map, B represents the horizontal attention weight map, F' represents the input feature map after feature extraction, and × represents the multiplication operation.
[0061] Furthermore, in the drone monitoring target classification method of the present invention using stereo feature fusion semantic segmentation, the process of obtaining the attention weight of the adaptive channel weight pooling module in step 2 specifically includes:
[0062] 1) Input feature map Perform global maximum pooling and global average pooling respectively, merge the pooled feature maps in series and input them into the multi-layer perceptron;
[0063] 2) Use 1×1 convolution to adjust the feature size of the pooled feature map to 1×1×c, and use the Sigmoid function to map the result to (0,1) to obtain the attention weight M of the adaptive channel weight pooling module:
[0064] M=σ(Kernel 1×1 ·FC(contact(avgPool(F),maxPool(F))))
[0065] Among them, σ represents the Sigmoid activation function, Kernel 1×1 represents a 1×1 convolution kernel, represents a convolution operation, FC represents a multi-layer perceptron, contact represents a cascade merge operation, avgPool represents global average pooling, maxPool represents global maximum pooling, h represents the height of the feature map, w represents the width of the feature map, c represents the depth of the feature map, and attention weights
[0066] Then the feature map F after the attention weight weighting of the adaptive channel weight pooling module is output for:
[0067] F output =M×F+F=(M+1)×F
[0068] Among them, × represents the multiplication operation, the weighted feature map
[0069] Furthermore, in the drone monitoring target classification method of the present invention using stereo feature fusion semantic segmentation, the processing process of the input feature map by the adaptive channel weight pooling module in step 2 specifically includes:
[0070] 1) Input feature map Divide into g×g (g∈1,3,5,7) grids, perform global average pooling on each grid, and get the output feature map P:
[0071] P = avgPool(F)
[0072] Among them, h represents the height of the feature map, w represents the width of the feature map, c represents the depth of the feature map, and avgPool represents the global average pooling.
[0073] 2) Adjust the number of channels of the output feature map P to 1 / 4 of the original one, and obtain the adjusted feature map P':
[0074] P'=Kernel 1×1 ·P
[0075] Among them, Kernel 1×1 represents a 1×1 convolution kernel, represents a convolution operation, and the adjusted feature map
[0076] 3) The adjusted feature maps P' are upsampled to the original image size, and are combined in series with the original input feature map F, and weighted by the weight of the adaptive channel to obtain the adaptive weighted feature map Y:
[0077] Y=(M+1)×(contact(up(P'),F))
[0078] Among them, up represents upsampling, contact represents the series merging operation, × represents the multiplication operation, M represents the attention weight of the adaptive channel, and the adaptive weighted feature map
[0079] 4) Adjust the number of channels to match the original input features Figure 1 The adjusted feature map is obtained:
[0080] F”=Kernel 1×1 ·Y
[0081] in,
[0082] Furthermore, in the drone monitoring target classification method of the present invention using stereo feature fusion semantic segmentation, the processing process of the encoder in the convolutional neural network in step 3 specifically includes:
[0083] The first 8 layers of the MobilenetV2 network convolutional layer are selected as the feature extraction backbone network, denoted as Conv1~Conv8 respectively. The stereo attention module is inserted between the convolutional layers, and the output feature maps of the Conv3, Conv4, and Conv8 layers are selected as the input of the iterative fusion network, denoted as t∈0,1,2, the final feature map is obtained through a three-step fusion network;
[0084] Among them, the three-step fusion network obtains the context information of the backbone network extraction features as follows:
[0085] When the step is greater than 2, the feature map depths of the three branches are adjusted to be consistent through 1×1 convolution, and the input feature map of step s and level t is Input feature map of s-1 steps and t levels operated by adaptive channel weight pooling module Input feature map of s-1 steps and t-1 layers operated by the adaptive channel weight pooling module Fusion to obtain fusion results
[0086]
[0087] Among them, Kernel 1×1 represents a 1×1 convolution kernel, · represents a convolution operation, and ⊙ represents the corresponding element-wise multiplication operation;
[0088] When the step is 1, the feature map depths on the two branches are adjusted to be consistent through 1×1 convolution, and the input feature map of s=0 step and t level is Input feature map of s=0 steps and t-1 layers operated by adaptive channel weight pooling module Fusion, get the fusion result F1 t :
[0089]
[0090] Among them, up represents upsampling.
[0091] Furthermore, in the drone monitoring target classification method of the present invention using 3D feature fusion semantic segmentation, the processing process of the decoder in the convolutional neural network in step 3 specifically includes:
[0092] The feature map output by the 3-step iterative fusion network is upsampled and concatenated with the Conv2 layer feature map to obtain the concatenated feature map E:
[0093] E=contact(up(F3 3 ),F Conv2 )
[0094] Among them, F Conv2 represents the feature map of the Conv2 layer, F3 represents the feature map output by the 3-step iterative fusion network, contact represents the serial merging operation, and up represents upsampling;
[0095] The concatenated feature map is upsampled through a 3×3 convolution layer, the width and height of the feature map are restored to the input image size, and then the depth of the feature map is adjusted to the number of classifications through a 1×1 convolution layer, and the pixel-level classification operation is completed through the Softmax layer to obtain the pixel classification result R:
[0096] R = ζ (kernel 1×1 ·up(kernel 3×3 ·E))
[0097] Among them, ζ represents the Softmax activation function, Kernel 1×1 Represents 1×1 convolution kernel, Kernel 3×3 represents a 3×3 convolution kernel, and up represents upsampling.
[0098] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:
[0099] 1. The drone monitoring target classification method of the present invention with stereo feature fusion semantic segmentation adopts MobilenetV2 network as the backbone network for feature extraction in the encoder part, which solves the problem of slow segmentation rate of large-resolution images and meets the real-time requirements in drone aerial photography scenarios.
[0100] 2. The drone monitoring target classification method of the present invention based on stereo feature fusion semantic segmentation introduces a stereo attention module into the backbone network, which solves the problem of low recognition and classification accuracy caused by the lightweight backbone network.
[0101] 3. The drone monitoring target classification method of the present invention based on stereo feature fusion semantic segmentation obtains context information through an iterative fusion module, improves recognition accuracy through an adaptive channel weight pooling module, and fuses context information of different levels and steps through a three-step iterative fusion network, thereby effectively improving the detection and classification accuracy of small targets. BRIEF DESCRIPTION OF THE DRAWINGS
[0102] The accompanying drawings are used to provide a further understanding of the present invention and are used together with the description to explain the embodiments of the present invention, but do not constitute a limitation of the present invention. In the accompanying drawings:
[0103] Figure 1 A schematic diagram of a stereo attention module of the drone monitoring target classification method of the present invention with stereo feature fusion semantic segmentation is shown.
[0104] Figure 2 A schematic diagram of the adaptive channel weight calculation process of the drone monitoring target classification method of the present invention based on stereo feature fusion semantic segmentation is shown.
[0105] Figure 3 A schematic diagram of an adaptive channel weight pooling module of the drone monitoring target classification method of the present invention based on stereo feature fusion semantic segmentation is shown.
[0106] Figure 4 A schematic diagram of an iterative fusion module of the drone monitoring target classification method of the present invention using stereo feature fusion and semantic segmentation is shown.
[0107] Figure 5 A schematic diagram of a convolutional neural network with a codec structure for the drone monitoring target classification method of the present invention using stereo feature fusion semantic segmentation is shown. DETAILED DESCRIPTION
[0108] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the present invention.
[0109] The description in this section is only for typical embodiments, and the present invention is not limited to the scope of the embodiments. The combination of different embodiments, the mutual replacement of some technical features in different embodiments, and the mutual replacement of the same or similar prior art means with some technical features in the embodiments are also within the scope of the description and protection of the present invention.
[0110] The present invention proposes a UAV monitoring target classification method based on stereo feature fusion semantic segmentation, which comprises the following steps:
[0111] Step 1: Construct a three-dimensional attention module, which can amplify the importance of feature information according to the horizontal and vertical positions of pixels. The module includes two branches: horizontal channel and vertical channel. In the vertical branch, the input feature map is firstly horizontally pooled, then downsampled, and then the number of channels is adjusted. Furthermore, the vertical attention weight map is obtained by upsampling. Similarly, the horizontal attention weight map can be obtained. Finally, the input feature map is weighted using the vertical attention weight map and the horizontal attention weight map.
[0112] First, the input feature map Feature extraction is performed through a convolutional layer, and the result is recorded as
[0113] The vertical branch operation process is as follows: First, the input feature map Average pooling is performed in the w direction, and the result is recorded as The formula is as follows:
[0114] Z=CrossPool(F)
[0115] Among them, CrossPool represents average pooling in the w direction;
[0116] Then, the pooled feature map is downsampled and the result is recorded as The formula is as follows:
[0117] Z'=dw(Z)
[0118] Among them, dw represents downsampling;
[0119] Next, three 1×1 convolutional layers are used to extract features. First, the feature map depth is compressed with a scaling ratio of r to reduce the number of parameters and complete regularization. The result is recorded as The formula is as follows:
[0120] Z1=kenerl 1×1 ·Z'
[0121] The feature map is then deeply enhanced and activated using the Relu activation function. The result is recorded as The formula is as follows:
[0122] Z2=δ(kenerl 1×1 ·Z1)
[0123] Among them, δ represents the Relu activation function;
[0124] Furthermore, the feature map depth is adjusted to be consistent with F', and the result is mapped to (0,1) using the Sigmoid function. The result is recorded as The formula is as follows:
[0125] Z3=σ(kenerl 1×1 ·Z2)
[0126] Among them, σ represents the Sigmoid activation function;
[0127] Next, Z3 is vertically upsampled and restored to the input feature map size to obtain the vertical attention weight map. The result is recorded as The formula is as follows:
[0128] A=up(Z3)
[0129] The horizontal branch operation process is as follows: First, the input feature map Average pooling is performed in the h direction, and the result is recorded as The formula is as follows:
[0130] U=VerticalPool(F)
[0131] Among them, VerticalPool represents the average pooling in the h direction;
[0132] Then, the pooled feature map is downsampled and the result is recorded as The formula is as follows:
[0133] U'=dw(U)
[0134] Among them, dw represents downsampling;
[0135] Next, three 1×1 convolutional layers are used to extract features. First, the feature map depth is compressed with a scaling ratio of r to reduce the number of parameters and complete regularization. The result is recorded as The formula is as follows:
[0136] U1=kenerl 1×1 ·U'
[0137] The feature map is then deeply enhanced and activated using the Relu activation function. The result is recorded as The formula is as follows:
[0138] U2=δ(kenerl 1×1 U1)
[0139] Among them, δ represents the Relu activation function;
[0140] Furthermore, the feature map depth is adjusted to be consistent with F', and the result is mapped to (0,1) using the Sigmoid function. The result is recorded as The formula is as follows:
[0141] U3=σ(kenerl 1×1 U2)
[0142] Among them, σ represents the Sigmoid activation function;
[0143] Next, U3 is horizontally upsampled and horizontally restored to the size of the input feature map to obtain the horizontal attention weight map. The result is recorded as The formula is as follows:
[0144] B=up(U3)
[0145] According to the movement direction of the drone, there are overlapping areas in the vertical and horizontal directions of the continuously shot images. The number of vertically overlapping pixels between the two adjacent frames on the historical motion trajectory is recorded as λ A , the number of horizontally overlapping pixels is denoted as λ B , the direction adaptive weight W is obtained according to the number of vertical and horizontal overlapping pixels d , the formula is as follows:
[0146]
[0147] Finally, weighted operations are performed based on the longitudinal and lateral attention weight maps and the direction adaptive weights. The formula is as follows:
[0148]
[0149] Here, × represents a multiplication operation.
[0150] Step 2: Construct an iterative fusion module to obtain the contextual information of the features extracted by the backbone network. This module consists of three branches. First, the feature maps of the two input branches of the current step are used to obtain global feature information using the adaptive channel weight pooling module. Finally, the outputs of the two branches are fused with the input branch of the previous step.
[0151] First, the adaptive channel weight calculation process is as follows:
[0152] For the input feature map Perform global maximum pooling and global average pooling respectively, then combine the results in series and input them into the multi-layer perceptron (MLP), then use 1×1 convolution to adjust the feature size to 1×1×c, and finally use the Sigmoid function to map the result to (0,1) to obtain the feature channel attention weight The formula is as follows:
[0153] M=σ(Kernel 1×1 ·FC(contact(avgPool(F),maxPool(F))))
[0154] Among them, h represents the height of the feature map, w represents the width of the feature map, c represents the depth of the feature map, avgPool represents the global average pooling, maxPool represents the global maximum pooling, contact represents the serial merging operation, FC represents the multi-layer perceptron (MLP), Kernel 1×1 represents a 1×1 convolution kernel, represents a convolution operation, and σ represents a Sigmoid activation function.
[0155] Furthermore, the weighted feature map The calculation formula is as follows:
[0156] F output =M×F+F=(M+1)×F
[0157] Here, × represents a multiplication operation.
[0158] Furthermore, the process of constructing the adaptive channel weight pooling module in the iterative fusion module is as follows:
[0159] The input feature map Divide into g×g (g∈1,3,5,7) grids, perform global average pooling on each grid, and output feature maps as The formula is as follows:
[0160] P = avgPool(F)
[0161] Among them, h represents the height of the feature map, w represents the width of the feature map, c represents the depth of the feature map, and avgPool represents the global average pooling;
[0162] Then, the number of channels of the output feature map P is adjusted to 1 / 4 of the original one, and the adjusted feature map is recorded as The formula is as follows:
[0163] P'=Kernel 1×1 ·P
[0164] Among them, Kernel 1×1 represents a 1×1 convolution kernel, represents a convolution operation;
[0165] Next, the adjusted feature map P' is upsampled to the original image size, and then concatenated and merged with the original input feature map F, and then weighted by the adaptive channel weight. The result is recorded as The formula is as follows:
[0166] Y=(M+1)×(contact(up(P'),F))
[0167] Among them, up represents upsampling, contact represents the serial merging operation, and × represents the multiplication operation;
[0168] Furthermore, the number of channels is adjusted to match the original input features. Figure 1 The adjusted feature map is denoted as The formula is as follows:
[0169] F”=Kernel 1×1 ·Y
[0170] Step 3: Construct a convolutional neural network with a codec structure based on the MobilenetV2 network. The encoder part uses the MobilenetV2 network as the backbone network for feature extraction to solve the problem of slow segmentation rate of large-resolution images. At the same time, a stereo attention module is introduced into the MobilenetV2 backbone network to take into account recognition accuracy. The context information is obtained through the adaptive channel weight pooling module, and the context information obtained at different levels and steps is fused through a three-step iterative fusion network. The decoder part adjusts the channel and size of the feature map through channel splicing and convolutional layers, and restores the feature map to the original image size through upsampling. Finally, the classification result is obtained through the Softmax layer; the convolutional neural network finally constructed includes convolutional layers, stereo attention modules, activation function layers, pooling layers, BN layers, iterative fusion modules, Softmax layers and three-step iterative fusion networks.
[0171] The convolutional neural network with an encoder-decoder structure consists of two parts, the encoder and the decoder.
[0172] In the encoder, the first 8 layers of the MobilenetV2 network convolutional layer are selected as the feature extraction backbone network, respectively denoted as Conv1~Conv8, and the stereo attention module is inserted between the convolutional layers. The output feature maps of Conv3, Conv4, and Conv8 layers are selected as the fusion network input, respectively denoted as The final feature map is obtained by three-step iterative fusion network. The fusion network obtains the context information of the features extracted by the backbone network. The operation is as follows: When the step is greater than 2, the feature map depths on the three branches are adjusted to be consistent through 1×1 convolution. For s (s≥2) steps and t-level input feature maps It is combined with the input feature map of s-1 steps and t levels operated by the adaptive channel weight pooling module And the input feature map of s-1 steps and t-1 levels operated by the adaptive channel weight pooling module The fusion result is recorded as The formula is as follows:
[0173]
[0174] When the step is 1, the feature map depths on the two branches are adjusted to be consistent through 1×1 convolution. For s=0 step and t level input feature map It is combined with the input feature map of s=0 steps and t-1 layers operated by the adaptive channel weight pooling module The formula is as follows:
[0175]
[0176] In the decoder, the output feature map of the three-step iterative fusion network is first upsampled and concatenated with the feature map of the Conv2 layer. The result is recorded as E, and the formula is as follows:
[0177] E=contact(up(F3 3 ),F Conv2 )
[0178] Among them, F Conv2 Represents the Conv2 layer feature map;
[0179] Then pass through a 3×3 convolution layer, then upsample to restore the width and height of the feature map to the input image size. Further, through a 1×1 convolution layer, the depth of the feature map is adjusted to the number of categories. Finally, a Softmax layer is used to complete the pixel-level classification operation. The result is recorded as R, and the formula is as follows:
[0180] R = ζ (kernel 1×1 ·up(kernel 3×3 ·M))
[0181] Among them, ζ represents the Softmax activation function.
[0182] Step 4: Use the trained convolutional neural network to detect and classify objects in drone aerial images.
[0183] Compared with traditional semantic algorithms, the proposed method for drone monitoring target classification based on stereo feature fusion semantic segmentation adopts MobilenetV2 network as the backbone network for feature extraction in the encoder part, which solves the problem of too slow segmentation rate of large resolution images and meets the real-time requirements in drone aerial photography scenarios. At the same time, a stereo attention module is introduced into the backbone network to take into account recognition accuracy. At the same time, context information is obtained through the iterative fusion module, and the recognition accuracy is improved through the adaptive channel weight pooling module therein, and the context information of different levels and steps is fused through the three-step iterative fusion network to improve the detection and classification accuracy of small targets.
[0184] The description and application of the present invention here are illustrative, and it is not intended to limit the scope of the present invention to the above-mentioned embodiments. The effects or advantages involved in the specification may not be reflected in the actual experimental examples due to the uncertainty of specific condition parameters or other factors, and the effects or advantages are not used to limit the scope of the invention. The deformation and change of the embodiments disclosed here are possible, and the replacement of the embodiments and the various equivalent parts are well known to those of ordinary skill in the art. It should be clear to those skilled in the art that the present invention can be implemented in other forms, structures, arrangements, proportions, and with other components, materials and parts without departing from the spirit or essential features of the present invention. Other deformations and changes can be made to the embodiments disclosed here without departing from the scope and spirit of the present invention.
Claims
1. A method for classifying drone monitoring targets by fusion of stereo features and semantic segmentation, characterized in that: include: Step 1: Construct a stereo attention module to magnify the importance of feature information according to the horizontal position and vertical position of the pixel, respectively. The stereo attention module includes a horizontal channel branch and a vertical channel branch; sequentially perform pooling, downsampling, depth compression, depth enhancement, depth adjustment and upsampling on the input feature map obtained by the drone to obtain a vertical attention weight map and a horizontal attention weight map; use the vertical attention weight map and the horizontal attention weight map to weight the input feature map; Step 2: Construct an iterative fusion module to obtain the context information of the features proposed by the backbone network; the iterative fusion module includes three branches, and the adaptive channel weight pooling module in the iterative fusion module is used to obtain the global feature information of the feature maps input by the two branches of the current step, and the outputs of the two branches are fused with the feature map input by a branch of the previous step for output; Step 3: Construct a convolutional neural network with a codec structure, in which the encoder uses the MobilenetV2 network as the backbone network for feature extraction, and inserts a stereo attention module into the MobilenetV2 backbone network, then obtains context information through the iterative fusion module, and fuses context information of different levels and steps through a three-step iterative fusion network; the decoder adjusts the channel and size of the feature map through channel splicing and convolution layers, restores the feature map to the original size through upsampling, and obtains the classification result through the Softmax layer; the convolutional neural network finally constructed includes convolution layers, stereo attention modules, activation function layers, pooling layers, BN layers, iterative fusion modules, Softmax layers and iterative fusion networks; Step 4: Use the trained convolutional neural network to detect and classify objects in drone aerial images.
2. The method for classifying drone monitoring targets by fusion of stereo features and semantic segmentation according to claim 1 is characterized in that: The vertical attention weight map obtained in the stereo attention module constructed in step 1 specifically includes: 1) Input feature map Perform feature extraction through a convolution layer to obtain the input feature map after feature extraction in Represents the three-dimensional features of the feature map, h represents the feature map height, w represents the feature map width, and c represents the feature map depth; 2) Perform average pooling on the input feature map F horizontally to obtain the feature map Z after horizontal pooling: Z=CrossPool(F) Among them, CrossPool represents horizontal average pooling, and the feature map after horizontal pooling 3) Downsample the feature map Z after horizontal pooling vertically to obtain the feature map Z' after vertical downsampling: Z'=dw(Z) Among them, dw represents downsampling, the feature map after vertical downsampling 4) Use three 1×1 convolutional layers to extract the features of the feature map Z' after vertical downsampling: 4-1) Compress the depth of the feature map Z' after vertical downsampling to obtain the feature map Z1 after vertical compression: Z1=kenerl 1×1 ·Z' Among them, the scaling ratio is r, kenerl 1×1 is a 1×1 convolution kernel, the compressed feature map 4-2) The feature map Z1 after longitudinal compression is depth-enhanced and activated using the Relu activation function to obtain the feature map Z2 after longitudinal depth enhancement: Z2=δ(kenerl 1×1 ·Z1) Among them, δ represents the Relu activation function, and the feature map after the vertical depth is improved 4-3) Adjust the depth of the feature map Z2 after the longitudinal depth enhancement to make it consistent with the depth of the input feature map F' after feature extraction, and use the Sigmoid function to map the result to (0,1) to obtain the longitudinal feature weight vector Z3: Z3=σ(kenerl 1×1 ·Z2) Among them, σ represents the Sigmoid activation function, and the vertical feature weight vector 5) Upsample the vertical feature weight vector Z3 vertically to restore it to the size of the input feature map vertically, and obtain the vertical attention weight map A: A=up(Z3) Among them, the vertical attention weight map 3. The method for classifying drone monitoring targets by fusion of stereo features and semantic segmentation according to claim 1 is characterized in that: The horizontal attention weight map obtained in the stereo attention module constructed in step 1 specifically includes: 1) Input feature map Perform average pooling in the vertical direction to obtain the feature map U after vertical pooling: U=VerticalPool(F) Among them, VerticalPool represents the vertical average pooling, the feature map after vertical pooling; 2) The feature map U after vertical pooling is horizontally downsampled to obtain the feature map U' after horizontal downsampling: U'=dw(U) Among them, dw represents downsampling, the feature map after horizontal downsampling 3) Use three 1×1 convolutional layers to extract the features of the feature map U' after horizontal downsampling: 3-1) Compress the depth of the feature map U' after horizontal downsampling to obtain the horizontally compressed feature map U1: U1=kenerl 1×1 ·IN' Among them, the scaling ratio is r, kenerl 1×1 is a 1×1 convolution kernel, the compressed feature map 3-2) The feature map U1 after horizontal compression is depth-enhanced and activated using the Relu activation function to obtain the feature map U2 after horizontal depth enhancement: U2=δ(kenerl 1×1 ·U1) Among them, δ represents the Relu activation function, and the feature map after horizontal depth enhancement 3-3) Adjust the depth of the feature map U2 after horizontal depth enhancement to make it consistent with the depth of the input feature map F' after feature extraction, and use the Sigmoid function to map the result to (0,1) to obtain the horizontal feature weight vector U3: U3=σ(kenerl 1×1 ·U2) Among them, σ represents the Sigmoid activation function, and the horizontal feature weight vector 4) Upsample the horizontal feature weight vector U3 horizontally to restore it to the size of the input feature map horizontally, and obtain the horizontal attention weight map B: B=up(U3) Among them, the horizontal attention weight map 4. The method for classifying drone monitoring targets by fusion of stereo features and semantic segmentation according to claim 1 is characterized in that: In step 1, the vertical attention weight map and the horizontal attention weight map are used to weight the input feature map, specifically including: 1) The number of vertically overlapping pixels between the two adjacent frames of images on the UAV’s motion trajectory is recorded as λ A , the number of horizontally overlapping pixels is denoted as λ B , the direction adaptive weight W is obtained according to the number of vertical and horizontal overlapping pixels d : Among them, h represents the height of the feature map, and w represents the width of the feature map; 2) Perform weighted operations based on the longitudinal attention weight map, the lateral attention weight map, and the direction adaptive weight to obtain the weighted feature map Among them, A represents the vertical attention weight map, B represents the horizontal attention weight map, F' represents the input feature map after feature extraction, and × represents the multiplication operation.
5. The method for classifying drone monitoring targets by fusion of stereo features and semantic segmentation according to claim 1 is characterized in that: The process of obtaining the attention weight of the adaptive channel weight pooling module in step 2 specifically includes: 1) Input feature map Perform global maximum pooling and global average pooling respectively, merge the pooled feature maps in series and input them into the multi-layer perceptron; 2) Use 1×1 convolution to adjust the feature size of the pooled feature map to 1×1×c, and use the Sigmoid function to map the result to (0,1) to obtain the attention weight M of the adaptive channel weight pooling module: M=σ(Kernel 1×1 ·FC(contact(avgPool(F),maxPool(F)))) Among them, σ represents the Sigmoid activation function, Kernel 1×1 represents a 1×1 convolution kernel, represents a convolution operation, FC represents a multi-layer perceptron, contact represents a cascade merge operation, avgPool represents global average pooling, maxPool represents global maximum pooling, h represents the height of the feature map, w represents the width of the feature map, c represents the depth of the feature map, and attention weights Then the feature map F after the attention weight weighting of the adaptive channel weight pooling module is output for: F output =M×F+F=(M+1)×F Among them, × represents the multiplication operation, the weighted feature map 6. The method for classifying drone monitoring targets by fusion of stereo features and semantic segmentation according to claim 1 is characterized in that: The processing process of the input feature map by the adaptive channel weight pooling module in step 2 specifically includes: 1) Input feature map Divide into g×g grids, perform global average pooling on each grid, and obtain the output feature map P: P = avgPool(F) Among them, h represents the height of the feature map, w represents the width of the feature map, c represents the depth of the feature map, avgPool represents the global average pooling, g∈1,3,5,7, 2) Adjust the number of channels of the output feature map P to 1 / 4 of the original one, and obtain the adjusted feature map P': P'=Kernel 1×1 ·P Among them, Kernel 1×1 represents a 1×1 convolution kernel, represents a convolution operation, and the adjusted feature map 3) The adjusted feature maps P' are upsampled to the original image size, and are combined in series with the original input feature map F, and weighted by the weight of the adaptive channel to obtain the adaptive weighted feature map Y: Y=(M+1)×(contact(up(P'),F)) Among them, up represents upsampling, contact represents the series merging operation, × represents the multiplication operation, M represents the attention weight of the adaptive channel, and the adaptive weighted feature map 4) Adjust the number of channels to be consistent with the original input feature map to obtain the adjusted feature map: F”=Kernel 1×1 ·Y in, 7. The method for classifying drone monitoring targets by fusion of stereo features and semantic segmentation according to claim 1 is characterized in that: The processing of the encoder in the convolutional neural network in step 3 specifically includes: The first 8 layers of the MobilenetV2 network convolutional layer are selected as the feature extraction backbone network, denoted as Conv1~Conv8 respectively. The stereo attention module is inserted between the convolutional layers, and the output feature maps of the Conv3, Conv4, and Conv8 layers are selected as the input of the iterative fusion network, denoted as The final feature map is obtained through a three-step fusion network; Among them, the three-step fusion network obtains the context information of the backbone network extraction features as follows: When the step is greater than 2, the feature map depths of the three branches are adjusted to be consistent through 1×1 convolution, and the input feature map of step s and level t is Input feature map of s-1 steps and t levels operated by adaptive channel weight pooling module Input feature map of s-1 steps and t-1 layers operated by the adaptive channel weight pooling module Fusion to obtain fusion results Among them, Kernel 1×1 represents a 1×1 convolution kernel, · represents a convolution operation, and ⊙ represents the corresponding element-wise multiplication operation; When the step is 1, the feature map depths on the two branches are adjusted to be consistent through 1×1 convolution, and the input feature map of s=0 step and t level is Input feature map of s=0 steps and t-1 layers operated by adaptive channel weight pooling module Fusion, get the fusion result F1 t : Among them, up represents upsampling.
8. The method for classifying drone monitoring targets by fusion of stereo features and semantic segmentation according to claim 1 is characterized in that: The processing of the decoder in the convolutional neural network in step 3 specifically includes: The feature map output by the 3-step iterative fusion network is upsampled and concatenated with the Conv2 layer feature map to obtain the concatenated feature map E: Among them, F Conv2 represents the feature map of the Conv2 layer, F3 represents the feature map output by the 3-step iterative fusion network, contact represents the serial merging operation, and up represents upsampling; The concatenated feature map is upsampled through a 3×3 convolution layer, the width and height of the feature map are restored to the input image size, and then the depth of the feature map is adjusted to the number of classifications through a 1×1 convolution layer, and the pixel-level classification operation is completed through the Softmax layer to obtain the pixel classification result R: R=ζ(kernel 1×1 ·up(kernel 3×3 ·E)) Among them, ζ represents the Softmax activation function, Kernel 1×1 Represents 1×1 convolution kernel, Kernel 3×3 represents a 3×3 convolution kernel, and up represents upsampling.