A method for multi-scale fusion and segmentation of semantic features of UAV images

Through the feature fusion of the encoder-decoder network model and element-by-element addition and multi-head self-attention mechanism, the problem of difficulty in capturing global and local context details in drone images is solved, and the accuracy and semantic representation ability of aerial image segmentation are improved .

CN117496378BActive Publication Date: 2024-11-05CHINA ELECTRONICS TECH GRP NO 7 RES INST
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311382041.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-23
Publication Date
2024-11-05
Estimated Expiration
2043-10-23

AI Technical Summary

Technical Problem

The difficulty in capturing global and local contextual details of drone images, especially in complex backgrounds and multi-scale variation aerial image segmentation tasks, leads to poor semantic segmentation, especially for small-category objects such as humans. Low.

Method used

An encoder-decoder network model is designed to generate local and global context details through 4 feature modules of different scales, and features are fusion using element-by-element addition and multi-head self-attention mechanisms to generate the final segmentation output.

Benefits of technology

It improves the semantic segmentation accuracy of drone images, can capture the inherent features and rich semantic representation of aerial images more accurately, and enhances the network's feature representation ability, especially when dealing with complex scenes and small-category objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117496378B_ABST
    Figure CN117496378B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for multi-scale fusion and segmentation of semantic features of drone images, as follows: construct an encoder-decoder network model, input the acquired drone image into the encoder-decoder network model to obtain the final segmentation output; the encoder-decoder network model includes an encoder network and a decoder network; the encoder network includes 4 feature modules connected in sequence for generating features of different scales; each feature module includes a token module for generating features of different scales for fusing local spatial context details between adjacent pixels, and a global aggregation module for capturing global context details and fine-grained local context details in drone images. The present invention has the advantage of capturing global context details and fine-grained local context details in drone images, and for drone images, a more accurate segmentation effect can be obtained compared with commonly used general solutions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a method for multi-scale fusion and segmentation of semantic features of unmanned aerial vehicle images. Background Art

[0002] Dense prediction at the pixel level of an image is widely used in computer vision. One of them is semantic segmentation, which involves assigning a label to each pixel. Autonomous driving, medical images, point cloud segmentation, and satellite scenes are its main application areas. In recent years, due to the development of intelligent technologies such as drones and robots, drones have been widely used, providing a simple and cheap solution for capturing images at remote locations. The images captured by drone devices contain a lot of information and can be used for scene understanding, traffic estimation, infrastructure planning, infrastructure planning, and road or building extraction using deep learning. With the increasing application of drones and other unmanned aerial devices, aerial image segmentation has brought new opportunities and challenges for urban scene understanding.

[0003] Existing techniques based on Convolutional Neural Networks (CNNs) have always dominated image segmentation. Most existing frameworks also use different encoder variants based on Fully Convolutional Networks (FCNs). FCN-based backbones tend to produce coarse predictions and the receptive field usually becomes constant after a certain limit. Due to the small kernel size in CNNs, the receptive field is local in nature, which enables it to extract fine-grained local context details and it is difficult to capture global context details.

[0004] However, aerial images usually contain complex backgrounds,scenes. There are multiple variations in object appearance and scale,which makes the segmentation task challenging. Existing networks designed for aerial scene segmentation can produce satisfactory results,but their feature extractor networks have difficulty capturing the true intrinsic characteristics of aerial images. Their feature extractor networks have difficulty segmenting minority categories,especially tiny objects like humans, which have the least pixels in the image. For segmenting highly complex drone images,capturing both global and local details is an important factor to obtain accurate results.

[0005] Aerial images have great variability in scale, object appearance, and complex backgrounds. The task of semantic segmentation becomes challenging when capturing the underlying features in the global and local context of drone images. Accurate semantic segmentation techniques for urban aerial scenes are still relatively scarce. How to use the network to accurately capture the inherent features of aerial images and generate an overall semantically rich feature representation. And further process these features using a multi-scale feature fusion strategy to enhance the network's feature representation capabilities is an urgent problem to be solved. Summary of the invention

[0006] In order to solve the above problems of the shortcomings and defects in the prior art, the present invention provides a method suitable for multi-scale fusion and segmentation of semantic features of drone images.

[0007] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows:

[0008] A method for multi-scale fusion and segmentation of semantic features of drone images, the method comprising the following steps:

[0009] Construct an encoder-decoder network model and input the acquired drone image into the encoder-decoder network model to obtain the final segmentation output;

[0010] The encoder-decoder network model includes an encoder network that takes the drone image as input and generates four different scale features, and a decoder network that fuses the four different scale features generated by the encoder network using element-by-element addition and generates a semantic map of each class of the drone image;

[0011] The encoder network includes four feature modules connected in sequence for generating features of different scales; each feature module includes a token module for generating features of different scales that fuse local spatial context details between adjacent pixels, and a global aggregation module for capturing global context details and fine-grained local context detail information in drone images;

[0012] The decoder network includes an upsampling module, an element-by-element addition module, and a 1x1 convolution operation module;

[0013] The upsampling module performs upsampling processing on the features of different scales generated by the four feature modules to obtain four features of the same scale as the input drone image;

[0014] The element-by-element addition module performs element-by-element addition processing on the 4 features output by the upsampling module that are of the same scale as the input drone image;

[0015] The 1x1 convolution operation module projects the dimension of the features output by the element-by-element addition module to the number of categories N to obtain the final segmentation output.

[0016] Preferably, the token module performs overlapping tokenization on the input drone image or on the features output by the feature module of the previous stage; during the tokenization process, the labeled feature space is projected to the embedding dimension.

[0017] Furthermore, the tokenization is represented by converting the feature dimension [h, w, c] of the feature into a 2-dimensional tag sequence [M, t 2 ×E], where M is the total number of tokens generated on the feature, M = (h × w) / t 2, the dimension of each token is [t, t], and E is the embedding dimension.

[0018] Furthermore, the four feature modules use a size of [m 1 , m 2 , m 3 , m 4 ] token; the embedding dimensions of the corresponding stage of each feature module are set to [E 1 ,E 2 ,E 3 ,E 4 ].

[0019] Preferably, the global aggregation module comprises a first normalization layer, a multi-head self-attention mechanism layer, a second normalization layer, and a multi-layer perceptron connected in sequence;

[0020] Among them, the input of the first normalization layer is also jump-connected to the output of the multi-head self-attention layer;

[0021] The input of the second normalization layer is also skip-connected to the output of the multilayer perceptron.

[0022] Furthermore, the multi-head self-attention mechanism layer includes multiple self-attention operations, and the sequence label [M, E] is projected into the QKV vector. The dimension of QKV is [M, C]. The formula of self-attention is as follows:

[0023]

[0024] Where G(x) = LayerNorm(Conv R×R (x)); W Q ,W K ,W V They correspond to the linear projection weights of the Q, K, and V vectors respectively; in G(x), x represents the K or V vector, which is implemented using RxR convolution operations and normalization, and R represents the sequence reduction ratio.

[0025] Furthermore, the Q, K, V vectors are divided into h parts and multi-head attention operations are performed in parallel, where h represents the number of heads, and the output of each head is concatenated and projected to the embedding dimension.

[0026] Furthermore, the multilayer perceptron includes two fully connected layers and a convolutional layer between the two fully connected layers, and the activation function adopts RELU.

[0027] Furthermore, the upsampling module includes a 3x3 convolution layer, a batch normalization layer, a Relu activation function layer and an upsampling multiplier factor operation layer.

[0028] A computer device comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the method for multi-scale fusion and segmentation of semantic features of drone images as described above is implemented

[0029] The beneficial effects of the present invention are as follows:

[0030] The global aggregation module in the encoder network of the present invention has the advantage of capturing global context details and fine-grained local context details in drone images. The token module in the encoder network is used to generate different scale features that fuse local spatial context details between adjacent pixels. This multiple different scale feature representations help capture the local context around adjacent pixels with different complexities. The decoder network uses element-by-element addition to fuse the different scale features to obtain the final segmentation output. The use of element-by-element addition for fusion enriches the semantic expression, and for drone images, a more accurate segmentation effect can be obtained compared to the commonly used general scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a block diagram of the working principle of the method for multi-scale fusion and segmentation of semantic features of drone images described in the present invention.

[0032] Figure 2 It is a flow chart of the encoder network described in the present invention.

[0033] Figure 3 It is a flow chart of the global aggregation module described in the present invention.

[0034] Figure 4 is a flow chart of the decoder network of the present invention. DETAILED DESCRIPTION

[0035] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0036] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0037] Example 1

[0038] like Figure 1 , Figure 2 As shown, a method for multi-scale fusion and segmentation of semantic features of drone images is provided, and the method comprises the following steps:

[0039] Construct an encoder-decoder network model and input the acquired drone image into the encoder-decoder network model to obtain the final segmentation output;

[0040] The encoder-decoder network model includes an encoder network that takes the drone image as input and generates four different scale features, and a decoder network that fuses the four different scale features generated by the encoder network using element-by-element addition and generates a semantic map of each class of the drone image;

[0041] The encoder network includes four feature modules connected in sequence for generating features of different scales; each feature module includes a token module for generating features that fuse local spatial context details between adjacent pixels, and a global aggregation module for capturing global context details and fine-grained local context detail information in drone images;

[0042] The decoder network includes an upsampling module, an element-by-element addition module, and a 1x1 convolution operation module;

[0043] The upsampling module performs upsampling processing on the features of different scales generated by the four feature modules to obtain four features of the same scale as the input drone image;

[0044] The element-by-element addition module performs element-by-element addition processing on the 4 features output by the upsampling module that are of the same scale as the input drone image;

[0045] The 1x1 convolution operation module projects the dimension of the features output by the element-by-element addition module to the number of categories N to obtain the final segmentation output.

[0046] In this embodiment, Figure 1 , 2 As shown in the figure, the encoder network takes the drone image as input and generates four features of different scales; F1, F2, F3 and F4 in the figure represent four feature modules connected in sequence, which output features of different scales respectively. The decoder network takes these features as input and processes them to make them have the same resolution, and then fuses the features of the same resolution to generate the final segmentation result.

[0047] In this embodiment, the token module performs overlapping tokenization on the input drone image or on the features output by the feature module in the previous stage; during the tokenization process, the labeled feature space is projected to the embedding dimension.

[0048] In this embodiment, the tokenization is represented by converting the feature dimensions [h, w, c] of features of different scales into a 2-dimensional tag sequence [M, t 2 ×E], where M is the total number of tokens generated on the feature, M = (h × w) / t 2 , the dimension of each token is [t,t], E is the embedding dimension, which is obtained by converting the previous parameters. The calculation of the embedding dimension E is as follows: the feature dimension is 3D [h,w,c], which is later changed to 2D [M,t 2 ×E], the algorithm of M is known, requiring hwc = M × t 2 ×E, since h, w, c are all known and the value of t does not affect the calculation, the embedding dimension E is calculated.

[0049] In this embodiment, the four feature modules use a size of [m 1 , m 2 , m 3 , m 4 ] token; the embedding dimensions of the corresponding stage of each feature module are set to [E 1 ,E 2 ,E 3 ,E 4 ];

[0050] Usually, choosing a small token will achieve better results. Based on experience, the four feature modules use tokens of size [7, 3, 3, 3]. The embedding dimensions of each feature module are set to [64, 128, 320, 512]. This tokenization method generates features of different scales and fuses local spatial context details between adjacent pixels. Adjacent pixels are very correlated with each other, and fusing their details between tokens helps capture local context details. The size of the token [m 1 , m 2 , m 3 , m 4 ], and the embedding dimensions of the feature modules [64, 128, 320, 512] can be adjusted as long as a certain proportional relationship is maintained. After each feature module, the spatial resolution of the feature is reduced by 1 / 4 compared to the previous stage, and the feature dimension increases. The output of the first feature module generates the highest resolution feature, and the last feature module generates the lowest resolution feature.

[0051] In this embodiment, if Figure 3As shown, the global aggregation module includes a first normalization layer, a multi-head self-attention mechanism layer, a second normalization layer, and a multi-layer perceptron connected in sequence;

[0052] Among them, the input of the first normalization layer is also jump-connected to the output of the multi-head self-attention layer;

[0053] The input of the second normalization layer is also skip-connected to the output of the multilayer perceptron.

[0054] The multi-head self-attention mechanism layer in this embodiment has no specific requirements on the type of the multi-head self-attention mechanism, and this embodiment is not limited to a fixed type.

[0055] In this embodiment, each feature module includes several global aggregation modules, wherein the number of global aggregation modules of the four feature modules is set to [3, 6, 40, 3].

[0056] Furthermore, the multi-head self-attention mechanism layer includes multiple self-attention operations, and the sequence label [M, E] is projected into the QKV vector. The dimension of QKV is [M, C]. The formula of self-attention is as follows:

[0057]

[0058] Where G(x) = LayerNorm(Conv R×R (x)); W Q ,W K ,W V They correspond to the linear projection weights of the Q, K, and V vectors respectively; in G(x), x represents the K or V vector, which is implemented using RxR convolution operations and normalization, and R represents the sequence reduction ratio.

[0059] In this embodiment, a smaller token size will bring the cost of a longer sequence length, and performing the self-attention operation will become more expensive. Considering the memory and computational burden, the sequence length of the K and V vectors can be appropriately reduced.

[0060] In the above formula, the dot product between Q and K calculates the similarity (i.e., attention score) between all sequence tokens. The attention score is then divided by the dimension of the vector K, where the C in the denominator is used to control stability, guarantee, and scale. The Softmax operation is then applied to the resulting matrix to normalize the attention score. The softmax is a commonly used normalization method and will not be described in detail here.

[0061] Finally, the normalized attention score is multiplied by the vector V to filter out the most important feature space for further processing; each token interacts with other tokens during the self-attention operation and thus perceives the global context details.

[0062] In the embodiment, a multi-head attention mechanism is introduced to divide the Q, K, and V vectors into h parts, and multi-head attention operations are performed in parallel, where h represents the number of heads, and the output of each head is connected together and projected to the embedding dimension.

[0063] The self-attention operation can capture relatively distant context details, which helps the network maintain semantically rich global context information for complex drone scenes. The specific setting of h is determined according to the actual situation and is a configurable parameter.

[0064] Furthermore, the multilayer perceptron includes two fully connected layers and a convolutional layer between the two fully connected layers, and the activation function adopts RELU.

[0065] In general, the encoder network generates features of different scales and uses self-attention and convolution operations to capture global context information in drone images. This embodiment helps maintain the semantic feature representation of complex aerial scenes through multiple different scale feature outputs, and also captures edge information well.

[0066] In this embodiment, the decoder network includes an upsampling module, an element-by-element addition module, and a 1x1 convolution operation module;

[0067] The upsampling module performs upsampling processing on the features of different scales generated by the four feature modules to obtain four features of the same scale as the input drone image;

[0068] The element-by-element addition module performs element-by-element addition processing on the 4 features output by the upsampling module that are of the same scale as the input drone image;

[0069] The 1x1 convolution operation module projects the dimension of the features output by the element-by-element addition module to the number of categories N to obtain the final segmentation output.

[0070] Furthermore, the upsampling module includes a 3x3 convolution layer, a batch normalization layer, a Relu activation function layer and an upsampling multiplier factor operation layer.

[0071] like Figure 4 As shown in Figure 2, the decoder network processes the different scale feature outputs of the encoder network. The encoder network outputs four features with input image ratios of 1 / 4, 1 / 8, 1 / 16, and 1 / 32, and the structure is as follows: Figure 4 shown.

[0072] The decoder network fuses these multiple scale features to achieve the final pixel-level prediction of the drone image. Figure 4The C4 features in the encoder network are processed, and the features undergo three upsampling steps to convert them to the same scale as the input drone image. Since this feature comes from the last stage of the encoder network, it contains very coarse information. Each upsampling step consists of a 3x3 convolution, batch normalization, Relu activation function, and an upsampling multiplier factor operation. Similarly, the features with a scale of 1 / 8 and 1 / 16 also undergo one and two upsampling steps respectively. In the upsampling step, the feature dimension C is set to 256, and the output features from the first feature module of the encoder network undergo a 3x3 convolution operation to expand their feature dimensions, followed by batch normalization and Relu activation. All four output features of the encoder network are now at a uniform scale of 1 / 4 of the input image. Finally, they are upsampled to the same size as the input image using bilinear upsampling with a scale factor of 4. These features are fused using element-wise addition. The fused feature output contains both low-level and high-level semantic information. The dimension of this feature is then projected to the number of categories N (this is determined by the data sample, divided into several categories, such as 2 categories, people and cars) using a 1x1 convolution operation. This operation generates a semantic map for each class of the input drone image.

[0073] The different scale features output from the encoder network contain rich semantic representation information with very rich global and local context details. Fusion of these features further makes the final semantic representation more robust, and the information of minority categories such as people and cars is well preserved.

[0074] In a specific embodiment, a computer device includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of implementing the method for multi-scale fusion and segmentation of semantic features of drone images as described above are as follows:

[0075] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0076] In another specific embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of implementing the method for multi-scale fusion and segmentation of semantic features of drone images as described above are as follows:

[0077] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0078] Obviously, the above embodiments of the present invention are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for multi-scale fusion and segmentation of semantic features of drone images, characterized by: The method comprises the following steps: Construct an encoder-decoder network model and input the acquired drone image into the encoder-decoder network model to obtain the final segmentation output; The encoder-decoder network model includes an encoder network that takes the drone image as input and generates four different scale features, and a decoder network that fuses the four different scale features generated by the encoder network using element-by-element addition and generates a semantic map of each class of the drone image; The encoder network includes four feature modules connected in sequence for generating features of different scales; each feature module includes a token module for generating features that fuse local spatial context details between adjacent pixels, and a global aggregation module for capturing global context details and fine-grained local context detail information in drone images; The decoder network includes an upsampling module, an element-by-element addition module, and a 1x1 convolution operation module; The upsampling module performs upsampling processing on the features of different scales generated by the four feature modules to obtain four features of the same scale as the input drone image; The element-by-element addition module performs element-by-element addition processing on the 4 features output by the upsampling module that are of the same scale as the input drone image; The 1x1 convolution operation module projects the dimension of the features output by the element-by-element addition module to the number of categories N to obtain the final segmentation output; wherein the token module performs overlapping tokenization on the input drone image or on the features output by the feature module of the previous stage; during the tokenization process, the labeled feature space is projected to the embedding dimension; and, The global aggregation module includes a first normalization layer, a multi-head self-attention mechanism layer, a second normalization layer, and a multi-layer perceptron connected in sequence; Among them, the input of the first normalization layer is also jump-connected to the output of the multi-head self-attention layer; The input of the second normalization layer is also skip-connected to the output of the multilayer perceptron.

2. The method for multi-scale fusion and segmentation of semantic features of drone images according to claim 1, characterized in that: The tokenization is represented as converting the feature dimension [h,w,c] of the feature into a 2-dimensional tag sequence [M,t 2 ×E], where M is the total number of tokens generated on the feature, M = (h × w) / t 2 , the dimension of each token is [t, t], and E is the embedding dimension.

3. The method for multi-scale fusion and segmentation of semantic features of drone images according to claim 2 is characterized by: The four feature modules use tokens of size [m1, m2, m3, m4]; the embedding dimensions of the corresponding stage of each feature module are set to [E1, E2, E3, E4] respectively.

4. The method for multi-scale fusion and segmentation of semantic features of drone images according to claim 1, characterized in that: The multi-head self-attention mechanism layer includes multiple self-attention operations, and the sequence label [M, E] is projected into the QKV vector. The dimension of QKV is [M, C]. The formula of self-attention is as follows: Where G(x) = LayerNorm(Conv R×R (x)); W Q ,W K ,W V They correspond to the linear projection weights of the Q, K, and V vectors respectively; in G(x), x represents the K or V vector, which is implemented using RxR convolution operations and normalization, and R represents the sequence reduction ratio.

5. The method for multi-scale fusion and segmentation of semantic features of drone images according to claim 4, characterized in that: The Q, K, V vectors are divided into h parts and multi-head attention operations are performed in parallel, where h represents the number of heads. The output of each head is concatenated and projected to the embedding dimension.

6. The method for multi-scale fusion and segmentation of semantic features of drone images according to claim 1, characterized in that: The multilayer perceptron includes two fully connected layers and a convolutional layer between the two fully connected layers, and the activation function adopts RELU.

7. The method for multi-scale fusion and segmentation of semantic features of drone images according to claim 1, characterized in that: The upsampling module includes a 3x3 convolution layer, a batch normalization layer, a Relu activation function layer and an upsampling multiplier factor operation layer.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the method for multi-scale fusion and segmentation of semantic features of drone images as described in any one of claims 1 to 7 is implemented.