Information processing device, segmentation method and program
The MAC model addresses the challenges of scale variation and complex backgrounds in aerial images by using a multi-scale attention cascade with Swin Transformer and FPN, enhancing segmentation accuracy for small and large objects.
Patent Information
- Application Number
- JP2023085680
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-12-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Conventional semantic segmentation techniques struggle with accurately segmenting objects of varying scales and handling complex backgrounds in aerial images, leading to challenges in identifying small and large objects and producing false positives.
A multi-scale attention cascade (MAC) model is employed, utilizing a Swin Transformer Tiny and Feature Pyramid Network (FPN) backbone, with multi-head self-attention modules applied at different scales and enhanced by Efficient Channel Attention (ECA) to process multi-scale features and suppress complex backgrounds.
The MAC model effectively segments both small and large objects in aerial images, improving accuracy at ambiguous boundaries and reducing false positives by leveraging multi-scale feature representations.
Smart Images

Figure 2024168754000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for performing segmentation on an aerial image. [Background technology]
[0002] In the following description, references are indicated with numbers such as [1], and the names of the references corresponding to the numbers are listed at the end of the specification.
[0003] High spatial resolution (HSR) remote sensing techniques are known. Images obtained by these techniques are characterized by their richness in geospatial information that provides the meaning and location of objects of interest, including buildings, vehicles, ships, etc. Understanding this information is essential for a variety of practical purposes, including urban monitoring, environmental change monitoring, disaster response, and route planning.
[0004] To effectively utilize HSR remote sensing images, aerial image segmentation is an important computer vision task that aims to separate (segment) foreground objects and background regions while assigning semantic labels to each image pixel from an aerial perspective.
[0005] Recently, many Vision Transformer-based methods have been proposed in the field of semantic segmentation. Although these methods can generate accurate predictions for small and ambiguous objects, they cannot solve the problem of large scale variations in aerial images.
[0006] In addition, the dense affinity learning based method [9] disclosed in Non-Patent Document 2 degrades its segmentation results due to complex background and noise context. Pointflow [9] is the latest work in the field of aerial image segmentation research that shows SoTA segmentation results, and employs sparse affinity learning by selecting and matching salient points between adjacent scale features in the Feature Pyramid Network (FPN) [4] disclosed in Non-Patent Document 1. Although the technique can handle complex background and noise, the method may lose small objects and weakly predict the boundaries of large objects. [Prior art documents] [Non-patent literature]
[0007] [Non-Patent Document 1] Tsung-Yi Lin, Piotr Doll´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition (CVPR), pages 2117-2125, 2017. [Non-Patent Document 2] Xiangtai Li, Hao He, Xia Li, Duo Li, Guangliang Cheng, Jianping Shi, Lubin Weng, Yunhai Tong, and Zhouchen Lin. Pointflow: Flowing semantics through points for aerial image segmentation. In Proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition (CVPR), 2021 Summary of the Invention [Problem to be solved by the invention]
[0008] Conventional semantic segmentation techniques have the problem that they are unable to properly perform segmentation (region identification) of aerial images.
[0009] The present invention has been made in consideration of the above points, and has an object to provide a technique for appropriately performing segmentation on aerial images. [Means for solving the problem]
[0010] According to the disclosed technology, there is provided an information processing device that performs segmentation on an image, comprising: An input unit for inputting the image; a computing unit for generating a second feature map by applying a plurality of multi-head self-attention modules to each of the first feature maps of the plurality of levels obtained from the image, and generating a segmentation result from the second feature maps of the plurality of levels; An information processing device comprising: Effect of the Invention
[0011] According to the disclosed technology, a technology is provided for appropriately performing segmentation on an aerial image. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a configuration diagram of an information processing device 100. [Diagram 2] 2 is a flowchart illustrating an operation of the information processing device 100. [Diagram 3] FIG. 1 is a diagram illustrating the configuration of a MAC model. [Figure 4] FIG. 2 is a diagram illustrating the configuration of a mac module. [Diagram 5] 1 is a diagram illustrating an example of a hardware configuration of an information processing device 100. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.
[0014] In the following description, "aerial image" may be replaced with aerial photograph, aerial image, aerial photography, etc. In addition, the "aerial image" below is assumed to be an image acquired (taken) from the sky (e.g., from an airplane or satellite) using HSR remote sensing technology, but is not limited to this. For example, the "aerial image" may be an image taken from the sky with a general camera.
[0015] Furthermore, "segmentation" may also be referred to as semantic segmentation, segmentation, region division, region discrimination, region identification, region classification, etc. By performing segmentation on an aerial image using the technology according to the present embodiment, for example, buildings, automobiles, airplanes, roads, the ground, etc. in the aerial image can be identified.
[0016] The technology according to the present invention can also be applied to images other than aerial images.
[0017] (About the assignment) Many conventional techniques for semantic segmentation perform general semantic segmentation tasks in natural scenes. However, unlike general semantic segmentation tasks, aerial image segmentation has three challenges:
[0018] 1) There is a problem that the scale of objects varies widely in the same scene: the scale of objects in aerial images varies over a very wide range, making it difficult to segment very small objects from very large objects.
[0019] 2) Aerial imagery, such as HSR imagery, faces the challenge of having many small objects spread across a large aerial image, making it difficult to clearly recognize and segment them, especially when they have fuzzy boundaries.
[0020] 3) Aerial images have the challenge of having unbalanced and complex backgrounds, i.e., the proportion of foreground is much less than the complex background, which may introduce noise into the modeling and, in turn, cause significant false positives in the output.
[0021] The technology according to the present embodiment for solving the above problems will be described in detail below.
[0022] (Device configuration) In this embodiment, information processing device 100 shown in FIG. 1 executes segmentation on an aerial image.
[0023] As shown in FIG. 1, the information processing device 100 includes an input unit 110, a calculation unit 120, an output unit 130, and a storage unit 140.
[0024] An outline of the operation of information processing device 100 will be described with reference to the flowchart in Fig. 2. In S101, an aerial image to be processed is input from input unit 110. The input image is stored in storage unit 140.
[0025] In S102, the calculation unit 120 executes a segmentation process by reading out the aerial image from the storage unit 140. In S103, the output unit 130 outputs the segmentation process result (for example, an image in which meaning is assigned to each region (each pixel)).
[0026] In this embodiment, the calculation unit 120 executes segmentation using a neural network model. For example, learned model parameters are stored in the storage unit 140. The calculation unit 120 reads out the model parameters from the storage unit 140, inputs the aerial image to a model to which the model parameters are applied, and obtains the segmentation result as the output of the model.
[0027] The above model will now be described in detail.
[0028] (Model Overview) An example of the configuration of the model in this embodiment is shown in Fig. 3. The model in this embodiment is called a Multi-scale Attention Cascade (MAC) model. As shown in Fig. 3, the MAC model has a Swin Transformer Tiny (Swin-T) [5] and an FPN [4] as a backbone 121 and a neck 122.
[0029] In the MAC model, a multi-scale attention cascade (MAC) module is applied to the output of each level of the FPN to further explore the features of the merged pyramid. The MAC module consists of three successive local multi-head self-attention (MSA) modules at different scales.
[0030] An ECA (Efficient channel attention) [6] module is applied to the output of each MAC module to realize enhanced inter-channel interaction. At the end of the model, the multi-level features are reshaped to the same size and the dimensions are concatenated to obtain the segmentation result output after a simple segmentation decoder.
[0031] The MAC model is explained in more detail below.
[0032] (FPN-based segmentation framework) First, we provide an overview of the FPN [4] used in the MAC model.
[0033] The MAC model uses input image I∈R H×W×3 Given,multiscale and multi-resolution features Ci={C2,···,C5} are generated by the backbone 121 via a bottom-up route [7,2,5].
[0034] At neck 122, the FPN applies lateral connections fi and upsampling Up2× to match the shapes and channels of different feature maps.
[0035] Then, the FPN builds a feature pyramid by fusing (adding) adjacent feature maps in a pixel-wise manner and propagating through a top-down path. After the above multi-scale feature fusion, one 3 × 3 convolution layer is applied on each merged feature map to resolve the aliasing effect.
[0036] Furthermore, additional global context features are obtained by applying a pyramid pooling module (PPM) [8] to C5. Finally, a pyramidal feature Pi = {P2, , P6} with a fixed number of channels (usually 256-D) is generated. The whole process can be expressed as follows:
[0037]
number
[0038] (Window-wise MSA (Window-wise Multi-head Self-Attention)) Next, we explain the process of window-wise MSA used in the mac module. To calculate the local MSA, the input feature map F∈R H×W×C is a set of non-overlapping windows Mi = {M1, ,Mn} ∈ R N×k×k×C where k×k is the size of each window, and N=h×w / k 2 is the number of windows. Each window is flattened into a 1-D sequence:
[0039]
number
[0040]
number
[0041]
number
[0042]
number
[0043]
number
[0044]
number
[0045]
number
[0046] The mac module (and the 1×1 Conv and ECA modules connected to it) are shown in Figure 4. As shown in Figure 4, the mac module contains three Casi stages (i∈{1,2,3}), and in each stage, the feature map is spatially divided into windows of different sizes. The size of the window in each stage i is ki×ki={2×2,4×4,7×7}. In each stage, we use windows of different sizes.
[0047] That is, in the first stage, we calculate self-attention for each small region (2×2), and then gradually expand the region to medium (4×4) and large (7×7) and calculate self-attention.
[0048] Pyramid feature Pi ∈ R at level i hi×wi×256 When input, it is first fed into a 1×1 convolutional layer (1×1Conv) to reduce its dimensions to
[0049]
number
[0050]
number
[0051] In addition to the above processing, to achieve cross-channel attention at each FPN level, ECA [6] is used to compact homogeneous features along the channel dimension. i out applies to.
[0052] Finally, the output from the ECA module is reconstructed by bilinear interpolation to obtain the input image I ∈ R H×W×3The image is resized to 1 / 4 of the original size, and the multi-scale contextual features Cas at each FPN level are calculated as shown in Equation (6) below. i out ´∈R H / 4×W / 4×192 is generated.
[0053]
number
[0054] (Segmentation Decoder) The segmentation decoder uses multi-level feature Cas i out ´∈R H / 4×W / 4×192 ,i={2,···,6} are fused to obtain the final segmentation output Out∈R H×W×class where class is the number of categories of the segmentation target. The details of the segmentation decoder are as follows:
[0055] After applying the mac module to the FPN, there is some scale overlap between different levels. Therefore, to match the same scales at different levels, we first apply channel concatenation to fuse the features of all levels into one feature map. Then, we apply the Squeeze and Excitation (SE) module [3] to the fusion result to achieve global channel attention. The final segmentation output is obtained by a 1 × 1 convolution layer. The above process can be expressed as follows:
[0056]
number
[0057] (Hardware configuration example) Any of the information processing devices 100 described in the present embodiment can be realized by, for example, causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.
[0058] That is, the device can be realized by executing a program corresponding to the processing performed by the device, using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.
[0059] Fig. 5 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 5 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are mutually connected by a bus BS. The computer may further include a GPU.
[0060] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.
[0061] The memory device 1003 reads out and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the information processing device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network or the like. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard and mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the results of calculations.
[0062] (Summary of the embodiment, effects, etc.) As described above, in this embodiment, by adopting the MAC model, it is possible to solve the problems of the conventional technology regarding segmentation of aerial images such as HSR remote sensing images.
[0063] In addition, by using an attention module with multiple window sizes, we can exploit the multi-scale feature representations at each FPN level, solving the problem of large scale variation while suppressing complex and imbalanced background.
[0064] We evaluate our method (MAC) against the conventional transformer-based method on the iSAID dataset and find that MAC outperforms the conventional transformer-based method, providing accurate segmentation results for both very small and very large objects, especially for fuzzy boundaries.
[0065] The following supplementary notes are further disclosed regarding the above embodiment.
[0066] <Additional Notes> (Additional note 1) An information processing device that performs segmentation on an image, Memory, at least one processor coupled to the memory; Including, The processor, Input the image, Applying a plurality of multi-head self-attention modules to each of the plurality of levels of the first feature maps obtained from the image to generate a second feature map, and generating a segmentation result from the plurality of levels of the second feature maps. Information processing device. (Additional note 2) The plurality of multi-head self-attention modules are cascaded. Item 1. An information processing device according to claim 1. (Additional note 3) Each of the multi-head self-attention modules divides an input feature map into a plurality of windows and calculates self-attention for each window. 3. The information processing device according to claim 1 or 2. (Additional note 4) In the plurality of cascaded multi-head self-attention modules, the window size of a multi-head self-attention module in a certain stage is smaller than the window size of a multi-head self-attention module connected to a subsequent stage of the multi-head self-attention module. 4. The information processing device according to claim 3. (Additional note 5) The processor generates the first feature map at multiple levels using a feature pyramid network. 5. An information processing device according to any one of claims 1 to 4. (Additional note 6) A segmentation method executed by an information processing device that performs segmentation on an image, comprising: inputting the image; generating a second feature map by applying a plurality of multi-head self-attention modules to each of the first feature maps of the plurality of levels obtained from the image, and generating a segmentation result from the second feature maps of the plurality of levels; A segmentation method comprising: (Additional note 7) A non-transitory storage medium storing a program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 5.
[0067] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [References] [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. [2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. [3] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition (CVPR), pages 7132-7141, 2018. [4] Tsung-Yi Lin, Piotr Doll´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition (CVPR), pages 2117-2125, 2017. [5] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer:Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), pages 10012-10022, 2021. [6] Pengfei Zhu Peihua Li Wangmeng Zuo Qilong Wang, Banggu Wu and Qinghua Hu. Eca-net: Efficient channel attention for deep convolutional neural networks. In Proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition (CVPR), pages 11534-11542, 2020. [7] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. [8] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition (CVPR), pages 2881-2890, 2017. [9] Xiangtai Li, Hao He, Xia Li, Duo Li, Guangliang Cheng, Jianping Shi, Lubin Weng, Yunhai Tong, and Zhouchen Lin. Pointflow: Flowing semantics through points for aerial image segmentation. In Proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition (CVPR), 2021 [Explanation of symbols]
[0068] 100 Information processing device 110 Input section 120 Arithmetic section 121 Backbone 122 Neck 130 Output section 140 Storage section 1000 Drive Device 1001 Recording media 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output device
Claims
1. An information processing device that performs segmentation on an image, An input unit for inputting the image; a computing unit for generating a second feature map by applying a plurality of multi-head self-attention modules to each of the first feature maps of the plurality of levels obtained from the image, and generating a segmentation result from the second feature maps of the plurality of levels; An information processing device comprising:
2. The plurality of multi-head self-attention modules are cascaded. The information processing device according to claim 1 .
3. Each of the multi-head self-attention modules divides an input feature map into a plurality of windows and calculates self-attention for each window. The information processing device according to claim 1 .
4. In the plurality of cascaded multi-head self-attention modules, the window size of a multi-head self-attention module in a certain stage is smaller than the window size of a multi-head self-attention module connected to a subsequent stage of the multi-head self-attention module. The information processing device according to claim 3 .
5. The computing unit generates the first feature map at multiple levels using a feature pyramid network. The information processing device according to claim 1 .
6. A segmentation method executed by an information processing device that performs segmentation on an image, comprising: inputting the image; generating a second feature map by applying a plurality of multi-head self-attention modules to each of the first feature maps of the plurality of levels obtained from the image, and generating a segmentation result from the second feature maps of the plurality of levels; A segmentation method comprising:
7. A program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 5.
