A method for automatically generating annotations for large-scale remote sensing image datasets

Abstract

The present invention discloses a method for automatically generating annotations for large-scale remote sensing image datasets. In this method, a two-stage semi-supervised method is proposed for generating positioning annotations for remote sensing datasets. For a specific target, such as an airport, only two parts of sample images, one containing the specific target and the other not containing the specific target, are needed to generate pixel-level positioning annotations using the method of this system. At the same time, a new method is proposed to more accurately locate targets in remote sensing images. A dataset containing positive sample images (containing the specific target) and negative sample images (not containing the specific target) is passed through a three-classifier network to obtain a preliminary positioning area. The present invention replaces the dual classifier with a three-classifier to improve accuracy and generation speed.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image annotation, and specifically provides a method for automatically generating annotations for large-scale remote sensing image datasets. Background Art

[0002] With the widespread use of deep learning, there is an increasing demand for large-scale image datasets for training models. Remote sensing images often contain objects such as airports and train stations, and pixel-level localization of these objects is very important in many applications.

[0003] However, generating a large-scale image dataset with accurate annotations for training localization models is not an easy task, as manual annotation is time-consuming and labor-intensive. Therefore, we choose to utilize automatic annotation methods. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for automatically generating annotations for large-scale remote sensing image datasets in order to solve the above-mentioned problems.

[0005] The technical solution adopted by the present invention is as follows: a method for automatically generating annotations for a large-scale remote sensing image dataset, the method comprising the following steps:

[0006] S1: It is necessary to establish a dataset that includes two types of images, namely images with specific targets and images without specific targets. Then, this system adopts a two-stage strategy to generate pixel-level positioning annotations. Figure 5 As shown in the figure, the prepared dataset is first fed into a three-class classification model to obtain preliminary pixel-level annotations of those images with specific objects (stage 1), and then a transformer-based model is applied to obtain more accurate annotations by refining the above annotations (stage 2).

[0007] S2: In the first stage, the input image first passes through the backbone, which is trained on the prepared two-category dataset for feature extraction, and then passes through three parallel classifiers for object localization. In the second stage, the transformer structure based on the moving window is used to further refine the results

[0008] S3: Use the classification model to obtain preliminary labels; using the classification model to locate objects of semantic interest has been shown to be an effective method. A similar approach is also used in the solution of this system, but this system further constructs a triple classifier model to capture more accurate areas caused by background clutter and target diversity. According to experiments, this triple classification model can achieve more accurate results than the model used in

[0009] S4: Use a transformer-based network to refine the results. In the transformer-based saliency model in the second stage, a moving window-based attention calculation mechanism is introduced to capture more global information.

[0010] S5: The transformer consists of Lc basic transformer layers. The encoder output T3 passes through this layer, resulting in output D3. Finally, D3 is processed by the decoder, using the RT2T structure as the backbone, to obtain the final refined region, thus concluding the entire generation process.

[0011] In a preferred embodiment, in step S4, the image is first divided into a labeled sequence T1 and fed into the encoder using the T2T-ViT backbone. Then, through two T2T processes, the system obtains T2 and T3, respectively. Furthermore, the system employs a moving window mechanism within these T2T processes to improve performance.

[0012] In a preferred embodiment, in step S4, the window-based attention module is replaced with a moving window-based attention module. For high-resolution remote sensing images, global attention computation between markers will incur a relatively large cost. To reduce this cost, this system introduces a moving window partitioning strategy.

[0013] In a preferred embodiment, in step S3, a dataset including positive sample images (including specific targets) and negative sample images (not including specific targets) is processed through a three-classifier network to obtain a preliminary positioning area.

[0014] In a preferred embodiment, in step S4, the transformer-based saliency model is divided into three parts: an encoder, a converter, and a decoder.

[0015] In a preferred embodiment, in step S5, similar to T2T, the RT2T architecture incorporates a moving window-based attention calculation method to improve final localization precision and accuracy. Based on D3, after two RT2T processes, the system obtains T2D and T1D, respectively. Finally, the system restores T1D as the final output.

[0016] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0017] In this paper, a two-stage semi-supervised method is proposed for generating location annotations for remote sensing datasets. For a specific target, such as an airport, only two sample images, one containing the specific target and one excluding the specific target, are needed to generate pixel-level location annotations using this system. A new method is also proposed to more accurately locate targets in remote sensing images. A dataset containing positive sample images (containing the specific target) and negative sample images (excluding the specific target) is passed through a three-classifier network to obtain a preliminary location region. The present invention replaces the two classifiers with a single three-classifier network to improve accuracy and generation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the method of the present invention;

[0019] Figure 2 It is the overall structural diagram of the present invention;

[0020] Figure 3 This is a diagram describing the first stage of the present invention;

[0021] Figure 4 This is a diagram describing the second stage of the present invention;

[0022] Figure 5 This is a diagram demonstrating the results of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0024] Reference Figure 1-5 ,

[0025] Example:

[0026] A method for automatically generating annotations for a large-scale remote sensing image dataset comprises the following steps:

[0027] S1: It is necessary to establish a dataset that includes two types of images, namely images with specific targets and images without specific targets. Then, this system adopts a two-stage strategy to generate pixel-level positioning annotations. Figure 1 As shown in the figure, the prepared dataset is first fed into a three-class classification model to obtain preliminary pixel-level annotations of those images with specific objects (stage 1), and then a transformer-based model is applied to obtain more accurate annotations by refining the above annotations (stage 2).

[0028] S2: In the first stage, the input image first passes through the backbone, which is trained on the prepared two-category dataset for feature extraction, and then passes through three parallel classifiers for object localization. In the second stage, the transformer structure based on the moving window is used to further refine the results

[0029] S3: Use the classification model to obtain preliminary labels; using the classification model to locate objects of semantic interest has been proven to be an effective method. A similar approach is also used in the solution of this system, but this system further constructs a triple classifier model to capture more accurate areas caused by background clutter and target diversity. The triple classifier model contains three parallel classifier structures. Each classifier structure contains some convolutional layers, GAP and a softmax layer. The feature map obtained by the first classifier is passed to the second classifier as input after the identification area is eliminated, and the feature map obtained by the second classifier is passed to the third classifier as input after the identification area is eliminated. Finally, the three saved feature maps are fused to obtain the final feature map, which is the final result after threshold conversion. According to experiments, this triple classification model can achieve more accurate results than the model used in

[0030] S4: Use a transformer-based network to refine the results. In the transformer-based saliency model in the second stage, a moving window-based attention calculation mechanism is introduced to capture more global information.

[0031] S5: The transformer consists of Lc basic transformer layers. The encoder output T3 passes through this layer, resulting in output D3. Finally, D3 is processed by the decoder, using the RT2T structure as the backbone, to obtain the final refined region, thus concluding the entire generation process.

[0032] In step S4, the image is first divided into a labeled sequence T1 and fed into the encoder using the T2T-ViT backbone. Then, through two T2T processes, the system obtains T2 and T3, respectively. Furthermore, the system employs a moving window mechanism in these T2T processes to improve performance.

[0033] In step S4, the window-based attention module is replaced by a moving window-based attention module. For high-resolution remote sensing images, global attention computation between markers will incur a relatively large cost. To reduce this cost, this system introduces a moving window partitioning strategy.

[0034] In step S3, a dataset containing positive sample images (containing specific targets) and negative sample images (not containing specific targets) is processed through a three-classifier network to obtain a preliminary positioning area.

[0035] In step S4, the transformer-based saliency model is divided into three parts: an encoder, a converter, and a decoder.

[0036] In step S5, similar to T2T, the RT2T architecture introduces a moving window-based attention calculation method to improve final positioning precision and accuracy. Based on D3, after two RT2T processes, the system obtains T2D and T1D, respectively. Finally, the system restores T1D as the final output.

[0037] In this paper, a two-stage semi-supervised method is proposed for generating location annotations for remote sensing datasets. For a specific target, such as an airport, only two sample images, one containing the specific target and one excluding the specific target, are needed to generate pixel-level location annotations using this system. A new method is also proposed to more accurately locate targets in remote sensing images. A dataset containing positive sample images (containing the specific target) and negative sample images (excluding the specific target) is passed through a three-classifier network to obtain a preliminary location region. The present invention replaces the two classifiers with a single three-classifier network to improve accuracy and generation speed.

[0038] like Figure 2 As shown in the overall structure, this system proposes a two-stage semi-supervised method for generating localization annotations for remote sensing datasets. For a specific target, such as an airport, only two sample images, one containing the target and one excluding the target, are needed to generate pixel-level localization annotations using this system.

[0039] like Figure 3 As shown in the figure, in the first stage of the classification model, due to the complexity of remote sensing images, this system replaces the dual classifier with a three-classifier to improve accuracy. Previously, some people proposed applying classification networks to optical images to capture the most important areas and use them as image annotations. However, the accuracy of these annotations is often insufficient to train deep networks. Inspired by this approach, this system proposes a new method to more accurately locate targets in remote sensing images. A dataset containing positive sample images (containing specific targets) and negative sample images (not containing specific targets) is passed through a three-classifier network to obtain preliminary localization areas.

[0040] like Figure 4As shown in Figure 2, in the transformer-based saliency model in the second stage, this system introduces a moving window-based attention calculation mechanism to capture more global information. These two operations can further improve the accuracy of annotations.

[0041] We conducted experiments to generate annotations for large-scale datasets such as airports, stadiums, and train stations. The scale and annotation quality of these datasets demonstrate the effectiveness of our approach.

[0042] To demonstrate the effectiveness of our method, we collected an airport dataset consisting of 4,998 images (2,499 airport images and 2,499 non-airport images), a train station dataset consisting of 364 images (182 train station images and 182 non-train station images), and a stadium dataset consisting of 406 images (203 stadium images and 203 non-stadium images). All images were resized to 512×512, and our method was used to generate localization annotations for these three image categories.

[0043] Figure 5 Several annotated examples are provided for comparison. The circled area represents the preliminary results obtained in the first stage using a two-classifier network, denoted as S1. The circled area represents the preliminary results obtained in the first stage using a three-classifier network, denoted as S2. Comparing S1 and S2, our system demonstrates that the three-classifier model achieves superior results. These comparative results demonstrate that our approach achieves superior results compared to the techniques in Section A.

[0044] Furthermore, the circled area represents the result obtained after the second stage using the basic Transformer model, denoted as S3. The circled area represents the final result obtained after the second stage using the improved Transformer architecture, denoted as S4. Comparing S3 and S4, we can see that the introduction of the moving window attention calculation mechanism also played a significant role. These comparative results are sufficient to demonstrate that our approach achieves superior results compared to the techniques mentioned at the end of Section B.

[0045] Therefore, by inputting the original image, S4 can be obtained as the annotation through the method of this system, so that the annotation of the data set can be automatically generated.

[0046] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0047] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically generating annotations for a large-scale remote sensing image dataset, characterized by: The method for automatically generating annotations for a large-scale remote sensing image dataset comprises the following steps: S1: Build a dataset of two types of images, i.e., images with and without objects, and adopt a two-stage strategy to generate pixel-level localization annotations. The prepared dataset is first fed into a three-class classification model to obtain preliminary pixel-level annotations for those images with objects, and then a transformer-based model is applied to obtain more accurate annotations by refining the annotations. S2: In the first stage, the input image first passes through the backbone, which is trained on a prepared two-category dataset for feature extraction, and then passes through three parallel classifiers for object localization. In the second stage, a moving window-based transformer structure is used to further refine the results; S3: Use the classification model to obtain preliminary labels; use the classification model to locate objects of semantic interest, and further build a triple classifier model to capture more accurate areas caused by background clutter and target diversity; the triple classifier model contains three parallel classifier structures; each classifier structure contains some convolutional layers, GAP and a softmax layer; the feature map obtained by the first classifier is passed to the second classifier as input after the identification area is eliminated, and the feature map obtained by the second classifier is passed to the third classifier as input after the identification area is eliminated; finally, the three saved feature maps are fused to obtain the final feature map, which is the final result after threshold conversion; S4: Use a transformer-based network to refine the results. In the transformer-based saliency model in the second stage, a moving window-based attention calculation mechanism is introduced to capture more global information. S5: The transformer consists of Lc basic transformer layers. The encoder output T3 passes through this layer to obtain the output D3. Finally, D3 is processed by the decoder, using the RT2T structure as the backbone to obtain the final refined region, and then the entire generation process ends. In step S4, the window-based attention module is replaced by the moving window-based attention module; for high-resolution remote sensing images, a moving window division strategy is introduced; the input is an 8×8 patch feature map, which contains 4×4 patches in the l-layer window, and the window is divided into 2×2 windows of size 4×4; in the l+1 layer, a moving window division strategy is used; some of the newly generated windows span the boundaries between adjacent windows in the previous layer; the 4×4 window in the upper left corner of the l-layer enters the new four windows in the l+1 layer respectively; the 2×2 part in the upper left corner of the l-layer 4×4 window becomes a new independent window, the 2×2 part in the upper right corner of the l-layer 4×4 window is merged into the 2×4 new window in the l+1 layer, the 2×2 part in the lower left corner of the l-layer 4×4 window is merged into the 4×2 new window in the l+1 layer, and the 2×2 part in the lower right corner of the l-layer 4×4 window is merged into the 4×4 new window in the l+1 layer.

2. The method for automatically generating annotations for a large-scale remote sensing image dataset according to claim 1, characterized in that: In step S4, the image is divided into a labeled sequence T1 and fed into the encoder with T2T-ViT as the backbone; then, two T2T processes are performed to obtain T2 and T3 respectively; a moving window mechanism is used in the T2T process to improve performance.

3. The method for automatically generating annotations for a large-scale remote sensing image dataset according to claim 1, wherein: In step S3, the dataset containing the positive sample images and the negative sample images is passed through a three-classifier network to obtain a preliminary positioning area.

4. The method for automatically generating annotations for a large-scale remote sensing image dataset according to claim 1, wherein: In step S4, the transformer-based saliency model is divided into three parts: an encoder, a converter, and a decoder.

5. The method for automatically generating annotations for a large-scale remote sensing image dataset according to claim 1, characterized in that: In step S5, the RT2T structure introduces an attention calculation method based on a moving window; Based on D3, two RT2T processes are performed to obtain T2D and T1D respectively; finally, T1D is restored to the final output.

Images