A change detection and annotation method for remote sensing images

Abstract

The present invention discloses a change detection and annotation method for remote sensing images. The method inputs two remote sensing images of a target scene taken at different times. After selecting an automatic annotation method, the contours of the selected annotated targets in the remote sensing images are extracted to assist in generating data labels using an AI model. The AI ​​model sets a special weighted attention mechanism between the encoder and decoder of the original CDNET network structure. The weighted attention mechanism consists of a channel attention module, a spatial attention module, and a scale attention module connected in series. Through the improved AI model structure, the present invention improves the annotation efficiency, accuracy, and consistency of change detection in remote sensing images, reduces reliance on manual annotation, and thus alleviates some of the shortcomings of manual annotation, facilitating the rapid batch annotation of changed objects in remote sensing images.

Description

Technical Field

[0001] The present invention relates to the technical field of image detection, and in particular to a change detection and annotation method for remote sensing images. Background Art

[0002] Remote sensing image processing refers to the process of digitally processing and analyzing image data collected by satellites, aircraft, or other sensors using remote sensing technology to extract geographic, environmental, or resource information. Remote sensing image processing has a wide range of applications, including land management, agriculture, environmental protection, resource management, urban planning, and weather forecasting. Remote sensing image processing can provide more geographic information, thereby helping people better understand and manage the Earth's surface. Change detection and annotation of remote sensing images is important because it provides critical information about surface changes in multiple fields, with wide applications in environmental monitoring, resource management, urban planning, and emergency response. Currently, the mainstream method for annotating change targets involves manually comparing remote sensing data such as buildings and environments in two different images at different times, identifying differences, and then drawing outlines to mark them. This process requires a large number of professionals to perform specialized operations.

[0003] Existing change detection methods for remote sensing images often rely on manual annotation, which is time-consuming, labor-intensive, and inefficient. Due to factors such as the nature of the task, cost, accuracy, and scalability, these methods suffer from the following drawbacks:

[0004] 1. Labor-intensive: Manual annotation usually requires a lot of human resources, especially for large-scale datasets. This leads to high costs and long annotation times, especially for large projects.

[0005] 2. Limited scalability: Manual annotation does not easily scale to large-scale datasets, so in many cases the amount of data is limited, which may limit the performance of deep learning models.

[0006] 3. High time cost: Labeling large-scale datasets requires a lot of time, which delays the progress of the project and may lead to loss of competitiveness in a highly competitive market.

[0007] 4. Difficulty in labeling complex tasks: For some complex tasks, such as soil and water conservation, manual labeling may require professional knowledge, which will increase the labeling cost.

[0008] 5. Privacy and security issues: Some map data contain sensitive information, and manual annotation may involve privacy and security risks, especially when outsourcing annotation tasks.

[0009] 6. Sample bias and inconsistent annotation quality: During the manual annotation process, the subjective preferences of the annotator may lead to data sample bias, which may affect the generalization ability of the model. In addition, the different levels and experience of the annotators may lead to inconsistent annotation quality, which is a problem for supervised learning algorithms.

[0010] 7. Subjectivity and inconsistency: Different annotators may produce different contour annotation results for the same dataset because annotation is a subjective process. This leads to data inconsistency, which in turn affects model performance.

[0011] Based on this, there is an urgent need to design a change detection and annotation method for remote sensing images, so as to improve the annotation efficiency, accuracy and consistency of change detection in remote sensing images, reduce the dependence on manual annotation, and alleviate some of the shortcomings of manual annotation. Summary of the Invention

[0012] (1) Technical issues to be solved

[0013] Based on the above problems, the present invention provides a change detection and annotation method for remote sensing images, which utilizes artificial intelligence image processing technologies such as model quantization and transfer learning to improve the annotation efficiency, accuracy and consistency of change detection in remote sensing images, and reduce the dependence on manual annotation, thereby alleviating some shortcomings of manual annotation and facilitating batch annotation of changed objects in remote sensing images.

[0014] (2) Technical solution

[0015] Based on the above technical problems, the present invention provides a change detection and annotation method for remote sensing images. Two remote sensing images of a target scene taken at different times are input. After selecting the automatic annotation mode, the contours of the selected annotated targets in the remote sensing images are extracted to generate data labels through an AI model.

[0016] The AI ​​model sets up a weighted attention mechanism between the encoder and decoder of the original CDNET network structure. The weighted attention mechanism consists of a channel attention module, a spatial attention module, and a scale attention module connected in series. In the weighted attention mechanism, F is the input feature map, F^ is the output feature map, and C is the number of channels in the input feature map.

[0017] In the channel attention module, the global spatial information of F is first compressed using maximum pooling and average pooling to generate two feature maps of size 1×1×C. Then, two one-dimensional feature maps are obtained through multi-layer perception. The two one-dimensional feature maps are normalized to obtain weighted features. Figure 1 ; In the spatial attention module, the weighted features Figure 1Input the 1×1×1 convolution module to activate F, and input the result into the Sigmoid function to obtain the weighted feature Figure 2 ; In the scale attention module, the weighted features are first Figure 2 Perform global pooling, then perform full connection operation, and input the result into the linear rectification activation function, perform full connection operation on the output result again, and finally output it to the Sigmoid activation function to obtain weighted features Figure 3 ;

[0018] The weighted features Figure 1 , weighted features Figure 2 and weighted features Figure 3 The output feature map F^ is obtained by serial connection through element summation and then executing the Sigmoid activation function.

[0019] Furthermore, the encoding and decoding structure of the AI ​​model is as follows: the input image is processed through two 3*3 convolutions to obtain the output features. Figure 1 ,feature Figure 1 After one downsampling, the features are obtained through two 3*3 convolutions. Figure 2 , repeating 3*3 convolution and downsampling operations three times in sequence can obtain the features Figure 3 、 4 , 5, and then the feature Figure 5 After the linear projection dimensionality reduction process, it passes through the encoder-decoder structure Transformer and is sent to the weighted attention module to obtain the feature Figure 6 , the characteristics Figure 6 and features Figure 4 After fusion, the features are obtained by two 3*3 deconvolutions and upsampling. Figure 7 , the characteristics Figure 7 and features Figure 3 After fusion, features are obtained by two 3*3 deconvolutions and upsampling. Figure 8 , the characteristics Figure 8 and features Figure 2 After fusion, features are obtained by two 3*3 deconvolutions and upsampling. Figure 9 , the characteristics Figure 9 and features Figure 1 After fusion, the processed output image can be obtained by two 3*3 deconvolutions.

[0020] Furthermore, the change detection and annotation method of the remote sensing image also includes a manual annotation mode. After the manual annotation mode is selected, the remote sensing image is manually annotated using the Labelme annotation mode to obtain data labels.

[0021] Furthermore, the training data set of the AI ​​model is the large data set ImageNet, and the encoding and decoding feature information obtained by training on the ImageNet data set is migrated to the AI ​​model using transfer learning.

[0022] Furthermore, the change detection and annotation method for remote sensing images is developed under the PyQt graphics design framework, and QScrollArea is used in conjunction with QPainter to realize the display and annotation functions of remote sensing images.

[0023] Furthermore, the method of selecting the marked target specifically includes selecting the target by boxing it in the remote sensing image and / or directly clicking on the target.

[0024] The present invention also discloses a remote sensing image change detection and annotation system, comprising: at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the remote sensing image change detection and annotation method.

[0025] (3) Beneficial effects

[0026] The above technical solution of the present invention has the following advantages:

[0027] (1) The present invention designs AI-assisted annotation software for remote sensing image change detection datasets, and uses artificial intelligence technologies such as model quantization and transfer learning to assist in the automatic annotation of the outer contours of changed objects in remote sensing images, so as to improve the efficiency, accuracy and consistency of batch annotation of images, and reduce the dependence on manual annotation, thereby alleviating some shortcomings of manual annotation.

[0028] (2) For complex remote sensing images, in order to reproduce high-resolution feature maps in the target saliency map, the present invention selectively improves the CDNET network structure and sets a special weighted attention mechanism between the encoder and the decoder, so that the contour features of complex objects such as buildings and water environments in remote sensing images can be effectively extracted. A new image contour segmentation algorithm RemoteCDNet is obtained. The algorithm is used to realize real-time segmentation on remote sensing images. After the RemoteCDNet algorithm connects the weighted attention mechanism to the Transformer structure, it can well enhance the attention of channel, space and scale information, and still maintain the simplicity and symmetry of the overall image processing structure. The computational processing speed for color high-resolution image segmentation is fast, and it can perform fast automatic annotation in batch images. In addition, the present invention also uses transfer learning to speed up the training speed of the model. The training data set of the new segmentation algorithm RemoteCDNet improved based on CDNET is ImageNet. Transfer learning is used to migrate the feature information encoded and decoded in the algorithm trained for this data set to the new segmentation algorithm RemoteCDNet to improve the accuracy of the model.

[0029] (3) Through experiments, it can be known that the automatic annotation system of the present invention can be used in combination with the manual annotation system. That is, if the annotation box obtained by automatic annotation is appropriate, the annotation can be confirmed. If the annotation box is found to be inappropriate after comparing two images at different times, manual annotation can be used to freely annotate the changed objects in the remote sensing image that cannot be identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0031] Figure 1 It is a schematic diagram of the technical solution of the change detection and annotation method of remote sensing images in the present invention;

[0032] Figure 2 This is the AI ​​model structure diagram for automatic annotation of remote sensing images in the present invention. The blue arrows in the figure represent the convolution operation in the encoder or the deconvolution operation in the decoder, and the red arrows represent the upsampling operation in the decoder or the downsampling operation in the encoder.

[0033] Figure 3 This is a schematic diagram of the weighted attention mechanism structure in the AI ​​model of the present invention;

[0034] Figure 4 These are two image renderings taken at different times in an embodiment of the present invention;

[0035] Figure 5This is an effect diagram of partially magnifying two images simultaneously in an embodiment of the present invention;

[0036] Figure 6 This is an effect diagram of global reduction of two images simultaneously in an embodiment of the present invention;

[0037] Figure 7 This is a manually labeled result diagram in an embodiment of the present invention;

[0038] Figure 8 This is a result diagram of the AI-assisted annotation method in an embodiment of the present invention;

[0039] Figure 9 This is the result diagram of the second AI-assisted labeling method in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0041] Since the objects of change detection and annotation of remote sensing images are often complex and the remote sensing images required to be annotated are often of high resolution, in order to improve the processing efficiency, accuracy and consistency of annotation, and reduce the dependence on manual annotation and alleviate the shortcomings of manual annotation, the present invention improves and designs a change detection and annotation method for remote sensing images.

[0042] like Figure 1 As shown, the change detection and annotation method of remote sensing images of the present invention includes manual and automatic annotation schemes. First, based on the input of two remote sensing images T1 and T2 of the target scene at different times (i.e., remote sensing images of the same area obtained at different times), the operator selects manual annotation or automatic annotation based on the changed area in the image. If manual annotation is selected, the labelme annotation method is used to manually annotate and obtain the data label. If automatic annotation is selected, the outline of the target is extracted through AI-assisted annotation. The target is selected by clicking the dot or box to select the annotated target, and the data label is generated by the AI ​​model.

[0043] The AI ​​models used in automatic annotation are as follows: Figure 2 As shown in the figure, this AI model is an image segmentation algorithm RemoteCDNet based on the improved CDNET network structure. While maintaining the advantage of the CDNET network in reproducing high-resolution feature maps, RemoteCDNet can use this algorithm to achieve real-time and accurate segmentation on complex remote sensing images.

[0044] In the prior art, the original CDNET uses an encoder-decoder architecture with skip connections. The encoder network is responsible for obtaining an abstract feature map from the input image by preserving the object's structural information. Meanwhile, the decoder network aims to reproduce the high-resolution feature map from the low-resolution object saliency map obtained by the encoder network, thereby obtaining a refined object-level transmission map. CDNET consists of five encoder blocks and five decoder blocks. Each encoder block consists of a convolutional layer, a normalization layer, and an activation function. Each decoder block consists of a deconvolutional layer, a dropout layer, a normalization layer, a skip connection layer, and an activation function. The convolutional and deconvolution layers in the entire network have a 3×3 size, and a stride of 2 for both convolutional and deconvolutional layers.

[0045] See also Figure 2 As shown in the figure, based on the original CDNET architecture, this paper sets a special weighted attention mechanism between the encoder and decoder of CDNET, so as to effectively extract the contour features of complex objects such as buildings and water environments in remote sensing images. Specifically, the input image is processed through two 3*3 convolutions to obtain the output features. Figure 1 ,feature Figure 1 After one downsampling, the features are obtained through two 3*3 convolutions. Figure 2 , repeating 3*3 convolution and downsampling operations three times in sequence can obtain the features Figure 3 、 4 , 5. Then set the feature Figure 5 After the linear projection dimensionality reduction processing, it passes through the encoder-decoder structure Transformer and is sent to the weighted attention module to obtain the feature Figure 6 .feature Figure 6 and features Figure 4 After fusion, features are obtained by two 3*3 deconvolutions and upsampling. Figure 7 ,feature Figure 7 and features Figure 3 After fusion, features are obtained by two 3*3 deconvolutions and upsampling. Figure 8 ,feature Figure 8 and features Figure 2 After fusion, features are obtained by two 3*3 deconvolutions and upsampling. Figure 9 ,feature Figure 9 and features Figure 1 After fusion, the processed output image can be obtained by two 3*3 deconvolutions. That is, in the decoder, only the deconvolution and upsampling operations need to be repeated and fused with the corresponding feature map jump connection to obtain the final prediction image.

[0046] Figure 3The figure shows the newly added weighted attention mechanism in CDNET of the present invention, which includes a channel attention module, a spatial attention module, and a scale attention module. Compared with other attention mechanisms, the weighted attention mechanism of the present invention can pay attention to channel, spatial information, and scale information at the same time, thereby effectively extracting a variety of complex image features of high-resolution remote sensing images. Specifically, the weighted attention mechanism consists of a channel attention module, a spatial attention module, and a scale attention module connected in series. The output result of the convolutional layer in the encoder first passes through the channel attention module to obtain a weighted result, and then passes through the spatial attention module and the scale attention module to obtain the final weighted result.

[0047] The specific structure and relationship of the three parts in the weighted attention mechanism are as follows Figure 3 As shown in the figure, F is the input feature map output by the last layer encoder (i.e., feature Figure 5 Transformed feature map), H and W are the height and width of the feature map respectively, and C is the number of channels of the input feature map.

[0048] In the channel attention module, the global spatial information of the input feature map F is first compressed using maximum pooling and average pooling to generate two feature maps of size 1×1×C. Then two one-dimensional feature maps are obtained through multi-layer perception, and the two one-dimensional feature maps are normalized to obtain weighted features. Figure 1 That is, the channel attention module can compress the feature map in the spatial dimension, obtain a one-dimensional vector, and then perform operations.

[0049] In the spatial attention module, the weighted features Figure 1 Input the 1×1×1 convolution module to activate F, and input the result into the Sigmoid function to obtain the weighted feature Figure 2 That is, the spatial attention module compresses the channel and performs average pooling and maximum pooling in the channel dimension.

[0050] In the scale attention module, the weighted features are first Figure 2 Perform global pooling, then perform full connection operation, and input the result into the linear rectification activation function, perform full connection operation on the output result again, and finally output it to the Sigmoid activation function to obtain weighted features Figure 3 .

[0051] Weighted Features Figure 1 , weighted features Figure 2 and weighted features Figure 3 The output feature map F^ is obtained by serial connection through element summation and then executing the Sigmoid activation function.

[0052] It should be noted that, in the present invention, the feature Figure 5 and features Figure 6The overall computational complexity of the weighted attention mechanism between the input feature map F and the output feature map F^ is not large, and it can be well embedded in the original CDNET network. Although the three-step weighted calculation based on the input feature map F to the output feature map F^ prolongs the encoding and decoding step length, it has been found through experiments that it does not affect the overall running speed of the image segmentation processing too much, and can also well extract the channel, spatial information and scale information in the remote sensing image at the same time, so as to quickly and accurately complete the automatic annotation task of complex remote sensing images.

[0053] Furthermore, in the RemoteCDNet algorithm, by integrating the weighted attention mechanism into the Transformer architecture, transfer learning can be used to accelerate model training and improve accuracy. The new segmentation algorithm, RemoteCDNet, is trained on the ImageNet dataset. Transfer learning is used to transfer the encoded and decoded feature information from the algorithm trained on this dataset to the new segmentation algorithm, while maintaining high resolution for the output remote sensing segmented images.

[0054] like Figure 4-9 As shown, in order to verify the advantages of the method of the present invention, the present invention also conducted experiments shown in the following embodiments based on the above-mentioned change detection and annotation method of remote sensing images:

[0055] This embodiment designs AI-assisted annotation software for change detection datasets, and all experiments are conducted on the Windows 11 operating system. After importing two pictures or two folders, annotation can be performed while comparing dual-time images at the same time, and the pictures can be zoomed in and out, translated, etc., to reduce annotation errors and save annotation time. The software is developed under the PyQt graphics design framework. First, the software uses QScrollArea with QPainter to realize image display and annotation functions, and associates the zoom ratio and translation position of the two areas. Secondly, the software can be used Figure 2-3 The AI ​​model, trained on the large-scale Imagenet dataset, assists annotators with their annotation tasks. It features a low number of parameters, fast inference speed, and high detection accuracy, making it highly effective even with limited hardware. Finally, the software boasts comprehensive supporting features, real-time storage of annotation information, and a rich selection of annotation shapes, enabling faster and more convenient annotation tasks.

[0056] The effect diagram of software running can be seen in Figure 4-9 , Figure 4 To simultaneously open two image renderings of the target scene based on remote sensing images at different times, Figure 5 This is the effect diagram of partial magnification of two images at the same time. Figure 6 This is the global zoom-out effect of the two images at the same time. Figure 7For manually annotated renderings, Figure 8 This is the first AI-assisted labeling method. By clicking on the target, the AI ​​model automatically generates a labeling box. If the labeling box is appropriate, the labeling is confirmed. Figure 9 This is the second AI-assisted annotation method. Select the target by box, and the AI ​​model will automatically generate the annotation box. If the annotation box is appropriate, the annotation is confirmed. Figure 7 It can be seen that the accuracy of the building group annotation box obtained by manual annotation is not high enough, because the outline is often a hand-drawn straight line, which is easy to cover the road. Figure 8-9 The recognition of irregular-edged water bodies and green belt contours obtained by automatic labeling based on the AI ​​model is more accurate, and the image processing operation speed is faster, which is suitable for batch labeling of regular objects.

[0057] Finally, it should be noted that the above method can be converted into software program instructions, which can be implemented by running a detection system including a processor and a memory, or by computer instructions stored in a non-transitory computer-readable storage medium. The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations shall fall within the scope defined by the appended claims.

Claims

1. A change detection and annotation method for remote sensing images, characterized in that: Input two remote sensing images of the target scene at different times. After selecting the automatic annotation method, extract the outline of the target selected in the remote sensing image to generate data labels through the AI ​​model; The AI ​​model sets a weighted attention module between the encoder and decoder of the original CDNET network structure. The weighted attention module is composed of a channel attention module, a spatial attention module, and a scale attention module connected in series. In the weighted attention module, F is the input feature map, F^ is the output feature map, and C is the number of channels of the input feature map. In the channel attention module, the global spatial information of F is first compressed using maximum pooling and average pooling to generate two feature maps of size 1×1×C, and then two one-dimensional feature maps are obtained through multi-layer perception. The two one-dimensional feature maps are normalized to obtain a weighted feature map 1; in the spatial attention module, the weighted feature map 1 is input into the 1×1×1 convolution module to activate F, and the result is input into the Sigmoid function to obtain a weighted feature map 2; in the scale attention module, the weighted feature map 2 is first globally pooled, and then fully connected, and the result is input into the linear rectification activation function, the output result is once again fully connected, and finally output to the Sigmoid activation function to obtain a weighted feature map 3; The weighted feature map 1, the weighted feature map 2 and the weighted feature map 3 are sequentially connected in series by element summation, and then the Sigmoid activation function is executed to obtain the output feature map F^; The encoding and decoding structure of the AI ​​model is specifically as follows: the input image is subjected to two 3*3 convolutions to obtain the output feature map 1, the feature map 1 is downsampled once, and then subjected to two 3*3 convolutions to obtain the feature map 2, and the 3*3 convolution and downsampling operations are repeated three times to obtain feature maps 3, 4, and 5 in turn, and then the linear projection of the feature map 5 is subjected to dimensionality reduction processing through the encoding and decoding structure Transformer, and is sent to the weighted attention module to obtain feature map 6, the feature map 6 is fused with the feature map 4, and then two 3*3 deconvolutions and upsampling are performed to obtain feature map 7, the feature map 7 is fused with the feature map 3, and two 3*3 deconvolutions and upsampling are performed to obtain feature map 8, the feature map 8 is fused with the feature map 2, and two 3*3 deconvolutions and upsampling are performed to obtain feature map 9, and the feature map 9 is fused with the feature map 1, and then two 3*3 deconvolutions are performed to obtain the processed output image.

2. The change detection and annotation method for remote sensing images according to claim 1, characterized in that: The change detection and labeling method for remote sensing images also includes a manual labeling method. After selecting the manual labeling method, the remote sensing image is manually labeled using the Labelme labeling method to obtain data labels.

3. The change detection and annotation method for remote sensing images according to claim 1, characterized in that: The training dataset of the AI ​​model is the large dataset ImageNet, and transfer learning is used to migrate the encoding and decoding feature information obtained from training on the ImageNet dataset to the AI ​​model.

4. The change detection and annotation method for remote sensing images according to claim 1, characterized in that: The change detection and annotation method for remote sensing images is developed under the PyQt graphics design framework, and QScrollArea is used in conjunction with QPainter to realize the display and annotation functions of remote sensing images.

5. The remote sensing image change detection and annotation method according to claim 4, characterized in that: The method of selecting the marked target specifically includes selecting the target by using a box and / or directly clicking the target in the remote sensing image.

6. A remote sensing image change detection and annotation system, characterized in that: include: at least one processor; and at least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the change detection and annotation method for remote sensing images according to any one of claims 1 to 5.

Images