Optical and SAR image change detection method and device based on twin attention enhancement

Abstract

The present application discloses a method and device for detecting changes in optical and synthetic aperture radar (SAR) images based on twin attention enhancement, which involves disaster loss estimation and artificial intelligence technology, including: receiving a front-phase optical image and a back-phase SAR image; inputting the processed front-phase optical image and back-phase SAR image into a heterogeneous image translation network to output a back-phase optical image with a unified feature map; inputting the front-phase and back-phase optical images with a unified feature map into a twin attention enhanced change detection network for feature extraction; and taking the category with a higher binary classification probability value in the corresponding pixel in the twin attention enhanced change detection network as the final prediction result of the pixel to generate a complete binary change detection result. The present application can realize the effective conversion and high-precision change detection of heterogeneous images, and broaden its practical application in tasks such as disaster loss assessment and building damage analysis.

Description

Technical Field

[0001] The present application relates to the fields of disaster loss estimation and artificial intelligence technology, and in particular to a method and device for detecting optical and SAR image changes based on twin attention enhancement. Background Art

[0002] Accurately grasping the change information of ground objects is of great significance for tasks such as disaster loss assessment, building damage analysis and intelligent intelligence acquisition. However, for mission scenarios that require rapid and emergency response, if only a single homogeneous optical or synthetic aperture radar (SAR) image is used, the timeliness is often poor and it is difficult to meet the needs of actual applications. Change detection technology can accurately assess the location and degree of damage of the disaster-stricken area by analyzing remote sensing images before and after the disaster. However, due to the problems of single optical and SAR data being susceptible to weather interference and poor image interpretation in actual mission applications.

[0003] The main problems of implementing change detection methods for heterogeneous remote sensing images are: 1. After converting heterogeneous images to the same feature space, the change detection methods adopted mostly rely on unsupervised methods, such as traditional threshold segmentation or difference, but the inherent connection of heterogeneous data cannot be accurately described, and the change detection results are easily affected by noise, which will reduce the accuracy of change detection. 2. In the actual heterogeneous data change detection task, it often contains multiple scales of object categories, but the existing methods are not perfect enough to express the features of heterogeneous data, lack of modeling of different levels and context feature information of images, which easily leads to misclassification and omission of change detection models. Summary of the invention

[0004] The embodiments of the present application provide a method and device for optical and SAR image change detection based on twin attention enhancement, so as to achieve effective conversion of heterogeneous images and high-precision change detection, and broaden its practical application in tasks such as disaster loss assessment and building damage analysis.

[0005] The embodiment of the present application provides an optical and SAR image change detection method based on twin attention enhancement, comprising:

[0006] Receive the pre-phase optical image and post-phase SAR image;

[0007] Inputting the processed pre-phase optical image and post-phase SAR image into a heterogeneous image translation network to output a post-phase optical image having a unified feature map with the pre-phase optical image;

[0008] The optical images of the front and back phases of the unified feature map are input into the twin attention-enhanced change detection network for feature extraction;

[0009] The category with a higher binary classification probability value in the corresponding pixel in the twin attention-enhanced change detection network is taken as the final prediction result of the pixel to generate a complete binary change detection result.

[0010] The embodiment of the present application provides an optical and SAR image change detection device based on twin attention enhancement, comprising:

[0011] A receiving module, used for receiving the front-phase optical image and the back-phase SAR image;

[0012] A translation module, for inputting the processed pre-phase optical image and post-phase SAR image into a heterogeneous image translation network to output a post-phase optical image having a unified feature map with the pre-phase optical image;

[0013] A training module for inputting the optical images of the front and back phases of the unified feature map into the twin attention-enhanced change detection network for feature extraction;

[0014] The output module is used to take the category with a higher binary classification probability value in the corresponding pixel in the twin attention enhanced change detection network as the final prediction result of the pixel to generate a complete binary change detection result.

[0015] The method of the embodiment of the present application realizes the effective conversion of heterogeneous images and high-precision change detection, broadening its practical application in tasks such as disaster loss assessment and building damage analysis.

[0016] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0018] Figure 1 The basic process diagram of the optical and SAR image change detection method based on twin attention enhancement in an embodiment of the present application is shown;

[0019] Figure 2 This is a schematic diagram of the heterogeneous image translation network data processing flow in an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of the heterogeneous image translation network training process of an embodiment of the present application;

[0021] Figure 4 This is a schematic diagram of the data processing flow of the twin attention-enhanced change detection network in an embodiment of the present application;

[0022] Figure 5 This is a schematic diagram of the training process of the twin attention-enhanced change detection network in an embodiment of the present application;

[0023] Figure 6 Schematic diagram of an optical and SAR image change detection device based on twin attention enhancement according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0025] In view of the limitations of the existing technology in the detection of changes in heterogeneous remote sensing images, an improved method is proposed in the embodiment of the present application. This application uses the conditional generative adversarial network cGAN technology to translate heterogeneous (Synthetic Aperture Radar, SAR) images into images that are consistent with the expression of optical images. On this basis, the twin attention enhanced change detection network is further applied to generate a binary map representing the change situation, so as to improve the accuracy and efficiency of change detection, while reducing the deficiencies in the existing technology. The specific embodiment of the present application provides a method for optical and SAR image change detection based on twin attention enhancement, such as Figure 1 As shown, the following steps are included:

[0026] In step S101, a front-phase optical image and a back-phase SAR image are received. After receiving the images, the method further includes: preprocessing the optical image and the SAR image, wherein the preprocessing includes one or more of the following: radiation calibration, atmospheric correction, geometric correction, and optical and radar image registration. The preprocessing step can improve the accuracy of the model in the subsequent recognition and analysis stages.

[0027] In step S102, the processed pre-phase optical image and post-phase SAR image are input into a heterogeneous image translation network, and the two heterogeneous images are mapped to a unified feature space for image translation, so as to output a post-phase optical image having a unified feature mapping with the pre-phase optical image.

[0028] In step S103, the front and back phase optical images of the unified feature map are input into the twin attention enhanced change detection network for feature extraction. In the specific example, the twin attention enhanced change detection network can be a fully convolutional neural network that introduces a twin structure and a multi-scale convolutional block attention mechanism.

[0029] In step S104, the category with a higher binary classification probability value in the corresponding pixel in the twin attention enhanced change detection network is used as the final prediction result of the pixel to generate a complete binary change detection result. Based on the above steps, the category with a higher binary classification probability value in the corresponding pixel in the full convolutional neural network can be used as the final prediction result of the pixel, and a complete binary change detection result can be generated.

[0030] This application proposes a change detection method based on twin attention enhancement - SAE_UNet++. The method of this application is implemented by a heterogeneous data translation model and a twin attention enhanced change detection model. The heterogeneous data translation module used uses a conditional generative adversarial network to map optical data and SAR data into the same feature space, and then sends it to the change detection network for feature extraction, thereby achieving accurate optical and SAR data change detection.

[0031] In the twin attention enhanced change detection model, the introduced multi-scale attention enhancement module can perform effective feature aggregation from both spatial and channel dimensions, enhance the feature expression of heterogeneous data, and well characterize the local detail information of the target objects at different scale layers, effectively improving the accuracy and robustness of change detection of heterogeneous data under various scales of objects.

[0032] In some embodiments, Figure 2 As shown, the heterogeneous image translation network includes a discriminant network and a generative network; inputting the processed pre-phase optical image and post-phase SAR image into the heterogeneous image translation network includes:

[0033] In the generation network, the later phase SAR image is generated into an optical image corresponding to the previous phase. In a specific example, in the generation network:

[0034] Extracting key features of the post-phase SAR image based on the generation network;

[0035] The extracted key features are used to generate an optical image that is consistent with the previous phase optical image in detail texture and image structure. The embodiment of the present application is based on the classic UNet network as the basic structure, which consists of an encoder, a decoder and a jump connection. In the encoding stage, a convolution kernel of size 5×5 is used to extract SAR image features. In addition, downsampling is performed to obtain different features of multiple scale layers. In the decoding stage, transposed convolution is used for upsampling to restore the original size of the image. The generation network of the embodiment of the present application includes a total of 5 groups of downsampling and 5 groups of upsampling layers. The included jump connection can realize the fusion and reuse of encoding features and decoding features on the same scale layer by splicing the feature map channel dimension, avoiding the loss of detail information during the feature extraction process, and also enabling the decoder to restore more ground object detail information during the upsampling process.

[0036] The discriminant network is used to guide the generative network to output an image that meets a specific goal. In a specific embodiment, in the discriminant network, multi-layer convolution is used to extract features from the input image;

[0037] Map the extracted features to a fully connected layer to form a set of high-dimensional feature vectors;

[0038] The Sigmoid activation function is connected to convert the output value into a probability value between 0 and 1. The probability value reflects the degree of match between the image judged to be true and the auxiliary image. The closer the value is to 1, the closer the image distribution of the generated image is to the real image distribution based on the auxiliary information.

[0039] In some embodiments, the method further includes configuring a loss function for the heterogeneous image translation model, wherein the loss function includes Loss, adversarial loss and SSIM loss satisfy:

[0040]

[0041] in, The classic loss function in image restoration, To combat losses, Introducing the loss function of ground object texture similarity, and Represents the coefficients of different loss functions. The loss can accurately measure the distance between the generated network output image and the target image. The smaller the distance, the more similar the two images are. The specific formula of loss is as follows: .

[0042] Adversarial Loss Function The function of is that it can not only help the discriminant network identify whether the input image and the real image are consistent in data distribution, but also measure the degree of match between the input data and the auxiliary data. The mathematical formula of the adversarial loss function is as follows: .

[0043] SSIM loss describes the visual and structural similarity between two images from the perspective of image texture. The similarity of texture features in the two images is reflected by calculating the local mean, variance and other values ​​of the images. The SSIM function is used to measure the local texture similarity between the two images. The specific formula is as follows: , where and Represented Image and The mean of the image, and represents the standard deviation of the two images. is the covariance of the two images, and It is a constant defined to avoid the denominator of the fraction being 0.

[0044] In some examples, a sliding window of specified size is used in the SSIM loss, for example Using the sliding window approach, the SSIM loss of the local window is expressed as:

[0045]

[0046] in, and Represents the sliding window In Image and image;

[0047] All windows Take the average to obtain the SSIM value in the whole image range. The SSIM loss of the whole image is expressed as: .

[0048] In some embodiments, Figure 3 As shown, the method further includes training the heterogeneous image translation model using the following steps:

[0049] Receive remote sensing images from different sources such as optical images and SAR images;

[0050] The optical image and SAR image data sets are constructed respectively, and divided into a training set and a test set. For example, the data sets are divided into a training set and a test set in a ratio of 8:2.

[0051] The training data set is sent to cGAN for training, and the loss function is used to optimize the model during the training process. In this process, the number of training iterations follows a preset value, such as 10000. The preset value can be customized according to actual task requirements.

[0052] After multiple iterations of optimization, the heterogeneous image translation model with stable output parameters and minimal image quality loss is finally achieved. The model is optimized through the loss function. The loss functions in the heterogeneous image translation network are respectively: The loss is composed of the generation adversarial loss and SSIM loss. The mathematical expression of the loss function satisfies: ,in, and Represents the coefficients of different loss functions. The heterogeneous image translation network will gradually reduce during multiple iterations of optimization. , and Through multiple iterations of training, the mapping effect of heterogeneous images is continuously optimized, and finally an image translation model with stable parameters, minimal image quality loss and best effect is output, which is used as the final preset heterogeneous image translation network.

[0053] In some embodiments, Figure 4 As shown, the detection process of the twin attention enhanced change detection network includes:

[0054] Receive multiple dual-temporal remote sensing images, for example, , ,in Represents the image source domain.

[0055] The dual-temporal remote sensing images are input into a twin network structure with dual-branch weight sharing for feature encoding;

[0056] A convolutional block attention mechanism is embedded in the multi-scale feature layer. The convolutional block attention mechanism includes spatial attention and channel attention, which can significantly highlight the changed information and suppress the unchanged information.

[0057] A deconvolution method similar to the UNet++ structure is used to restore the image size, and a dense skip connection structure is introduced between the encoder and the decoder, which can significantly enhance the model's ability to extract ground details and high-level features, narrow the semantic gap between shallow features and deep features, and make the binary prediction generated by the decoder have higher detection accuracy.

[0058] Figure 5 The following figure shows the training process of the twin attention-enhanced change detection network, which includes the following steps:

[0059] Receiving multiple dual-temporal remote sensing images is called , ,in Represents the image source domain.

[0060] The input image is a remote sensing image processed by the heterogeneous image translation network. Specifically, the SAR image is unified into the same feature space as the optical image with the help of the generative adversarial network, and it is sent together with the original optical image as input data to the subsequent change detection network for training.

[0061] The data is sent to the twin attention-enhanced change detection network for feature extraction, which adopts an encoding-before-decoding structure.

[0062] A loss function is set, and when the accuracy performance of the change detection network reaches the optimum under the training constraints of the loss function, the training is stopped, and the change detection network is used as the preset optimal change detection network.

[0063] In some embodiments, the twin attention enhanced change detection network is trained using a hybrid loss function, and the specific loss function satisfies:

[0064]

[0065] in, represents the weighted binary cross entropy loss function and Dice loss. In the specific example: the weighted binary cross entropy loss function is suitable for the binary classification change detection problem, which can effectively correct the problems caused by sample imbalance. It specifically meets the following requirements: .in, Represents the number of pixels in the entire image. It enables the network to increase its attention to positive samples and reduce the weight of negative samples during training. Representative Sample pixels, A value of 0 indicates no change. A value of 1 indicates a change. Used to express the probability of a change occurring.

[0066] The dice loss improves the model's classification performance by calculating the similarity between the model's predicted samples and the real samples. Its mathematical expression satisfies: , where represents the predicted probability value of the pixel judged as a change, and Represents the true value of the changed pixel.

[0067] During the training process, the loss is optimized and the results are output. The output results are used to calculate the probability of each pixel using the activation function Sigmoid. The category with a higher probability value in the binary classification is used as the final prediction result of the model, and a complete binary change detection result is generated. The specific Sigmoid function formula satisfies: .

[0068] Different from the previous unsupervised heterogeneous remote sensing image change detection method, the optical and SAR data change detection method based on twin attention enhancement in the embodiment of the present application adopts a combination of "unsupervised-supervised" methods. In the unsupervised mode, with the help of the heterogeneous image translation network designed by the conditional generative adversarial network, the heterogeneous optical images and SAR images can be mapped to the same feature space, solving the problem that heterogeneous remote sensing images cannot be systematically compared. The present application further accesses the supervised twin attention enhanced change detection network for feature extraction. By introducing the multi-scale convolutional block attention mechanism, effective feature aggregation can be performed from both spatial and channel dimensions, the feature expression of heterogeneous data can be enhanced, and the local detail information of the target object at different scale layers can be better characterized, effectively improving the accuracy and robustness of change detection of heterogeneous data under various scales of objects.

[0069] The present application also proposes an optical and SAR image change detection device based on twin attention enhancement, such as Figure 6 As shown, including:

[0070] A receiving module, used for receiving the front-phase optical image and the back-phase SAR image;

[0071] A translation module, for inputting the processed pre-phase optical image and post-phase SAR image into a heterogeneous image translation network to output a post-phase optical image having a unified feature map with the pre-phase optical image;

[0072] A training module for inputting the optical images of the front and back phases of the unified feature map into the twin attention-enhanced change detection network for feature extraction;

[0073] The output module is used to take the category with a higher binary classification probability value in the corresponding pixel in the twin attention enhanced change detection network as the final prediction result of the pixel to generate a complete binary change detection result.

[0074] It should be noted that in the various embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0075] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0077] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for optical and SAR image change detection based on twin attention enhancement, characterized in that: include: Receive the pre-phase optical image and post-phase SAR image; Inputting the processed pre-phase optical image and post-phase SAR image into a heterogeneous image translation network to output a post-phase optical image having a unified feature map with the pre-phase optical image; The optical images of the front and back phases of the unified feature map are input into the twin attention-enhanced change detection network for feature extraction; The category with a higher binary classification probability value for the corresponding pixel in the twin attention-enhanced change detection network is taken as the final prediction result for the pixel to generate a complete binary change detection result.

2. The optical and SAR image change detection method based on twin attention enhancement as claimed in claim 1, characterized in that: After receiving the front-phase optical image and the back-phase SAR image, the method further includes: preprocessing the optical image and the SAR image, wherein the preprocessing includes one or more of the following: radiation calibration, atmospheric correction, geometric correction, and optical and radar image registration.

3. The optical and SAR image change detection method based on twin attention enhancement as claimed in claim 1, characterized in that: The heterogeneous image translation network includes a discriminant network and a generative network; Inputting the processed pre-phase optical image and post-phase SAR image into the heterogeneous image translation network includes: In the generation network, the SAR image of the later time phase is generated into an optical image corresponding to the previous time phase; The discriminant network is used to guide the generative network to output images that meet specific goals.

4. The optical and SAR image change detection method based on twin attention enhancement as claimed in claim 3, characterized in that: The optical image corresponding to the previous phase is generated from the later phase SAR image, including: Extracting key features of the post-phase SAR image based on the generation network; The extracted key features are used to generate an optical image that is consistent with the previous phase optical image in terms of detail texture and image structure.

5. The optical and SAR image change detection method based on twin attention enhancement as claimed in claim 4, characterized in that: Using the discriminant network, guiding the generation network to output images that meet specific goals includes: In the discriminant network, multi-layer convolution is used to extract features from the input image; Map the extracted features to the fully connected layer to form a feature vector; The Sigmoid activation function is connected to convert the output value into a probability value between 0 and 1, so as to use the probability value to reflect the matching degree between the image judged as true and the auxiliary image. The closer the value is to 1, the closer the generated image is to the real image.

6. The optical and SAR image change detection method based on twin attention enhancement as claimed in claim 5, characterized in that: It also includes configuring a loss function for the heterogeneous image translation model, wherein the loss function includes Loss, adversarial loss and SSIM loss satisfy: in, is the classic loss function in image restoration, To combat losses, To introduce the loss function of ground object texture similarity, and Represents the coefficients of different loss functions; In the SSIM loss, a sliding window of a specified size is used to express the SSIM loss of the local window as follows: in, and Represents the sliding window In Image and image; All windows Take the average to obtain the SSIM value in the whole image range. The SSIM loss of the whole image is expressed as: .

7. The optical and SAR image change detection method based on twin attention enhancement as claimed in claim 6, characterized in that: The method further includes training the heterogeneous image translation model using the following steps: Receive remote sensing images from different sources such as optical images and SAR images; Constructing the optical image and SAR image data sets respectively, and dividing them into a training set and a test set; The training data set is sent to the conditional generation adversarial network (cGAN) for training, and the loss function is used to optimize the model during the training process; After multiple iterations of optimization, a heterogeneous image translation model with stable output parameters and minimal image quality loss is achieved.

8. The optical and SAR image change detection method based on twin attention enhancement as claimed in claim 6, characterized in that: The detection process of the twin attention enhanced change detection network includes: Receive multiple dual-temporal remote sensing images; The dual-temporal remote sensing images are input into a twin network structure with dual-branch weight sharing for feature encoding; Embedding a convolutional block attention mechanism in a multi-scale feature layer, wherein the convolutional block attention mechanism includes spatial attention and channel attention; A deconvolution method similar to the UNet++ structure is used to restore the image, and a dense skip connection structure is introduced between the encoder and the decoder.

9. The optical and SAR image change detection method based on twin attention enhancement as claimed in claim 8, characterized in that: The twin attention-enhanced change detection network is trained using the following loss function: in, represents the weighted binary cross entropy loss function and Dice loss; During the training process, the loss is optimized and the results are output. The output results are used to calculate the probability of each pixel using the activation function Sigmoid. The category with a higher probability value in the binary classification is used as the final prediction result of the model, and a complete binary change detection result is generated.

10. Optical and SAR image change detection device based on twin attention enhancement, characterized in that: include: A receiving module, used for receiving the front-phase optical image and the back-phase SAR image; A translation module, for inputting the processed pre-phase optical image and post-phase SAR image into a heterogeneous image translation network to output a post-phase optical image having a unified feature map with the pre-phase optical image; A training module for inputting the optical images of the front and back phases of the unified feature map into the twin attention-enhanced change detection network for feature extraction; The output module is used to take the category with a higher binary classification probability value in the corresponding pixel in the twin attention enhanced change detection network as the final prediction result of the pixel to generate a complete binary change detection result.

Images