Remote sensing change detection method based on edge enhancement cross-attention and multidimensional loss

Abstract

This invention discloses a remote sensing change detection method based on edge-enhanced cross-attention and multidimensional loss. The method includes: inputting dual-temporal images into a trained Siamese backbone network to extract multi-scale backbone features from the dual-temporal images; inputting the multi-scale backbone features into a trained unsharpened mask cross-attention fusion model to extract spatial and semantic information from the dual-temporal images through cross-attention operations, and extracting edge information from the dual-temporal images through sharpening convolution kernels to obtain multi-level differential features at different scales; inputting the differential features into a trained multi-scale hierarchical connection decoder to obtain preliminary multi-scale prediction results for the changed region; and performing weighted fusion of the preliminary multi-scale prediction results to obtain the change prediction result. This invention can fully utilize the multi-scale information of the Siamese backbone network and improve the ability to extract edge information from dual-temporal images, thereby improving the accuracy of change detection.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image change detection technology, specifically relating to a remote sensing change detection method and device, electronic device, and storage medium based on edge enhancement cross attention and multidimensional loss. Background Technology

[0002] The classic paradigm for change detection tasks is to use a twin backbone network to extract dual-temporal features, and then perform binary segmentation of the changed and unchanged regions after obtaining the difference map by the difference module.

[0003] One popular change detection method is to fuse differential feature pyramids and attention mechanisms to obtain a D-Unet-based change detection model, which is then used for change detection. Another popular change detection method is to construct a remote sensing image building change detection network based on multi-scale attention for building change detection in remote sensing images.

[0004] Although current change detection methods have improved detection accuracy by adding multi-scale feature extraction and attention modules to traditional methods, popular change detection methods often only output a single prediction result without making full use of the multi-scale information of the backbone network, and have weak edge detection capabilities, resulting in low detection accuracy. Summary of the Invention

[0005] This invention provides a remote sensing change detection method based on edge enhancement cross-attention and multidimensional loss, which can solve the problem that the accuracy of current change detection methods has not been effectively improved.

[0006] In a first aspect, embodiments of the present invention provide a remote sensing change detection method based on edge-enhanced cross-attention and multidimensional loss, the method comprising:

[0007] The dual-temporal images are input into the trained twin backbone network, and multi-scale backbone features of the dual-temporal images are extracted. The dual-temporal images include a previous temporal image and a subsequent temporal image with the same shooting scene but different shooting times. The multi-scale backbone features include multi-level backbone features with different dimensions.

[0008] Multi-scale backbone features are input into a trained unsharpened mask cross-attention fusion model to extract multi-dimensional spatial and semantic information of dual-temporal images through cross-attention operations, and multi-dimensional edge information of dual-temporal images is extracted through sharpening convolution kernels to obtain multi-level difference features at different scales.

[0009] The differential features are input into the trained multi-scale hierarchical connected decoder, and each level of differential features is upsampled and / or downsampled by different factors to obtain the multi-scale preliminary prediction results of the change region.

[0010] The change prediction results are obtained by weighted fusion of the preliminary multi-scale prediction results.

[0011] Secondly, embodiments of the present invention provide a remote sensing change detection device based on edge-enhanced cross-attention and multidimensional loss, the device comprising:

[0012] The multi-scale backbone feature extraction unit includes a trained twin backbone network. The multi-scale backbone feature extraction unit is used to: input dual-temporal images into the trained twin backbone network, and extract multi-scale backbone features from the dual-temporal images. The dual-temporal images include a previous temporal image and a subsequent temporal image with the same shooting scene but different shooting times. The multi-scale backbone features include multi-level backbone features with different dimensions.

[0013] The bidirectional guided edge enhancement unit includes a trained lightweight unsharpened mask cross-attention fusion model. The bidirectional guided edge enhancement unit is used to: input multi-scale backbone features into the trained unsharpened mask cross-attention fusion model to extract multi-dimensional spatial and semantic information of dual-temporal images through cross-attention operations, and extract multi-dimensional edge information of dual-temporal images through sharpening convolution kernels to obtain multi-level difference features of different scales.

[0014] The preliminary prediction unit includes a trained multi-scale hierarchical connected decoder. The preliminary prediction unit is used to: input the differential features into the trained multi-scale hierarchical connected decoder, so as to perform upsampling and / or downsampling processing on each level of differential features by different multiples, so as to obtain the multi-scale preliminary prediction results of the change region.

[0015] The integrated prediction unit is used to weight and fuse the preliminary multi-scale prediction results to obtain the change prediction results.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the memory is used to store a computer program; the processor can be used to execute a calculator program (instructions) stored in the memory to implement the method of the first aspect described above.

[0017] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed, can implement the method described in the first aspect above.

[0018] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0019] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: According to the method provided by the present invention, by utilizing multi-scale backbone features to obtain multiple preliminary prediction results of different scales, and then combining these prediction results to obtain the final change prediction result, the multi-scale information extracted by the backbone network can be fully utilized; the USM cross-attention fusion model based on sharpened convolution kernel can enhance the ability to extract edge information, thereby improving the accuracy of change detection. Attached Figure Description

[0020] Figure 1 A flowchart illustrating the implementation of a remote sensing change detection method based on edge enhancement cross-attention and multidimensional loss, provided in an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of a dual-temporal image provided in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of a scenario for extracting differential features according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram illustrating a scenario for determining preliminary multi-scale prediction results based on differential features, provided by an embodiment of the present invention.

[0024] Figure 5 A schematic diagram of a binary mask image provided in an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram illustrating the implementation process of a training method for a twin backbone network, a USM cross-attention fusion model, and a multi-scale hierarchical connection decoder provided in an embodiment of the present invention.

[0026] Figure 7 A schematic diagram of the structure of a remote sensing change detection device based on edge enhancement cross attention and multidimensional loss provided in an embodiment of the present invention;

[0027] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0029] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0030] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0031] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0032] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0034] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0035] The remote sensing change detection method based on edge enhancement cross attention and multidimensional loss provided in this embodiment of the invention can be applied to electronic devices such as mobile terminals, personal laptops, and supercomputers. This embodiment of the invention does not impose any restrictions on the specific type of electronic device.

[0036] Figure 1The diagram illustrates a flowchart of a remote sensing change detection method based on edge enhancement cross-attention and multidimensional loss, provided by an embodiment of the present invention. As an example and not a limitation, this change detection method can be applied to the aforementioned electronic device. The method may include steps S101-S104, which are described below.

[0037] S101: Input the dual-temporal images into the trained twin backbone network and extract the multi-scale backbone features of the dual-temporal images.

[0038] For example, dual-phase images may include a first phase image X1 of the same scene but captured at different times (see [link]). Figure 2 (a) and post-phase image X2 (see [reference]). Figure 2 (b) in the middle.

[0039] It should be understood that the relevant parameters of the dual-phase images mentioned below include the relevant parameters of the preceding phase image and the relevant parameters of the following phase image.

[0040] In one example, multi-scale backbone features can include multi-level backbone features with different dimensions.

[0041] For example, see Figure 3 The twin backbone network can be a Swing Transformer model. The multi-scale backbone features Fi can include four levels of backbone features with different dimensions; that is, Fi = [Fi1, Fi2, Fi3, Fi4], where i = 1, 2, i = 1 represents the relevant parameters of the previous time-phase image, and i = 2 represents the relevant parameters of the subsequent time-phase image.

[0042] Specifically, the dimensions of each feature map can be: Where H and W are the height and width of the dual-phase image, respectively, and R represents a real number.

[0043] S102 inputs multi-scale backbone features into a trained unsharpened mask cross-attention fusion model to extract multi-dimensional spatial and semantic information of dual-temporal images through cross-attention operations, and extracts multi-dimensional edge information of dual-temporal images through sharpening convolution kernels to obtain multi-level difference features of different scales.

[0044] In some embodiments, the Unsharpen Mask (USM) cross-attention fusion model may include a cross-attention module and a USM convolution module.

[0045] For example, the cross-attention module can be used to perform cross-attention operations on multi-scale backbone features to aggregate spatial and semantic information of multiple dimensions of dual-temporal images, thereby obtaining multi-level aggregated features of dual-temporal images.

[0046] For example, the USM convolution module can extract edge information from the multi-level aggregated features of the dual-temporal image based on the sharpened convolution kernel to obtain a multi-level sharpened convolutional map of the dual-temporal image.

[0047] In one possible implementation, the cross-attention module can perform cross-attention operations based on the nth-level comprehensive backbone features of the bi-temporal images to extract spatial and semantic information and obtain the nth-level aggregated features of the bi-temporal images.

[0048] For example, n is a positive integer less than or equal to Nmax, where Nmax is the total number of levels of the backbone features output by the Siamese backbone network. The Nmax-th level comprehensive backbone feature is the Nmax-th level backbone feature; starting from Nmax-1, the nth level comprehensive backbone feature is obtained by adding the upsampled (n+1)th level sharpened feature map to the nth level backbone feature.

[0049] For example, see Figure 3 When Nmax = 4, the fourth-level comprehensive backbone feature is the fourth-level backbone feature Fi4. The third-level comprehensive backbone feature is obtained by adding the fourth-level sharpened feature map Ti4′ and the third-level backbone feature Fi3.

[0050] In one example, the cross-attention module first convolves the nth-level comprehensive backbone features of the two-temporal images to obtain the nth-level cross-attention vector. Then, it performs dimensionality transformation on the nth-level cross-attention vector according to the height and width of the two-temporal images to obtain the nth-level first cross-attention vector and the nth-level second cross-attention vector. Finally, it performs a bidirectional cross-attention operation based on the nth-level first cross-attention vector and the nth-level second cross-attention vector to obtain the nth-level aggregated feature Tin.

[0051] Performing cross-attention operations on the backbone features of each dimension at their original scale is computationally intensive. Therefore, current change detection methods discard low-dimensional backbone features, utilizing only the highest-dimensional backbone features for cross-attention operations. This sacrifices the use of low-dimensional spatial and semantic information from the bi-temporal images, reducing change detection accuracy. This invention, however, transforms the cross-attention vectors obtained from the backbone features according to the height and width of the bi-temporal images, performing cross-attention operations on vectors of different dimensions separately. This reduces computational cost, allows for the utilization of low-dimensional information, and improves change detection accuracy.

[0052] For example, the nth-level cross-attention vector may include the nth-level query vector of the two-temporal images. nth level key vector And the nth level value vector Vin.

[0053] For example, the dimension of the first cross-attention vector at level n is the height of the two-phase image, and the dimension of the second cross-attention vector at level n is the width of the two-phase image.

[0054] Specifically, the first cross-attention vector at level n may include: the first query vector at level n. Level n, first key value vector The first value vector at level n is ViH. The second cross-attention vector at level n may include: the second query vector at level n. Level n, second key value vector The second value vector of level n, ViW.

[0055] Specifically, the bidirectional cross-attention operation may include: dot-multiplying the nth-level second query vector of the previous temporal image respectively. and the second key value vector of the nth level of the post-temporal image The first query vector of the nth level of the previous phase image and the first key value vector of the nth level of the post-temporal image Obtain the pre-nth phase cross-attention map The second query vector of the nth level of the phase image after dot product The second key value vector of the nth level of the previous phase image The first query vector of the nth level of the phase image after dot product The first key value vector of the nth level of the previous phase image After obtaining the phase cross-attention map of level n Then, the pre-nth phase cross-attention map The second value vector of the nth level of the subsequent phase image is respectively compared with the second value vector of the subsequent phase image. The first value vector of the nth level Dot product is performed, and the result is concatenated with the residual of the nth-level backbone feature F1n of the previous temporal image to obtain the aggregated feature T1n of the previous temporal image; the nth-level cross-attention map of the subsequent temporal image is then used. Dot multiply V1H by the nth level second value vector V1W and the nth level first value vector of the previous time-phase image, respectively, and then multiply the result by the nth level backbone feature of the subsequent time-phase image. Residual connections yield aggregated features of the post-temporal image.

[0056] Specifically, the bidirectional cross-attention operation can be represented as:

[0057]

[0058]

[0059] Add indicates a residual join operation;

[0060] in:

[0061]

[0062] In one possible implementation, see Figure 3 The USM convolution module can extract edge information from the nth-level aggregated features of the bi-temporal image based on the sharpening convolution kernel to obtain the nth-level sharpening feature map of the bi-temporal image.

[0063] In one example, the sharpening convolution kernel can be constructed based on a Gaussian filter, which can satisfy the following formula:

[0064]

[0065] Where K is the sharpening convolution kernel.

[0066] For example, the nth-level sharpening feature map of a dual-temporal image can satisfy the following formula:

[0067]

[0068] Where Tin' is the nth level sharpening feature map of the previous time-phase image / the next time-phase image, Tin is the nth level aggregated feature of the previous time-phase image / the next time-phase image, Conv(·) represents the convolution operation, and σ is a constant.

[0069] Optionally, σ can take the value of 0.6.

[0070] In one possible implementation, see Figure 3 The nth level difference feature can be obtained by performing a difference operation on the nth level sharpened feature map of the previous phase image and the nth level sharpened feature map of the subsequent phase image.

[0071] In one example, the nth-level difference feature can satisfy the following formula:

[0072]

[0073] Where Dn is the nth level difference feature, and abs represents the difference operation.

[0074] Optionally, see Figure 4 When Nmax = 4, the first-level difference feature D1 can be a 4-fold downsampled feature, and its dimension can be... The second-level difference feature D2 can be an 8x downsampled feature, and its dimension can be... The third-level difference feature D3 can be a 16-downsampled feature, and its dimension can be... The fourth-level difference feature D4 can be a 32x downsampled feature, and its dimension can be...

[0075]

[0076] S103, input the differential features into the trained multi-scale hierarchical connected decoder, so that each level of differential features is upsampled and / or downsampled by different multiples to obtain the multi-scale preliminary prediction results of the change region.

[0077] In one possible implementation, if the USM cross-attention model outputs four levels of differential features, the initial multi-scale prediction results can include first, second, and third prediction results at different scales. The three-level predictions at different scales provide a more comprehensive representation.

[0078] For example, see Figure 5 The preliminary multi-scale prediction results can be represented as binary masks at different scales. Pixels in areas where changes have occurred have a value of 1 (white), while pixels in areas where no changes have a value of 0 (black).

[0079] In one example, see Figure 4 In (a), the multi-scale hierarchical connected decoder can be used to: add the 4th-level differential feature D4 (after 2x upsampling) and the 3rd-level differential feature D3 to obtain a 16x downsampling fused feature. The first 8-fold fusion feature is obtained by adding D3 (upsampled by 2x), the second-level difference feature D2, and the first-level difference feature D1 (downsampled by 2x). Will Compared with 2x upsampling The sums are used to obtain the first prediction result Y2.

[0080] In one example, see Figure 4 In (b), the multi-scale hierarchical connected decoder can also be used to: add the 2x upsampled D3 and D2 to obtain the second 8x fused feature. Will D2, and after 2x upsampling Adding them together yields the third 8-fold fusion feature. Adding D1 and D2 (after a 2x upsampling) yields the first 4x fusion feature. Will After 2x upsampling D1 addition yields the second 4x fusion feature Will D1、 After 2x upsampling The sums are used to obtain the second prediction result Y1.

[0081] In one example, the multi-scale hierarchical connection decoder can also be used to: splice D1, Y1 is obtained as a 4x feature after dimensional transformation; then, the 4x feature after dimensional transformation is upsampled by 4x to obtain the third prediction result Y0 with the same scale as the dual-temporal image.

[0082] S104, weighted fusion of the preliminary multi-scale prediction results to obtain the change prediction results.

[0083] In one example, the first, second, and third prediction results can be assigned different weights and then summed to obtain the change prediction result.

[0084] For example, the change prediction result can satisfy the following formula:

[0085] Y=1.0*Y0+0.3*Up4(Y1)+0.2*Up8(Y2)

[0086] Where Y represents the change prediction result, Y0 represents the third prediction result, Y1 represents the second prediction result, Y2 represents the first prediction result, Up4 represents a 4x upsampling operation, and Up8 represents an 8x upsampling operation.

[0087] According to the method provided by this invention, multiple preliminary prediction results at different scales are obtained by utilizing multi-scale backbone features, and then these prediction results are combined to obtain the final change prediction result, which can make full use of the multi-scale information extracted by the backbone network. The sharpening convolution kernel of the USM cross-attention fusion module is used to extract multi-dimensional edge information from the dual-temporal images, which enhances the ability to extract edge information and thus improves the accuracy of change detection. Furthermore, by performing dimensional transformation on the cross-attention vector according to the height and width of the dual-temporal images respectively, and performing cross-self-attention operations on the cross-attention vectors of different dimensions respectively, the computational load can be reduced while utilizing the multi-scale information extracted by the backbone network.

[0088] Figure 6 The diagram illustrates the implementation flow of a training method for a Siamese backbone network, a USM cross-attention fusion model, and a multi-scale hierarchical connection decoder provided in an embodiment of the present invention. As an example and not a limitation, this training method can also be applied to the aforementioned electronic device. The training method may include steps S601-S605, which are described below.

[0089] S601, perform data preprocessing on the dataset to obtain sample biphase images.

[0090] In one possible implementation, the training set and validation set can be divided according to a publicly available dataset or user-collected registered pre- and post-temporal images and their corresponding binary label images, based on a publicly available data list or a certain proportion. Then, the samples in the training set and validation set are preprocessed to obtain sample bi-temporal images.

[0091] For example, publicly available datasets such as the binary remote sensing building change detection dataset (LEVIR-CD+), the remote sensing change detection dataset based on edge enhancement cross-attention and multidimensional loss (Onera Satellite ChangeDetection Dataset, OSCD), S1GFloods, and the remote sensing farmland semantic and change annotation dataset (Hi-CAN) can be used as sample bi-temporal images in the training and validation sets.

[0092] In one possible implementation, the format of the sample dual-temporal images can be unified to uint8 format, and then the corresponding binary label images can be downsampled by 4 times and 8 times respectively to obtain the original scale true binary mask GT0, the 4 times feature scale true binary mask GT1, and the 8 times feature scale true binary mask GT2, which are the same scale as the prediction results of the first sample, the second sample, and the third sample, respectively; thus completing the data preprocessing of the sample dual-temporal images.

[0093] In one example, if a multispectral OSCD dataset is used, the images in this dataset have 13 bands, and the original data is in uint16 format. Therefore, channels [0,1,2] can be selected according to the required spectral channel parameters, and then the images are uniformly sized to [128,128,3] using a fixed-step sliding window cropping method. The processed images are then subjected to max-min normalization to obtain sample dual-temporal images in uint8 format.

[0094] In one example, if the binary label map is a binary mask map with a label of 0 (unchanged area) or 255 (changed area), then the pixel values ​​of all pixels with a pixel value of 255 can be converted to 1 first; then bilinear interpolation is used to perform 4x and 8x downsampling on the binary label map after pixel value transformation to obtain real binary mask maps [GT0,GT1,GT2] at different scales.

[0095] S602 inputs the sample dual-temporal images from the training set into the twin backbone network to extract the sample multi-scale backbone features from the sample dual-temporal images.

[0096] S603 inputs the multi-scale backbone features of the samples into the unsharpened mask cross-attention fusion model to obtain multi-level sample difference features of different scales.

[0097] S604 inputs the sample difference features to the multi-scale hierarchical connection decoder, so as to perform upsampling and / or downsampling processing on each level of sample difference features by different multiples, and obtain the preliminary prediction results of multi-scale samples in the changing region.

[0098] For example, the preliminary prediction results of multi-scale samples may include prediction results of the first sample, prediction results of the second sample, and prediction results of the third sample at different scales.

[0099] For example, the twin backbone network, USM cross-attention fusion model, and multi-scale hierarchical connection decoder in steps S602-S604 of the training method have the same functions as those described in steps S101-S103 of the change detection method above. The only difference is that the training method processes the sample dual-temporal images, while the change detection method processes the dual-temporal images to be detected. For details, please refer to steps S101-S103 of the change detection method, which will not be repeated here.

[0100] S605, compare the real binary mask image and the sample prediction results at each scale to obtain the total loss. Train the Siamese backbone network, the USM cross-attention fusion model and the multi-scale hierarchical connection decoder based on the total loss to obtain the trained Siamese backbone network, the trained USM cross-attention fusion model and the trained multi-scale hierarchical connection decoder.

[0101] In one possible implementation, a maximum number of training iterations can be set. During each training iteration, the bi-temporal images of the samples in the training set are partitioned. The bi-temporal images of the samples in each partition, along with their corresponding ground truth binary masks, are sequentially input into the Siamese backbone network, the USM cross-attention fusion model, and the multi-scale hierarchical connected decoder to obtain preliminary multi-scale sample prediction results corresponding to the bi-temporal images. Then, the preliminary multi-scale sample prediction results are compared with the ground truth binary masks to obtain the total loss for this round of training. Based on the total loss, the AdamW optimizer is used for gradient updates, and the learning rate is updated using a Poly decay strategy. After the update, the next training iteration continues until the maximum number of training iterations is reached.

[0102] For example, the maximum number of training iterations can be 50, the initial learning rate of the Siamese backbone network can be 6e-5, the initial learning rate of the USM cross-attention fusion model can be 6e-4, and the initial learning rate of the multi-scale hierarchical connection decoder can be 6e-4.

[0103] For example, after each training iteration, evaluation can be performed based on the biphase graph of samples in the validation set. Evaluation metrics can include Unchange IoU, Changed IoU, the mean of Unchange IoU and Changed IoU (mIoU), Unchanged F1 score, Changed F1 score, the mean of Unchanged F1 score and Changed F1 score (mF1Score), and Overall Accuracy (OA). Then, the training weights used in the training epoch with the best mIoU value across all training epochs are determined as the final weights of the model, resulting in the trained Siamese backbone network, the trained USM cross-attention fusion model, and the trained multi-scale hierarchical connection decoder.

[0104] In one example, the total loss can satisfy the following formula:

[0105] L = 1.0 * LCE ( Y0′,GT0 ) +0.3*LCE ( Y1′,GT1)+0.2*LCE ( Y2′,GT2 )

[0106] Where L is the total loss, LCE represents the cross-entropy loss, Y0′, Y1′, and Y2′ are the prediction results of the third, second, and first samples of the sample biphase images, respectively, and GT0, GT1, and GT2 are the true binary masks of the sample biphase images with the same scale as Y0′, Y1′, and Y2′, respectively.

[0107] According to the training method provided by this invention, determining the total loss based on the preliminary prediction results of multi-scale samples can improve label utilization and enhance the prediction performance of the multi-scale hierarchical connected decoder. The bidirectional cross-attention operation can reduce the computational cost of performing cross-attention operations on cross-attention vectors at each scale, thus accelerating the model training process.

[0108] Figure 7 The diagram shown illustrates the structure of a remote sensing change detection device based on edge enhancement cross-attention and multidimensional loss, as provided in an embodiment of the present invention. As an example and not a limitation, the device 700 may include a multi-scale backbone feature extraction unit 710, a bidirectional guided edge enhancement unit 720, a preliminary prediction unit 730, and a comprehensive prediction unit 740.

[0109] The multi-scale backbone feature extraction unit 710 includes a trained twin backbone network. The multi-scale backbone feature extraction unit 710 is used to: input dual-temporal images into the trained twin backbone network and extract multi-scale backbone features from the dual-temporal images. The dual-temporal images include a front-temporal image and a back-temporal image with the same shooting scene but different shooting times. The multi-scale backbone features include multi-level backbone features with different dimensions.

[0110] The bidirectional guided edge enhancement unit 720 includes trained lightweight unsharpened mask fusion features. The bidirectional guided edge enhancement unit 720 is used to: input multi-scale backbone features into the trained unsharpened mask cross-attention fusion model to extract multi-dimensional spatial and semantic information of dual-temporal images through cross-attention operation, and extract multi-dimensional edge information of dual-temporal images through sharpening convolution kernel to obtain multi-level difference features of different scales.

[0111] The preliminary prediction unit 730 includes a trained multi-scale hierarchical connected decoder. The preliminary prediction unit 730 is used to: input the differential features into the trained multi-scale hierarchical connected decoder, so as to perform upsampling and / or downsampling processing on each level of differential features by different multiples, so as to obtain the multi-scale preliminary prediction results of the change region.

[0112] The integrated prediction unit 740 is used to: weight and fuse the preliminary multi-scale prediction results to obtain the change prediction results.

[0113] According to the apparatus provided by the present invention, multiple preliminary prediction results at different scales are obtained by utilizing multi-scale backbone features, and then these prediction results are combined to obtain the final change prediction result; it can make full use of the multi-scale information extracted by the backbone network and improve the detection accuracy.

[0114] Figure 8 The diagram shown is a structural schematic of an electronic device provided in an embodiment of the present invention. Figure 8 The illustrated electronic device 800 may include: at least one processor 810 ( Figure 8 The diagram shows only one processor, a memory 820, and a computer program 830 stored in the memory 820 and executable on the at least one processor 810, wherein the processor 810 executes the computer program 830 to implement the steps in any of the above method embodiments.

[0115] The electronic device 800 may be a robot or other processing device capable of implementing the above methods. This embodiment of the invention does not impose any restrictions on the specific type of electronic device.

[0116] Those skilled in the art will understand that Figure 8This is merely an example of electronic device 800 and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or combine certain components, or use different components. For example, the electronic device 800 may also include input / output interfaces.

[0117] The processor 810 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASTCs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0118] In some embodiments, the memory 820 may be an internal storage unit, such as a hard disk or RAM. In other embodiments, the memory 820 may be an external storage device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital card (SD), or a flash card. Furthermore, the memory 820 may include both internal and external storage units. The memory 820 is used to store the operating system, applications, a boot loader, data, and other programs, such as the program code of the computer program. The memory 820 can also be used to temporarily store data that has been output or will be output.

[0119] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0120] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0121] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0122] This invention provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0123] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0124] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0125] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

Claims

1. A remote sensing change detection method based on edge enhancement cross-attention and multi-dimensional loss, characterized in that, include: The dual-temporal images are input into a trained twin backbone network, and multi-scale backbone features of the dual-temporal images are extracted. The dual-temporal images include a front-temporal image and a back-temporal image with the same shooting scene but different shooting times. The multi-scale backbone features include multi-level backbone features with different dimensions. The multi-scale backbone features are input into the trained unsharpened mask cross-attention fusion model to extract multi-dimensional spatial and semantic information of the dual-temporal images through cross-attention operations, and to extract multi-dimensional edge information of the dual-temporal images through sharpening convolution kernels, thereby obtaining multi-level difference features of different scales. The differential features are input into a trained multi-scale hierarchical connected decoder to upsample and / or downsample each level of the differential features by different factors, thereby obtaining a preliminary multi-scale prediction result for the changing region. The change prediction results are obtained by weighted fusion of the preliminary multi-scale prediction results; The unsharpened mask cross-attention fusion model includes a cross-attention module and an unsharpened mask convolution module. The cross-attention module is used to perform cross-attention operations on the multi-scale backbone features to achieve mutual guidance between the multi-scale backbone features of the previous time-phase image and the multi-scale backbone features of the subsequent time-phase image, and to aggregate the spatial and semantic information of the dual-time-phase images in multiple dimensions to obtain the multi-level aggregated features of the dual-time-phase images. Specifically, the cross-attention module is used for: The nth level comprehensive backbone feature of the dual-temporal image is convolved to obtain the nth level cross-attention vector of the dual-temporal image, where n is a positive integer less than or equal to Nmax, Nmax is the total number of levels of the backbone features output by the twin backbone network, the Nmax level comprehensive backbone feature is the Nmax level backbone feature, starting from Nmax-1, the nth level comprehensive backbone feature is obtained by adding the upsampled n+1 level sharpened feature map to the nth level backbone feature; The nth-level cross-attention vector is transformed according to the height and width of the dual-temporal image to obtain the nth-level first cross-attention vector and the nth-level second cross-attention vector of the dual-temporal image, wherein the dimension of the nth-level first cross-attention vector is the height of the dual-temporal image and the dimension of the nth-level second cross-attention vector is the width of the dual-temporal image. Based on the nth level first cross attention vector and the nth level second cross attention vector, a bidirectional cross attention operation is performed to obtain the nth level aggregated feature; The cross-attention vector includes a key vector, a value vector, and a query vector; the bidirectional cross-attention operation based on the nth-level first cross-attention vector and the nth-level second cross-attention vector to obtain the nth-level aggregated feature includes: respectively, the nth-level second query vector of the pre-time phase image and the second key-value vector of the post-time phase image, the nth-level first query vector of the pre-time phase image and the nth-level first key-value vector of the post-time phase image, to obtain an nth-level pre-time phase cross-attention graph ; respectively, the nth-level second query vector of the post-phase image and the nth-level second key-value vector of the pre-phase image, and the nth-level first query vector of the post-phase image and the nth-level first key-value vector of the pre-phase image, to obtain an nth-level post-phase cross-attention graph ; Will The nth level second value vector and the nth level first value vector of the subsequent time-phase image are respectively multiplied by the dot product, and the result of the dot product is connected with the nth level backbone feature residual of the previous time-phase image to obtain the nth level aggregate feature of the previous time-phase image; Will The nth-level aggregated feature of the subsequent phase image is obtained by multiplying the nth-level second value vector and the nth-level first value vector of the preceding phase image respectively, and then concatenating the result of the multiplication with the nth-level backbone feature residual of the subsequent phase image.

2. The method according to claim 1, characterized in that, The unsharpened mask convolution module is used to extract multi-dimensional edge information of the dual-temporal image based on the sharpening convolution kernel and the multi-level aggregation feature to obtain the multi-level sharpening feature map of the dual-temporal image. The nth level difference feature is obtained based on the difference between the nth level sharpening feature map of the previous temporal image and the subsequent temporal image.

3. The method according to claim 2, characterized in that, The nth-level sharpening feature map of the dual-temporal image satisfies the following formula: in, , Represents the previous phase image, Represents the post-phase image; This is the nth level sharpening feature map of the preceding / following temporal image. The nth-level aggregated feature of the previous phase image / later phase image This represents the convolution operation. For the sharpened convolution kernel, It is a constant.

4. The method according to claim 1, characterized in that, The unsharpened mask cross-attention fusion model outputs four levels of differential features. The multi-scale preliminary prediction results include first, second, and third prediction results at different scales. The multi-scale hierarchical connection decoder is used for: The fourth-level difference feature, after being upsampled by 2 times, is added to the third-level difference feature to obtain the fused feature after being downsampled by 16 times. The first 8-fold fusion feature is obtained by adding the third-level difference feature after 2x upsampling, the second-level difference feature, and the first-level difference feature after 2x downsampling. The first prediction result is obtained by adding the first 8x fused feature to the 16x downsampled fused feature after 2x upsampling; The second 8-fold fused feature is obtained by adding the third-level difference feature after 2x upsampling and the second-level difference feature. The second 8x fusion feature, the second-level difference feature, and the 16x downsampling fusion feature after 2x upsampling are added together to obtain the third 8x fusion feature; The first-level difference feature is added to the second-level difference feature after being upsampled by 2 times to obtain the first 4-fold fused feature; The second 4x fusion feature is obtained by adding the first 4x fusion feature, the second 8x fusion feature after 2x upsampling, and the first-level difference feature; The second 4x fusion feature, the first-level difference feature, the first 4x fusion feature, and the third 8x fusion feature after 2x upsampling are added together to obtain the second prediction result; By concatenating the first-level differential feature, the first 4x fusion feature, the second 4x fusion feature, and the second prediction result, a concatenated 4x feature is obtained. A convolution operation is performed on the spliced ​​4x features to transform the dimensions of the spliced ​​4x features, resulting in dimensionality-transformed 4x features; The third prediction result is obtained by upsampling the 4-fold features after dimensional transformation by 4 times.

5. The method according to claim 4, characterized in that, The twin backbone network, the unsharpened mask cross-attention fusion model, and the multi-scale hierarchical connection decoder are jointly trained based on a total loss, which satisfies the following formula: in, For the total loss, Represents cross-entropy loss, , , These are the prediction results for the third sample, the second sample, and the first sample of the dual-temporal image, respectively. , , For scale and , , The same true binary mask image of the sample dual-phase diagram.

6. The method according to claim 4, characterized in that, The predicted change results satisfy the following formula: in, The predicted change result, The third prediction result, The second prediction result, The first prediction result, This indicates a 4x upsampling operation. This indicates an 8x upsampling operation.

7. A remote sensing change detection device based on edge-enhanced cross-attention and multidimensional loss, characterized in that, include: A multi-scale backbone feature extraction unit, comprising a trained twin backbone network, is used to: input dual-temporal images into the trained twin backbone network and extract multi-scale backbone features from the dual-temporal images, wherein the dual-temporal images include a previous temporal image and a subsequent temporal image with the same shooting scene but different shooting times, and the multi-scale backbone features include multi-level backbone features with different dimensions; A bidirectional guided edge enhancement unit, comprising a trained unsharpened mask cross-attention fusion model, is used to: input the multi-scale backbone features into the trained unsharpened mask cross-attention fusion model to extract multi-dimensional spatial and semantic information of the dual-temporal images through cross-attention operations, and extract multi-dimensional edge information of the dual-temporal images through sharpening convolution kernels to obtain multi-level difference features at different scales; The preliminary prediction unit includes a trained multi-scale hierarchical connected decoder. The preliminary prediction unit is used to: input the differential features into the trained multi-scale hierarchical connected decoder, so as to perform upsampling and / or downsampling processing on each level of the differential features by different multiples, so as to obtain the multi-scale preliminary prediction results of the change region. A comprehensive prediction unit is configured to: perform weighted fusion on the multi-scale preliminary prediction results to obtain change prediction results; The unsharpened mask cross-attention fusion model includes a cross-attention module and an unsharpened mask convolution module. The cross-attention module is used to perform cross-attention operations on the multi-scale backbone features to achieve mutual guidance between the multi-scale backbone features of the previous time-phase image and the multi-scale backbone features of the subsequent time-phase image, and to aggregate the spatial and semantic information of the dual-time-phase images in multiple dimensions to obtain the multi-level aggregated features of the dual-time-phase images. Specifically, the cross-attention module is used for: The nth level comprehensive backbone feature of the dual-temporal image is convolved to obtain the nth level cross-attention vector of the dual-temporal image, where n is a positive integer less than or equal to Nmax, Nmax is the total number of levels of the backbone features output by the twin backbone network, the Nmax level comprehensive backbone feature is the Nmax level backbone feature, starting from Nmax-1, the nth level comprehensive backbone feature is obtained by adding the upsampled n+1 level sharpened feature map to the nth level backbone feature; The nth-level cross-attention vector is transformed according to the height and width of the dual-temporal image to obtain the nth-level first cross-attention vector and the nth-level second cross-attention vector of the dual-temporal image, wherein the dimension of the nth-level first cross-attention vector is the height of the dual-temporal image and the dimension of the nth-level second cross-attention vector is the width of the dual-temporal image. Based on the nth level first cross attention vector and the nth level second cross attention vector, a bidirectional cross attention operation is performed to obtain the nth level aggregated feature; The cross-attention vector includes a key vector, a value vector, and a query vector; the bidirectional cross-attention operation based on the nth-level first cross-attention vector and the nth-level second cross-attention vector to obtain the nth-level aggregated feature includes: By dot-multiplying the second query vector of the nth level of the previous time-phase image, the second key-value vector of the subsequent time-phase image, the first query vector of the nth level of the previous time-phase image, and the first key-value vector of the nth level of the subsequent time-phase image, the nth-level previous time-phase cross-attention map is obtained. ; By dot-multiplying the second query vector of the nth level of the later phase image with the second key-value vector of the nth level of the earlier phase image, and the first query vector of the nth level of the later phase image with the first key-value vector of the nth level of the earlier phase image, the nth-level later phase cross-attention map is obtained. ; Will The nth level second value vector and the nth level first value vector of the subsequent time-phase image are respectively multiplied by the dot product, and the result of the dot product is connected with the nth level backbone feature residual of the previous time-phase image to obtain the nth level aggregate feature of the previous time-phase image; Will The nth-level aggregated feature of the subsequent phase image is obtained by multiplying the nth-level second value vector and the nth-level first value vector of the preceding phase image respectively, and then concatenating the result of the multiplication with the nth-level backbone feature residual of the subsequent phase image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by an electronic device, it implements the method as described in any one of claims 1-6.

Images