Remote sensing change detection method based on edge enhanced cross attention and multi-dimensional loss

Abstract

The invention discloses a remote sensing change detection method based on edge enhanced cross attention and multi-dimensional loss. The method comprises the following steps: inputting a dual-time-phase image into a trained twin backbone network, and extracting multi-scale backbone features of the dual-time-phase image; inputting the multi-scale backbone features into a trained non-sharpening mask cross attention fusion model to extract spatial information and semantic information of the dual-temporal image through cross attention operation, and extracting edge information of the dual-temporal image through a sharpening convolution kernel to obtain multi-level differential features with different scales; inputting the differential features into a trained multi-scale hierarchical connection decoder to obtain a multi-scale preliminary prediction result of the change area; and carrying out weighted fusion on the multi-scale preliminary prediction result to obtain a change prediction result. According to the invention, the multi-scale information of the twin backbone network can be fully utilized, and the capability of extracting the edge information of the dual-time-phase image can be improved, so that the change detection accuracy can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image change detection, and specifically relates to a remote sensing change detection method and device based on edge enhancement cross attention and multi-dimensional loss, electronic equipment, and storage medium. Background Art

[0002] The classic paradigm for change detection tasks is to use a twin backbone network to extract dual-phase features, and the difference module obtains the difference map and performs binary segmentation operations on the changed and unchanged areas.

[0003] A popular change detection method is to perform change detection by fusing the differential feature pyramid and the attention mechanism to obtain a D-Unet-based change detection model. Another popular change detection method is to perform remote sensing image building change detection by constructing a remote sensing image building change detection network based on multi-scale attention.

[0004] Although current change detection methods improve detection accuracy by adding multi-scale feature extraction and attention modules to traditional methods; however, currently popular change detection methods often only output a single prediction result without fully utilizing the multi-scale information of the backbone network, and have weak edge detection capabilities, resulting in low detection accuracy. Summary of the invention

[0005] The embodiment of the present invention provides a remote sensing change detection method based on edge-enhanced cross-attention and multi-dimensional loss, which can solve the problem that the accuracy of current change detection methods has not been effectively improved.

[0006] In a first aspect, an embodiment of the present invention provides a remote sensing change detection method based on edge-enhanced cross attention and multi-dimensional loss, the method comprising:

[0007] Input the dual-phase image into the trained twin backbone network to extract the multi-scale backbone features of the dual-phase image, wherein the dual-phase image includes a front-phase image and a back-phase image with the same shooting scene but different shooting times, and the multi-scale backbone features include multi-level backbone features with different dimensions;

[0008] The multi-scale backbone features are input into the trained unsharp mask cross-attention fusion model to extract the multi-dimensional spatial information and semantic information of the dual-phase image through the cross-attention operation, and the multi-dimensional edge information of the dual-phase image is extracted through the sharpening convolution kernel to obtain multi-level differential features of different scales;

[0009] The differential features are input into the trained multi-scale hierarchical connection decoder to upsample and / or downsample the differential features at each level by different multiples to obtain a multi-scale preliminary prediction result for the changed area;

[0010] The multi-scale preliminary prediction results are weighted fused to obtain the change prediction results.

[0011] In a second aspect, an embodiment of the present invention provides a remote sensing change detection device based on edge-enhanced cross attention and multi-dimensional loss, the device comprising:

[0012] A multi-scale backbone feature extraction unit, the multi-scale backbone feature extraction unit includes a trained twin backbone network, and the multi-scale backbone feature extraction unit is used to: input a dual-phase image into the trained twin backbone network, and extract multi-scale backbone features of the dual-phase image, wherein the dual-phase image includes a front-phase image and a back-phase image with the same shooting scene but different shooting times, and the multi-scale backbone features include multi-level backbone features with different dimensions;

[0013] A bidirectionally guided edge enhancement unit, the bidirectionally guided edge enhancement unit includes a trained lightweight unsharp mask cross-attention fusion model, and the bidirectionally guided edge enhancement unit is used to: input the multi-scale backbone features into the trained unsharp mask cross-attention fusion model to extract multi-dimensional spatial information and semantic information of the bi-phase image through the cross-attention operation, and extract multi-dimensional edge information of the bi-phase image through the sharpening convolution kernel, so as to obtain multi-level differential features of different scales;

[0014] A preliminary prediction unit, the preliminary prediction unit includes a trained multi-scale hierarchical connection decoder, and the preliminary prediction unit is used to: input the differential features into the trained multi-scale hierarchical connection decoder, so as to upsample and / or downsample each level of differential features by different multiples, and obtain a multi-scale preliminary prediction result of the change area;

[0015] The comprehensive prediction unit is used to perform weighted fusion on the multi-scale preliminary prediction results to obtain the change prediction results.

[0016] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store computer programs; the processor can be used to execute the computer program (instructions) stored in the memory to implement the method of the first aspect above.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed, the method of the first aspect described above can be implemented.

[0018] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0019] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: according to the method provided by the present invention, by utilizing multi-scale backbone features to obtain preliminary prediction results of multiple scales, and then combining these prediction results to obtain the final change prediction result, the multi-scale information extracted by the backbone network can be fully utilized; the USM cross-attention fusion model can enhance the ability to extract edge information based on the sharpening convolution kernel, thereby improving the accuracy of change detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a remote sensing change detection method based on edge-enhanced cross-attention and multi-dimensional loss provided in an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of a dual-phase image provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a scenario for extracting differential features provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of a scenario for determining a multi-scale preliminary prediction result based on differential features provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of a binary mask image provided by an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of the implementation flow of a twin backbone network, a USM cross-attention fusion model, and a training method for a multi-scale hierarchical connection decoder provided in an embodiment of the present invention;

[0026] Figure 7 A schematic diagram of the structure of a remote sensing change detection device based on edge-enhanced cross attention and multi-dimensional loss provided by an embodiment of the present invention;

[0027] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0029] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0030] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0031] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0032] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0034] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0035] The remote sensing change detection method based on edge-enhanced cross-attention and multi-dimensional loss provided in the embodiment of the present invention can be applied to electronic devices such as mobile terminals, personal laptops, supercomputers, etc. The embodiment of the present invention does not impose any restrictions on the specific type of electronic devices.

[0036] Figure 1The flowchart shown is a remote sensing change detection method based on edge-enhanced cross-attention and multi-dimensional loss provided by an embodiment of the present invention. As an example but not a limitation, the change detection method can be applied to the above-mentioned electronic device, and the change detection method can include steps S101-S104, and each step is described below.

[0037] S101, input the dual-phase image into the trained twin backbone network to extract the multi-scale backbone features of the dual-phase image.

[0038] For example, the dual-phase image may include a front-phase image X1 (see Figure 2 (a)) and the later phase image X2 (see Figure 2 (b) in the figure.

[0039] It should be understood that the relevant parameters of the dual-phase image described below include relevant parameters of the front-phase image and relevant parameters of the back-phase image.

[0040] In one example, the multi-scale backbone features may include multi-level backbone features of different dimensions.

[0041] For example, see Figure 3 , the twin backbone network can be a Swin Transformer model. The multi-scale backbone feature Fi can include 4 levels of backbone features with different dimensions; that is, Fi = [Fi1, Fi2, Fi3, Fi4], where i = 1, 2, i = 1 represents the relevant parameters of the previous phase image, and i = 2 represents the relevant parameters of the later phase image.

[0042] Specifically, the dimensions of each feature map can be: Where H, W are the height and width of the dual-phase image, respectively, and R represents a real number.

[0043] S102, input the multi-scale backbone features into the trained unsharp mask cross-attention fusion model to extract the multi-dimensional spatial information and semantic information of the dual-phase image through the cross-attention operation, and extract the multi-dimensional edge information of the dual-phase image through the sharpening convolution kernel to obtain multi-level differential features of different scales.

[0044] In some embodiments, the unsharpen mask (USM) cross-attention fusion model may include a cross-attention module and a USM convolution module.

[0045] Exemplarily, the cross-attention module can be used to perform cross-attention operations on multi-scale backbone features to aggregate spatial information and semantic information of multiple dimensions of the bi-temporal image to obtain multi-level aggregated features of the bi-temporal image.

[0046] Exemplarily, the USM convolution module can extract edge information of the dual-phase image based on the sharpening convolution kernel according to the multi-level aggregation features of the dual-phase image to obtain a multi-level sharpening convolution map of the dual-phase image.

[0047] In a possible implementation, the cross-attention module can perform a cross-attention operation based on the n-th level comprehensive backbone features of the dual-phase image to extract the spatial information and semantic information therein to obtain the n-th level aggregated features of the dual-phase image.

[0048] Exemplarily, n is a positive integer less than or equal to Nmax, and Nmax is the total number of backbone features output by the twin backbone network. The Nmax-level comprehensive backbone feature is the Nmax-level backbone feature; starting from Nmax-1, the n-level comprehensive backbone feature is obtained by adding the upsampled n+1-level sharp feature map to the n-level backbone feature.

[0049] For example, see Figure 3 When Nmax=4, the fourth-level comprehensive backbone feature is the fourth-level backbone feature Fi4. The third-level comprehensive backbone feature is obtained by adding the fourth-level sharp feature map Ti4′ to the third-level backbone feature Fi3.

[0050] In one example, the cross-attention module can first convolve the n-th level comprehensive backbone feature of the dual-phase image to obtain the n-th level cross-attention vector of the dual-phase image. Then, the n-th level cross-attention vector is dimensionally transformed according to the height and width of the dual-phase image to obtain the n-th level first cross-attention vector and the n-th level second cross-attention vector of the dual-phase image. Finally, a bidirectional cross-attention operation is performed based on the n-th level first cross-attention vector and the n-th level second cross-attention vector to obtain the n-th level aggregate feature Tin.

[0051] The amount of calculation is large when the backbone features of each dimension are cross-attentioned according to the original scale. Therefore, the current change detection method will abandon the backbone features of low dimensions and only use the backbone features of the highest dimension for cross-attention operations; thereby abandoning the use of low-dimensional spatial information and semantic information of the dual-phase image, reducing the accuracy of change detection. The present invention performs dimensional transformation on the cross-attention vector obtained based on the backbone features according to the height and width of the dual-phase image, and performs cross-attention operations on vectors of different dimensions respectively; it can reduce the amount of calculation, realize the use of low-dimensional information, and improve the accuracy of change detection.

[0052] Exemplarily, the n-th level cross-attention vector may include the n-th level query vector of the bi-temporal image The n-th level key-value vector And the nth level value vector Vin.

[0053] Exemplarily, the dimension of the n-th level first cross-attention vector is the height of the bi-phase image, and the dimension of the n-th level second cross-attention vector is the width of the bi-phase image.

[0054] Specifically, the n-th level first cross-attention vector may include: the n-th level first query vector The first key vector of level n The n-th level first value vector ViH. The n-th level second cross-attention vector may include: the n-th level second query vector The second key vector of the nth level The nth level second value vector ViW.

[0055] Specifically, the bidirectional cross attention operation may include: respectively dot-multiplying the n-th level second query vector of the previous phase image and the second key vector of the nth level of the later phase image The first query vector of the nth level of the previous phase image and the first key vector of the nth level of the later phase image Get the n-th level front-phase cross attention map The second query vector of the nth level of the phase image after dot multiplication and the nth level second key vector of the previous phase image The first query vector of the nth level of the phase image after dot multiplication and the first key vector of the nth level of the previous phase image Get the cross-attention map after the nth level Then the n-th level front-phase cross attention map The nth level second value vector of the later phase image The first value vector of level n Point multiplication, and the result of the point multiplication is connected with the residual of the n-th level backbone feature F1n of the previous phase image to obtain the aggregated feature T1n of the previous phase image; the cross attention map of the n-th level after the phase The n-th level second value vector V1W and the n-th level first value vector V1H of the previous phase image are respectively multiplied by each other, and the result of the dot multiplication is combined with the n-th level backbone feature vector V1W of the next phase image. Residual connection obtains aggregate features of the later phase image

[0056] Specifically, the bidirectional cross-attention operation can be expressed as:

[0057]

[0058]

[0059] Add represents the residual connection operation;

[0060] in:

[0061]

[0062] In one possible implementation, see Figure 3 The USM convolution module can extract edge information from the n-th level aggregation feature of the dual-phase image based on the sharpening convolution kernel to obtain the n-th level sharpening feature map of the dual-phase image.

[0063] In one example, the sharpening convolution kernel may be constructed based on Gaussian filtering, which may satisfy the following formula:

[0064]

[0065] Among them, K is the sharpening convolution kernel.

[0066] Exemplarily, the n-th level sharpening feature map of the dual-phase image may satisfy the following formula:

[0067]

[0068] Wherein, Tin' is the n-th level sharp feature map of the pre-phase image / post-phase image, Tin is the n-th level aggregation feature of the pre-phase image / post-phase image, Conv(·) represents the convolution operation, and σ is a constant.

[0069] Optionally, the value of σ may be 0.6.

[0070] In one possible implementation, see Figure 3 , the n-th level differential feature can be obtained by performing a differential operation on the n-th level sharp feature map of the previous phase image and the n-th level sharp feature map of the subsequent phase image.

[0071] In one example, the nth level differential feature may satisfy the following formula:

[0072]

[0073] Where Dn is the nth level differential feature, and abs represents the differential operation.

[0074] Alternatively, see Figure 4 , when Nmax=4, the first-level differential feature D1 can be a 4-fold downsampled feature, and its dimension can be The second-level differential feature D2 can be an 8-fold downsampled feature, and its dimension can be The third level differential feature D3 can be a 16-fold downsampled feature, and its dimension can be The fourth level differential feature D4 can be a 32-fold downsampled feature, and its dimension can be

[0075]

[0076] S103, inputting the differential features into the trained multi-scale hierarchical connection decoder to upsample and / or downsample each level of differential features by different multiples, so as to obtain a multi-scale preliminary prediction result of the changed area.

[0077] In a possible implementation, if the USM cross attention model outputs a total of 4 levels of differential features, the multi-scale preliminary prediction results may include a first prediction result, a second prediction result, and a third prediction result of different scales. The three-level prediction effect of different scales has a more comprehensive representation capability.

[0078] For example, see Figure 5 , the multi-scale preliminary prediction result can be a binary mask image of different scales, where the pixel value of the changed area is 1 (white), and the pixel value of the unchanged area is 0 (black).

[0079] In one example, see Figure 4 In (a), the multi-scale hierarchical connection decoder can be used to add the 4th level differential feature D4 and the 3rd level differential feature D3 after 2 times upsampling to obtain a 16-fold downsampled fusion feature The first 8-fold fusion feature is obtained by adding D3 after 2 times upsampling, the second-level differential feature D2, and the first-level differential feature D1 after 2 times downsampling. Will Compared with 2 times upsampling Add together to obtain the first prediction result Y2.

[0080] In one example, see Figure 4 In (b), the multi-scale hierarchical connection decoder can also be used to add the 2x upsampled D3 and D2 to obtain the second 8x fusion feature Will D2, and 2x upsampling Add together to get the third 8-fold fusion feature Add D1 and D2 after 2 times upsampling to get the first 4 times fusion feature Will After 2 times upsampling D1 is added to get the second 4-fold fusion feature Will D1. After 2 times upsampling The addition results in the second prediction result Y1.

[0081] In one example, the multi-scale hierarchical connection decoder can also be used to: concatenate D1, and Y1 to obtain the 4-fold features after dimension conversion; then the 4-fold features after dimension conversion are upsampled 4 times to obtain the third prediction result Y0 with the same scale as the dual-phase image.

[0082] S104, weighted fusion is performed on the multi-scale preliminary prediction results to obtain a change prediction result.

[0083] In one example, different weights may be assigned to the first prediction result, the second prediction result, and the third prediction result, and then added together to obtain a change prediction result.

[0084] Exemplarily, the change prediction result may satisfy the following formula:

[0085] Y=1.0*Y0+0.3*Up4(Y1)+0.2*Up8(Y2)

[0086] Among them, Y is the change prediction result, Y0 is the third prediction result, Y1 is the second prediction result, Y2 is the first prediction result, Up4 represents a 4-fold upsampling operation, and Up8 represents an 8-fold upsampling operation.

[0087] According to the method provided by the present invention, by using multi-scale backbone features to obtain preliminary prediction results of multiple scales, and then combining these prediction results to obtain the final change prediction result, the multi-scale information extracted by the backbone network can be fully utilized; the edge information of the dual-phase image is extracted by the sharpening convolution kernel of the USM cross-attention fusion module to enhance the ability to extract edge information, thereby improving the accuracy of change detection. Furthermore, by transforming the dimensions of the cross-attention vector according to the height and width of the dual-phase image, and performing cross-self-attention operations on the cross-attention vectors of different dimensions, the amount of calculation can be reduced while utilizing the multi-scale information extracted by the backbone network.

[0088] Figure 6 The following is a schematic diagram of the implementation process of a training method for a twin backbone network, a USM cross-attention fusion model, and a multi-scale hierarchical connection decoder provided in an embodiment of the present invention. As an example but not a limitation, the training method can also be applied to the above-mentioned electronic device, and the training method can include steps S601-S605, and each step is described below.

[0089] S601, performing data preprocessing on the data set to obtain a sample dual-phase image.

[0090] In a possible implementation, the image data can be divided into a training set and a validation set according to a public data list or a certain ratio based on the publicly available data set or the registered homologous anterior and posterior phase images and their corresponding binary label maps collected by the user. Then, the samples of the training set and the validation set are preprocessed to obtain the sample dual-phase images.

[0091] Exemplarily, publicly available binary remote sensing building change detection datasets (LEVIR-CD+), edge-enhanced cross-attention and multidimensional loss-based remote sensing change detection datasets (Onera Satellite Change Detection Dataset, OSCD), S1GFloods, and Remote Sensing Farmland Semantic and Change Annotation Dataset (Hi-CAN) and other datasets can be used as sample dual-phase images in the training set and validation set.

[0092] In a possible implementation, the format of the sample dual-phase image can be unified into the uint8 format, and then the corresponding binary label map is downsampled by 4 times and 8 times respectively to obtain the original scale true binary mask map GT0, the true binary mask map GT1 of 4 times the feature scale, and the true binary mask map GT2 of 8 times the feature scale, which are the same as the scales of the first sample prediction result, the second sample prediction result, and the third sample prediction result, respectively; the data preprocessing of the sample dual-phase image is completed.

[0093] In one example, if a multispectral OSCD dataset is used, the images in the dataset have 13 bands and the original data is in uint16 format. Therefore, the [0, 1, 2] channels can be selected according to the required spectral channel parameters, and then the image can be unified to [128, 128, 3] size using a fixed step size sliding window cropping method, and the processed image is normalized to the maximum and minimum to obtain a sample dual-phase image in uint8 format.

[0094] In an example, if the binary label image is a binary mask image with a label of 0 (unchanged area) or 255 (changed area), the pixel values ​​of all pixels with a pixel value of 255 can be first converted to 1; then the binary label image after pixel value transformation is downsampled by 4 times and 8 times using bilinear interpolation to obtain true binary mask images of different scales [GT0, GT1, GT2].

[0095] S602, inputting the sample dual-phase images in the training set into the Siamese backbone network to extract sample multi-scale backbone features of the sample dual-phase images.

[0096] S603, input the multi-scale backbone features of the samples into the unsharp mask cross attention fusion model to obtain multi-level sample differential features of different scales.

[0097] S604, inputting the sample differential features into a multi-scale hierarchical connection decoder to upsample and / or downsample the sample differential features at each level by different multiples, to obtain a preliminary prediction result of the multi-scale samples of the changed area.

[0098] Exemplarily, the multi-scale sample preliminary prediction results may include a first sample prediction result, a second sample prediction result, and a third sample prediction result of different scales.

[0099] Exemplarily, the functions of the twin backbone network, USM cross-attention fusion model and multi-scale hierarchical connection decoder in steps S602-S604 of the training method are the same as the functions of the twin backbone network, USM cross-attention fusion model and multi-scale hierarchical connection decoder described in steps S101-S103 of the above-mentioned change detection method, except that the training method processes sample dual-phase images, and the change detection method processes dual-phase images to be detected; the specific content can be found in S101-S103 in the change detection method, which will not be repeated here.

[0100] S605, respectively compare the true binary mask image and sample prediction results of each scale to obtain the total loss, and train the twin backbone network, the USM cross-attention fusion model and the multi-scale hierarchical connection decoder according to the total loss to obtain the trained twin backbone network, the trained USM cross-attention fusion model and the trained multi-scale hierarchical connection decoder.

[0101] In one possible implementation, the maximum number of training times can be set. Each time the training is performed, the sample bi-phase images in the training set are partitioned, and the sample bi-phase images in each zone and their corresponding true binary mask images are input into the twin backbone network, USM cross-attention fusion model, and multi-scale hierarchical connection decoder in turn to obtain the multi-scale sample preliminary prediction results corresponding to the sample bi-phase images. Then, the preliminary prediction results of the multi-scale samples and the true binary mask images are compared to obtain the total loss of this round of training. According to the total loss, the AdamW optimizer is used to update the gradient and the Poly attenuation strategy is used to update the learning rate; after the update, the next training is continued until the maximum number of training times is reached.

[0102] Exemplarily, the maximum number of training times can be 50, the initial learning rate of the twin backbone network can be 6e-5, the initial learning rate of the USM cross-attention fusion model can be 6e-4, and the initial learning rate of the multi-scale hierarchical connection decoder can be 6e-4.

[0103] Exemplarily, after each training, an evaluation can be performed based on the sample bi-phase diagram in the validation set. Evaluation indicators can include the intersection over union of unchanged regions (Unchange_IoU,), intersection over union of changed regions (Changed_IoU), mean of Unchange_IoU and Changed_IoU (mIoU), F1 score of unchanged regions (Unchanged_F1Score), F1 score of changed regions (Changed_F1Score), mean of Unchanged_F1Score and Changed_F1Score (mF1Score), and overall classification accuracy (Overall_Accuracy, OA). Then, the training weights used in the training round with the best mIoU value in all training rounds are determined as the final weights of the model, and a trained twin backbone network, a trained USM cross-attention fusion model, and a trained multi-scale hierarchical connection decoder are obtained.

[0104] In one example, the total loss can satisfy the following formula:

[0105] L=1.0*LCE ( Y0′,GT0 ) +0.3*LCE ( Y1′,GT1)+0.2*LCE ( Y2′,GT2 )

[0106] Among them, L is the total loss, LCE represents the cross entropy loss, Y0′, Y1′, and Y2′ are the third sample prediction results, the second sample prediction results, and the first sample prediction results of the sample bi-phase image, respectively, and GT0, GT1, and GT2 are the true binary mask images of the sample bi-phase images with the same scale as Y0′, Y1′, and Y2′, respectively.

[0107] According to the training method provided by the present invention, determining the total loss based on the preliminary prediction results of multi-scale samples can improve the label utilization rate and enhance the prediction effect of the multi-scale hierarchical connection decoder. The bidirectional cross-attention operation can reduce the amount of calculation when the cross-attention vector of each scale performs the cross-attention operation, thereby accelerating the model training process.

[0108] Figure 7 The structure diagram of a remote sensing change detection device based on edge-enhanced cross-attention and multi-dimensional loss provided by an embodiment of the present invention is shown. As an example but not a limitation, the device 700 may include a multi-scale backbone feature extraction unit 710, a bidirectionally guided edge enhancement unit 720, a preliminary prediction unit 730 and a comprehensive prediction unit 740.

[0109] The multi-scale backbone feature extraction unit 710 includes a trained twin backbone network, and the multi-scale backbone feature extraction unit 710 is used to: input a dual-phase image into the trained twin backbone network, and extract multi-scale backbone features of the dual-phase image, wherein the dual-phase image includes a front-phase image and a rear-phase image with the same shooting scene but different shooting times, and the multi-scale backbone features include multi-level backbone features with different dimensions;

[0110] The bidirectionally guided edge enhancement unit 720 includes a trained lightweight unsharp mask fusion feature, and the bidirectionally guided edge enhancement unit 720 is used to: input the multi-scale backbone feature into the trained unsharp mask cross-attention fusion model to extract multi-dimensional spatial information and semantic information of the bi-phase image through the cross-attention operation, and extract multi-dimensional edge information of the bi-phase image through the sharpening convolution kernel to obtain multi-level differential features of different scales;

[0111] The preliminary prediction unit 730 includes a trained multi-scale hierarchical connection decoder, and the preliminary prediction unit 730 is used to: input the differential features into the trained multi-scale hierarchical connection decoder, so as to upsample and / or downsample each level of differential features by different multiples, and obtain a multi-scale preliminary prediction result of the change area;

[0112] The comprehensive prediction unit 740 is used to perform weighted fusion on the multi-scale preliminary prediction results to obtain the change prediction result.

[0113] According to the device provided by the present invention, preliminary prediction results of multiple scales are obtained by utilizing multi-scale backbone features, and then these prediction results are combined to obtain the final change prediction result; the multi-scale information extracted by the backbone network can be fully utilized to improve the detection accuracy.

[0114] Figure 8 FIG. 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 8 The electronic device 800 shown may include: at least one processor 810 ( Figure 8 Only one processor is shown in the figure), a memory 820, and a computer program 830 stored in the memory 820 and executable on the at least one processor 810, wherein the processor 810 implements the steps of any of the above-mentioned method embodiments when executing the computer program 830.

[0115] The electronic device 800 may be a processing device such as a robot that can implement the above method. The embodiment of the present invention does not impose any limitation on the specific type of the electronic device.

[0116] Those skilled in the art will understand that Figure 8The electronic device 800 is merely an example and does not constitute a limitation on the electronic device. The electronic device 800 may include more or fewer components than shown in the figure, or may combine certain components, or may include different components. For example, the electronic device 800 may also include an input and output interface.

[0117] The processor 810 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASTC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0118] The memory 820 may be an internal storage unit in some embodiments, such as a hard disk or a memory. The memory 820 may also be an external storage device in other embodiments, such as a plug-in hard disk, a smart memory card (SmartMemory Card, SMC), a secure digital (Secure Dngntal, SD) card, a flash card (Flash Card), etc. Further, the memory 820 may also include both an internal storage unit and an external storage device. The memory 820 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as the program code of the computer program, etc. The memory 820 may also be used to temporarily store data that has been output or is to be output.

[0119] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0120] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0121] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0122] An embodiment of the present invention provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0124] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0125] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

Claims

1. A remote sensing change detection method based on edge-enhanced cross-attention and multi-dimensional loss, characterized in that: include: Inputting a dual-phase image into a trained twin backbone network, extracting multi-scale backbone features of the dual-phase image, wherein the dual-phase image includes a front-phase image and a back-phase image with the same shooting scene but different shooting times, and the multi-scale backbone features include multi-level backbone features with different dimensions; Input the multi-scale backbone features into the trained unsharp mask cross-attention fusion model to extract the multi-dimensional spatial information and semantic information of the dual-phase image through the cross-attention operation, and extract the multi-dimensional edge information of the dual-phase image through the sharpening convolution kernel to obtain multi-level differential features of different scales; Inputting the differential features into a trained multi-scale hierarchical connection decoder to upsample and / or downsample the differential features at each level by different multiples to obtain a multi-scale preliminary prediction result of the change area; The multi-scale preliminary prediction results are weighted and fused to obtain a change prediction result.

2. The method according to claim 1, characterized in that: The unsharp mask cross attention fusion model includes a cross attention module and an unsharp mask convolution module; The cross attention module is used to perform a cross attention operation on the multi-scale backbone features to achieve mutual guidance between the multi-scale backbone features of the front-phase image and the multi-scale backbone features of the back-phase image, and aggregate the spatial information and semantic information of multiple dimensions of the dual-phase image to obtain the multi-level aggregated features of the dual-phase image; The unsharp mask convolution module is used to extract multi-dimensional edge information of the dual-phase image based on the sharpening convolution kernel according to the multi-level aggregation features to obtain a multi-level sharp feature map of the dual-phase image, wherein the n-th level differential feature is obtained based on the difference between the n-th level sharp feature maps of the front-phase image and the rear-phase image.

3. The method according to claim 2, characterized in that The cross attention module is specifically used for: Convolving the nth level comprehensive backbone feature of the dual-phase image to obtain the nth level cross-attention vector of the dual-phase image, wherein n is a positive integer less than or equal to Nmax, Nmax is the total number of backbone features output by the twin backbone network, the Nmaxth level comprehensive backbone feature is the Nmaxth level backbone feature, and starting from Nmax-1, the nth level comprehensive backbone feature is obtained by adding the upsampled n+1th level sharp feature map to the nth level backbone feature; Respectively transforming the dimension of the n-th level cross-attention vector according to the height and width of the dual-phase image to obtain the n-th level first cross-attention vector and the n-th level second cross-attention vector of the dual-phase image, wherein the dimension of the n-th level first cross-attention vector is the height of the dual-phase image, and the dimension of the n-th level second cross-attention vector is the width of the dual-phase image; A bidirectional cross-attention operation is performed based on the n-th level first cross-attention vector and the n-th level second cross-attention vector to obtain an n-th level aggregate feature.

4. The method according to claim 2 or 3, characterized in that: The n-th level sharpening feature map of the dual-phase image satisfies the following formula: Wherein, i=1,2, i=1 represents the front phase image, represents the back phase image; Tin' is the nth level sharpening feature map of the front phase image / back phase image, Tin is the nth level aggregation feature of the front phase image / back phase image, Conv(·) represents the convolution operation, is the sharpening convolution kernel, and σ is a constant.

5. The method according to claim 1, characterized in that The unsharp mask cross attention fusion model outputs a total of 4 levels of differential features, and the multi-scale preliminary prediction results include a first prediction result, a second prediction result, and a third prediction result of different scales; the multi-scale hierarchical connection decoder is used to: The 4th level differential feature after 2 times upsampling is added to the 3rd level differential feature to obtain the 16 times downsampling fusion feature; The third-level differential features after 2 times upsampling, the second-level differential features, and the first-level differential features after 2 times downsampling are added to obtain the first 8-fold fusion features; Adding the first 8-fold fusion feature to the 16-fold down-sampling fusion feature after 2-fold up-sampling to obtain the first prediction result; Adding the third-level differential features after the 2-fold upsampling and the second-level differential features to obtain a second 8-fold fusion feature; The second 8-fold fusion feature, the second-level differential feature, and the 16-fold down-sampling fusion feature after the 2-fold upsampling are added to obtain a third 8-fold fusion feature; Adding the first-level differential feature to the second-level differential feature after 2-fold upsampling to obtain a first 4-fold fusion feature; Adding the first 4-fold fusion feature, the second 8-fold fusion feature after 2-fold upsampling, and the first-level differential feature to obtain a second 4-fold fusion feature; Adding the second 4-fold fusion feature, the 1-level differential feature, the first 4-fold fusion feature, and the third 8-fold fusion feature after 2-fold upsampling to obtain the second prediction result; Splicing the first-level differential feature, the first 4-fold fused feature, the second 4-fold fused feature, and the second prediction result to obtain a spliced ​​4-fold feature; Performing a convolution operation on the spliced ​​4-fold features to convert the dimensions of the spliced ​​4-fold features to obtain 4-fold features after dimension conversion; The 4-fold features after the dimension conversion are upsampled 4 times to obtain the third prediction result.

6. The method according to claim 5, characterized in that The twin backbone network, the unsharp mask cross attention fusion model and the multi-scale hierarchical connection decoder are jointly trained according to the total loss, and the total loss satisfies the following formula: L=1.0*LCE ( Y0′,GT0 ) +0.3*LCE ( Y1′,GT1)+0.2*LCE ( Y2′,GT2 ) Among them, L is the total loss, LCE represents the cross entropy loss, Y0′, Y1′, and Y2′ are respectively the third sample prediction result, the second sample prediction result, and the first sample prediction result of the sample dual-phase image, and GT0, GT1, and GT2 are the true binary mask images of the sample dual-phase image with the same scale as Y0′, Y1′, and Y2′, respectively.

7. The method according to claim 5, characterized in that The change prediction result satisfies the following formula: Y=1.0*Y0+0.3*Up4(Y1)+0.2*Up8(Y2) Among them, Y is the change prediction result, Y0 is the third prediction result, Y1 is the second prediction result, Y2 is the first prediction result, Up4 represents a 4-fold upsampling operation, and Up8 represents an 8-fold upsampling operation.

8. A remote sensing change detection device based on edge-enhanced cross attention and multi-dimensional loss, characterized in that: include: A multi-scale backbone feature extraction unit, the multi-scale backbone feature extraction unit includes a trained twin backbone network, and the multi-scale backbone feature extraction unit is used to: input a dual-phase image into the trained twin backbone network to extract multi-scale backbone features of the dual-phase image, wherein the dual-phase image includes a front-phase image and a rear-phase image with the same shooting scene but different shooting times, and the multi-scale backbone features include multi-level backbone features with different dimensions; A bidirectionally guided edge enhancement unit, the bidirectionally guided edge enhancement unit includes a trained unsharp mask cross-attention fusion model, and the bidirectionally guided edge enhancement unit is used to: input the multi-scale backbone features into the trained unsharp mask cross-attention fusion model to extract multi-dimensional spatial information and semantic information of the bi-phase image through a cross-attention operation, and extract multi-dimensional edge information of the bi-phase image through a sharpening convolution kernel to obtain multi-level differential features of different scales; A preliminary prediction unit, the preliminary prediction unit comprising a trained multi-scale hierarchical connection decoder, the preliminary prediction unit being used to: input the differential features into the trained multi-scale hierarchical connection decoder, so as to upsample and / or downsample the differential features at each level by different multiples, and obtain a multi-scale preliminary prediction result for the changed area; The comprehensive prediction unit is used to perform weighted fusion on the multi-scale preliminary prediction results to obtain a change prediction result.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the electronic device, the method according to any one of claims 1 to 7 is implemented.

Images