Remote sensing image change detection method and device, electronic equipment and storage medium

Abstract

The application discloses a remote sensing image change detection method and device, electronic equipment and storage medium, and belongs to the field of remote sensing image change detection. The method comprises the following steps: acquiring n_way*k_shot first double-time-phase remote sensing images and n_way*x_query second double-time-phase remote sensing images; processing each first and second double-time-phase remote sensing image through a feature network to obtain n_way*k_shot and n_way*x_query fusion difference feature maps; obtaining a prototype decision space according to the n_way*k_shot fusion difference feature maps; determining the distance between each fusion difference feature map in the n_way*x_query fusion difference feature maps and each prototype vector in the prototype decision space; determining the predicted change category of each second double-time-phase remote sensing image according to the distance, and optimizing the feature network according to the predicted change category and the label carried by each second double-time-phase remote sensing image; and performing change detection on a target double-time-phase remote sensing image by using the optimized feature network, thereby enhancing the accuracy and robustness of remote sensing image change detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image change detection, and particularly relates to a remote sensing image change detection method and device, an electronic device and a storage medium. BACKGROUND

[0002] Remote sensing image change detection technology is crucial in geographic condition monitoring, land use, environmental protection and disaster assessment engineering. It can timely and accurately monitor surface changes, support scientific decision-making and resource management, and help cope with the challenges brought by changes, which has important practical significance and application value.

[0003] The initial remote sensing image change detection relies on visual analysis and simple image processing techniques such as image difference method and threshold method. Although these methods are intuitive, their precision is limited. Subsequently, with the advancement of computer technology, statistical methods and machine learning techniques have been introduced, and principal component analysis and support vector machine methods have been used for feature extraction and classification, significantly improving the effect of remote sensing image change detection. Later, researchers further improved the accuracy and efficiency of remote sensing image detection by using emerging methods such as multi-temporal image analysis and ensemble learning. However, although these traditional methods have promoted the development of remote sensing image change detection to some extent, they still face challenges such as insufficient feature extraction capability and poor model interpretability.

[0004] Deep learning, as a machine learning technology that has emerged in recent years in the big data environment, has made revolutionary success in remote sensing image change detection and other fields by constructing a multi-layer structure model and a large amount of training data to learn complex features in the data. However, the current mainstream deep learning algorithms generally lack interpretability, and their "black box" characteristics make the internal structure of the model complex and difficult to understand, resulting in a lack of clear decision basis for the prediction results, which is particularly significant in change detection scenarios that require high credibility and traceability. SUMMARY

[0005] To solve the above problems, the present application embodiment provides a remote sensing image change detection method, device, electronic equipment and storage medium.

[0006] In a first aspect, the present application embodiment provides a remote sensing image change detection method, comprising:

[0007] acquire n_way*k_shot first dual-time remote sensing images and n_way*x_query second dual-time remote sensing images, the first dual-time remote sensing images and the second dual-time remote sensing images are respectively one of a plurality of dual-time remote sensing images containing n_way changed categories, the n_way, N_way, k_shot, x_query are all positive integers, the n_way≤N_way, the dual-time remote sensing images are labeled with labels representing changed categories;

[0008] process each first and second dual-time remote sensing image through a feature network, to obtain n_way*k_shot and n_way*x_query fusion difference feature maps;

[0009] According to the n_way*k_shot fusion difference feature map, a prototype decision space containing n_way+1 prototype vectors is obtained.

[0010] Determine the distance between each fusion difference feature map in the n_way*x_query fusion difference feature map and each prototype vector in the prototype decision space.

[0011] According to the distance, determine the predicted changed category of each second dual-time remote sensing image.

[0012] According to the determined predicted changed category of each second dual-time remote sensing image and the label carried by each second dual-time remote sensing image, optimize the feature network.

[0013] Use the optimized feature network to perform change detection on the target dual-time remote sensing image.

[0014] In one possible implementation, the feature network includes a DenseNet network, an SE (Squeeze-and-Excitation) mechanism, and an AWF-FPN (Adaptive Weighted Feature Pyramid Network).

[0015] The processing of each first and second dual-time remote sensing image through the feature network to obtain n_way*k_shot and n_way*x_query fusion difference feature maps includes:

[0016] For each first dual-time remote sensing image and each second dual-time remote sensing image, extract n_way*k_shot sets of M-level feature map pairs and n_way*x_query sets of M-level feature map pairs through the DenseNet network and the SE mechanism.

[0017] The n_way*k_shot group of M-level feature map pairs is subjected to pixel-by-pixel subtraction operation at each level to obtain an n_way*k_shot group of M difference feature maps, and the n_way*x_query group of M-level feature map pairs is subjected to pixel-by-pixel subtraction operation at each level to obtain an n_way*x_query group of M difference feature maps;

[0018] Each of the n_way*k_shot group of M difference feature maps is fused by using an adaptive weighted feature pyramid network AWF-FPN to obtain n_way*k_shot fused difference feature maps, and each of the n_way*x_query group of M difference feature maps is fused to obtain n_way*x_query fused difference feature maps.

[0019] In a possible implementation, the first and second dual-time-phase remote sensing images each have a first target size;

[0020] The n_way*k_shot group of M-level feature map pairs and the n_way*x_query group of M-level feature map pairs are extracted for each first dual-time-phase remote sensing image and each second dual-time-phase remote sensing image by using a DenseNet network and an SE mechanism, and the extraction includes:

[0021] Step one, for each first and second dual-time-phase remote sensing image, n_way*k_shot initial convolution feature map pairs and n_way*x_query initial convolution feature map pairs each having a second target size are extracted by using an initial convolution module of a DenseNet network;

[0022] Step two, the n_way*k_shot initial convolution feature map pairs and the n_way*x_query initial convolution feature map pairs are respectively outputted by using a first stage module of the DenseNet network to obtain n_way*k_shot initial feature map pairs and n_way*x_query initial feature map pairs each having a first size;

[0023] Step three, the n_way*k_shot initial feature map pairs and the n_way*x_query initial feature map pairs each having the first size are subjected to an SE mechanism to obtain a first vector of 1×1×C1, where C1 is a channel number;

[0024] Step four, the first vector is used to assign weights to the n_way*k_shot initial feature map pairs and the n_way*x_query initial feature map pairs each having the first size in a channel-by-channel manner to respectively obtain n_way*k_shot first feature map pairs and n_way*x_query first feature map pairs each having the first size;

[0025] Step 5, and so on, for the (M-1)th feature map pair of the n_way*k_shot group and the n_way*x_query group of size M-1, repeat steps 2-4 above through the M-th stage module of the DenseNet network and the SE mechanism to obtain the Mth feature map pair of the n_way*k_shot group and the n_way*x_query group of size M; thus, M layers of feature map pairs of the n_way*k_shot group and the n_way*x_query group are obtained.

[0026] In one possible implementation, the step of utilizing the Adaptive Weighted Feature Pyramid Network (AWF-FPN) to fuse each set of differential feature maps in the n_way*k_shot group of M differential feature maps to obtain n_way*k_shot fused differential feature maps, and fusing each set of differential feature maps in the n_way*x_query group of M differential feature maps to obtain n_way*x_query fused differential feature maps, includes:

[0027] Using the Adaptive Weighted Feature Pyramid Network (AWF-FPN), combining spatial and channel weights, the differential feature maps of each of the M differential feature maps in the n_way*k_shot group are fused layer by layer from the top down to obtain n_way*k_shot fused differential feature maps. Furthermore, combining spatial and channel weights, the differential feature maps of each of the M differential feature maps in the n_way*x_query group are fused layer by layer from the top down to obtain n_way*x_query fused differential feature maps. The result of each fusion layer is then smoothed using a 3×3 convolution to obtain the input for the next layer.

[0028] In one possible implementation, obtaining the prototype decision space containing n_way+1 prototype vectors based on the n_way*k_shot fused difference feature maps includes:

[0029] Extract the features of all pixels in the region corresponding to the change category of each of the n_way*k_shot fusion difference feature maps, as well as the features of all pixels in the background region, to obtain n_way+1 feature sets;

[0030] Based on the n_way+1 feature sets, determine the prototype decision space containing n_way+1 prototype vectors.

[0031] In one possible implementation, determining the distance between each of the n_way*x_query fusion difference feature maps and each prototype vector in the prototype decision space includes:

[0032]

[0033] wherein the sim k (x, y) represents any one of the n_way*x_query fusion difference feature maps F xy The cosine similarity of the pixel position (x, y) to P k The cosine similarity of the pixel position (x, y) to P k is any one of the prototype vectors in the prototype decision space, and a is a scaling factor, k [1, n_way+1].

[0034] In a possible implementation, the determining the predicted change class of each second dual-time-phase remote sensing image according to the distance comprises:

[0035] The cosine similarity of the pixel position (x, y) to P xy The cosine similarity of the pixel position (x, y) to P k The cosine similarity of the pixel position (x, y) to P

[0036] The first tensor is up-sampled to obtain a second tensor, the size of the second tensor is the same as that of the dual-time-phase remote sensing image, and the second tensor represents the response score of each pixel to the prototype vector of all classes at the original image resolution.

[0037] The predicted change class of each second dual-time-phase remote sensing image is determined according to the second tensor.

[0038] In a second aspect, an embodiment of the present application provides a remote sensing image change detection device, comprising:

[0039] A remote sensing image acquisition module is configured to acquire n_way*k_shot first dual-time-phase remote sensing images and n_way*x_query second dual-time-phase remote sensing images, wherein the first dual-time-phase remote sensing images and the second dual-time-phase remote sensing images are respectively one of a plurality of dual-time-phase remote sensing images containing n_way change classes, n_way, N_way, k_shot and x_query are positive integers, n_way≤N_way, and the dual-time-phase remote sensing images are labeled with change class indicators.

[0040] A processing module is configured to process each first and second dual-time-phase remote sensing image through a feature network to obtain n_way*k_shot and n_way*x_query fusion difference feature maps.

[0041] A prototype decision space obtaining module is configured to obtain a prototype decision space containing n_way+1 prototype vectors according to the n_way*k_shot fusion difference feature maps.

[0042] a distance determination module configured to determine a distance between each of the n_way*x_query fused difference feature maps and each of the prototype vectors in the prototype decision space;

[0043] a predicted change class determination module configured to determine a predicted change class of each of the second dual-temporal remote sensing images according to the distances;

[0044] a feature network optimization module configured to optimize the feature network according to the determined predicted change class of each of the second dual-temporal remote sensing images and the label carried by each of the second dual-temporal remote sensing images;

[0045] a change detection module configured to perform change detection on a target dual-temporal remote sensing image using the optimized feature network.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, comprising:

[0047] a memory and a processor, the processor and the memory communicate with each other through a bus; the memory stores program instructions executable by the processor, and the processor calling the program instructions can execute the method of the first aspect and each step in various possible implementations.

[0048] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect and each step in various possible implementations.

[0049] In a fifth aspect, an embodiment of the present application provides a computer program product containing instructions, which, when the computer program product is run on a computer, causes the method of the first aspect and each step in various possible implementations to be executed by the computer.

[0050] The technical scheme provided by the embodiment of the present application has the beneficial effects that: the present application combines prototype learning with remote sensing image change detection, aiming to improve the transparency of the remote sensing image change detection decision process and enhance the trustworthiness of the results. In addition, in order to deal with the challenge of weak feature difference in current change detection, a feature network (including DenseNet network, SE mechanism and AWF-FPN) is introduced into the model. The network aims to improve the feature difference between dual-temporal remote sensing images, thereby enhancing the accuracy and robustness of remote sensing image change detection. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 a flowchart of a remote sensing image change detection method provided by an embodiment of the present application;

[0052] Figure 2 This is a schematic block diagram of a remote sensing image change detection device provided in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0056] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0057] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if monitoring (the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when monitoring (the stated condition or event)," or "in response to monitoring (the stated condition or event)."

[0058] Currently, traditional methods, such as image differencing and thresholding methods that rely on visual analysis and simple image processing techniques, principal component analysis and support vector machine methods, multi-temporal image analysis and ensemble learning methods, have promoted the development of remote sensing image change detection to some extent, but they still face challenges such as insufficient feature extraction capabilities and poor model interpretability.

[0059] Deep learning, a machine learning technology that has emerged in recent years in the context of big data, has achieved revolutionary success in fields such as remote sensing image change detection by building multi-layered models and massive amounts of training data to learn complex features in the data. However, current mainstream deep learning algorithms generally suffer from insufficient interpretability. Their "black box" nature makes the internal structure of the model complex and difficult to understand, resulting in a lack of clear decision-making basis for the prediction results. This is particularly evident in change detection scenarios that require high reliability and traceability.

[0060] In view of this, embodiments of the present invention provide a flowchart of a remote sensing image change detection method. For example... Figure 1 As shown, the method may include the following steps:

[0061] Step 101: Obtain n_way*k_shot first bi-temporal remote sensing images and n_way*x_query second bi-temporal remote sensing images. The first and second bi-temporal remote sensing images are one of multiple bi-temporal remote sensing images containing n_way change categories. n_way, N_way, k_shot, and x_query are all positive integers, and n_way ≤ N_way. The bi-temporal remote sensing images are labeled with labels representing change categories.

[0062] Step 102: For each of the first and second dual-temporal remote sensing images, process them through a feature network to obtain n_way*k_shot and n_way*x_query fused difference feature maps respectively.

[0063] Step 103: Based on the n_way*k_shot fused difference feature maps, obtain the prototype decision space containing n_way+1 prototype vectors.

[0064] Step 104: Determine the distance between each fusion difference feature map in the n_way*x_query fusion difference feature maps and each prototype vector in the prototype decision space.

[0065] Step 105: Determine the predicted change category for each second bi-temporal remote sensing image based on the distance.

[0066] Step 106: Optimize the feature network based on the predicted change category of each second bi-temporal remote sensing image and the label attached to each second bi-temporal remote sensing image.

[0067] Step 107: Use the optimized feature network to perform change detection on the target dual-temporal remote sensing image.

[0068] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments of the present invention. It should be noted that the terms "first" and "second" involved in the embodiments of the present invention do not have limitations on size, order, or quantity, but are only used to distinguish them by name. For example, "first tensor" and "second tensor" are used to distinguish two different tensors.

[0069] First, in conjunction with the embodiments of the present invention, step 101 above, namely, "acquiring n_way*k_shot first bi-temporal remote sensing images and n_way*x_query second bi-temporal remote sensing images, wherein the first bi-temporal remote sensing images and the second bi-temporal remote sensing images are each one of multiple bi-temporal remote sensing images containing n_way change categories, where n_way, N_way, k_shot, and x_query are all positive integers, n_way ≤ N_way, and the bi-temporal remote sensing images are labeled with labels representing change categories", will be described in detail.

[0070] In this embodiment of the invention, all change categories of the dual-temporal remote sensing image are determined to be N_way change categories, where N_way is a positive integer. That is, the possible change categories of the dual-temporal remote sensing image are N_way change categories. The preceding and subsequent temporal remote sensing images in the dual-temporal remote sensing image are considered as a data pair, and the change category of the subsequent temporal remote sensing image relative to the preceding temporal remote sensing image may include class1, class2, ..., classN_way.

[0071] For example, the N_way change categories can be class1, class2, class3, class4, and class5. It should be noted that in addition to the N_way change categories, the dual-temporal remote sensing image also includes a background area.

[0072] In this embodiment of the invention, firstly, n_way change categories are selected from N_way change categories. For example, class1, class2, and class3. Further, multiple bi-temporal remote sensing images including the n_way change categories are determined. For example, multiple bi-temporal remote sensing images including class1, class2, and class3. Then, k_shot support sets and x_query query sets corresponding to each of the n_way change categories are obtained from the multiple bi-temporal remote sensing images including the n_way change categories. Each support set in the k_shot support set includes a first bi-temporal remote sensing image, and each query set in the x_query query set includes a second bi-temporal remote sensing image. The first and second bi-temporal remote sensing images are each one of the multiple bi-temporal remote sensing images including the n_way change categories. The size of the first and second bi-temporal remote sensing images is a first target size, n_way is a positive integer less than or equal to N_way, and the bi-temporal remote sensing images are labeled to indicate the change category of the bi-temporal remote sensing images.

[0073] In other words, for each change category, only k_shot support sets and x_query query sets corresponding to each change category are selected from multiple bi-temporal remote sensing images including n_way change categories.

[0074] It should be noted that step 101 only shows an example of selecting n_way change categories as class1, class2, and class3 from N_way change categories. Step 101 can also select n_way change categories as class1, class2, and class4 from N_way change categories. That is, the n_way change categories can be any three change categories randomly selected multiple times from class1, class2, class3, class4, and class5. Furthermore, in this embodiment of the invention, the n_way change categories can be one, two, ..., or N_way change categories, and are not necessarily three.

[0075] The following describes in detail step 102, namely, "processing each first and second dual-temporal remote sensing image through a feature network to obtain n_way*k_shot and n_way*x_query fused difference feature maps, with reference to embodiments of the present invention.

[0076] In this embodiment of the invention, each first bi-temporal remote sensing image and each second bi-temporal remote sensing image are processed by a feature network to obtain n_way*k_shot fused difference feature maps and n_way*x_query fused difference feature maps. The aforementioned feature network can also be called a feature extraction and feature fusion network.

[0077] As one possible implementation, the feature network includes: DenseNet network, SE mechanism, and Adaptive Weighted Feature Pyramid Network (AWF-FPN). Specifically, for each first bi-temporal remote sensing image and each second bi-temporal remote sensing image, processing through the feature network to obtain n_way*k_shot fusion difference feature maps and n_way*x_query fusion difference feature maps can be achieved as follows: For each first bi-temporal remote sensing image and each second bi-temporal remote sensing image, using the DenseNet network and SE mechanism, n_way*k_shot groups of M-level feature map pairs and n_way*x_query groups of M-level feature map pairs are extracted; the n_way*k_shot groups of M-level feature map pairs undergo pixel-wise subtraction at each level to obtain n_ M differential feature maps are obtained from the group of way*k_shot, and the feature maps of the group of n_way*x_query are subtracted pixel by pixel at each level to obtain the group of n_way*x_query differential feature maps. Then, the differential feature maps of each group of n_way*k_shot are fused using the Adaptive Weighted Feature Pyramid Network (AWF-FPN) to obtain the group of n_way*k_shot fused differential feature maps. Finally, the differential feature maps of each group of n_way*x_query are fused to obtain the group of n_way*x_query fused differential feature maps.

[0078] First, a detailed explanation is given of "extracting feature map pairs of M levels from the n_way*k_shot group and the n_way*x_query group from each first bi-temporal remote sensing image and each second bi-temporal remote sensing image using the DenseNet network and SE mechanism".

[0079] As one possible implementation, for each first bi-temporal remote sensing image and each second bi-temporal remote sensing image, extracting n_way*k_shot groups of M-level feature map pairs and n_way*x_query groups of M-level feature map pairs using the DenseNet network and SE mechanism can be specifically as follows:

[0080] Step 1: For each of the first and second dual-temporal remote sensing images with a size equal to the size of the first target, the initial convolutional module of the DenseNet network is used to extract n_way*k_shot and n_way*x_query initial convolutional feature map pairs with a size equal to the size of the second target.

[0081] For example, for each first bi-temporal remote sensing image with a size of 512×512×3 and each second bi-temporal remote sensing image with a size of 512×512×3, the initial convolutional module of the DenseNet network (64 7×7 convolutional kernels, stride 2, padding 3) extracts n_way*k_shot and n_way*x_query feature map pairs with a size of 256×256×64. These n_way*k_shot and n_way*x_query feature map pairs with a size of 256×256×64 are then subjected to batch normalization, ReLU activation, and max pooling operations (3×3 window, stride 2, padding 1), outputting n_way*k_shot and n_way*x_query initial convolutional feature map pairs with a size of 128×128×64.

[0082] Step 2: For the initial convolutional feature map pairs of n_way*k_shot and n_way*x_query, respectively, pass them through the first stage module of the DenseNet network to output the initial feature map pairs of n_way*k_shot and n_way*x_query groups with the first size.

[0083] It should be noted that, in this embodiment of the invention, the first stage module includes a first stage dense block and a first stage transition layer, ..., the Mth stage module includes an Mth stage dense block and a BatchNorm layer. The stage modules between the first stage module and the Mth stage module each include a corresponding stage dense block and a corresponding stage transition layer. When M=4, the DenseNet network's first stage module includes a first stage dense block and a first stage transition layer, the second stage module includes a second stage dense block and a second stage transition layer, the third stage module includes a third stage dense block and a third stage transition layer, and the fourth stage module includes a fourth stage dense block and a BatchNorm layer.

[0084] Specifically, features are extracted from the first-stage dense block of the DenseNet network for each of the n_way*k_shot and n_way*x_query initial convolutional feature map pairs. Then, channel compression and downsampling are performed through the first-stage transition layer to output n_way*k_shot and n_way*x_query initial feature map pairs with a size of the first size.

[0085] For example, n_way*k_shot initial convolutional feature map pairs and n_way*x_query initial convolutional feature map pairs with a size of 128×128×64 are processed through the first-stage dense block of the DenseNet network (containing six dense layers, with channels increasing layer by layer (at a growth rate of 32), and the number of output channels is 256 (64 + 6×32)) to extract features; then, channel compression (down to 128) and downsampling are performed through the first-stage transition layer, outputting n_way*k_shot initial feature map pairs and n_way*x_query initial feature map pairs with a size of 64×64×128.

[0086] Step 3: The initial feature map pairs of the n_way*k_shot group and the n_way*x_query group with the first size are processed by the SE mechanism to obtain a first vector of 1×1×C1, where C1 is the number of channels.

[0087] Specifically, the initial feature map pairs of the n_way*k_shot group and the initial feature map pairs of the n_way*x_query group, which are of the first size, are compressed into a second vector of 1×1×C1 by the SE mechanism compression module using global average pooling operation, where C1 is the number of channels.

[0088] For example, the initial feature map pairs of the n_way*k_shot group and the initial feature map pairs of the n_way*x_query group with the above size of 64×64×128 are compressed into a second vector of 1×1×128 by the compression module of the SE mechanism (128 channels) using global average pooling operation.

[0089] The second vector, after passing through the first fully connected layer of the Excitation module using the SE mechanism, has its size changed from 1×1×C1 to 1×1×C1 / R; it then passes through the second fully connected layer of the Excitation module using the SE mechanism to obtain a first vector of 1×1×C1. The activation function of the first fully connected layer is ReLU, and the activation function of the second fully connected layer is Sigmoid. There is a hyperparameter R between the two fully connected layers.

[0090] For example, the first vector of 1×1×128 changes its size from 1×1×128 to 1×1×128 / R after passing through the first fully connected layer of the Excitation module of the SE mechanism; then it passes through the second fully connected layer of the Excitation module of the SE mechanism, and the vector size changes from 1×1×128 / R to 1×1×128, thus obtaining the first vector of 1×1×128.

[0091] Step four: Using the first vector, assign weights to each channel of the initial feature maps of the n_way*k_shot group and the n_way*x_query group, respectively, to obtain the first feature map pairs of the n_way*k_shot group and the n_way*x_query group, respectively, with the first vector representing the first channel. "Channel-by-channel" refers to each individual channel.

[0092] For example, a first vector of size 1×1×128 is used to assign channel-wise weights to the initial feature map pairs of the n_way*k_shot group and the n_way*x_query group of size 64×64×128, thereby obtaining the first feature map pairs of the n_way*k_shot group and the n_way*x_query group of size 64×64×128.

[0093] Step 5, and so on, for the (M-1)th feature map pair of the n_way*k_shot group and the n_way*x_query group of size M-1, repeat steps 2-4 above through the M-th stage module of the DenseNet network and the SE mechanism to obtain the Mth feature map pair of the n_way*k_shot group and the n_way*x_query group of size M; thus, M layers of feature map pairs of the n_way*k_shot group and the n_way*x_query group are obtained.

[0094] It should be noted that, in the embodiments of the present invention, the output of each stage module of the DenseNet network after the SE mechanism and channel-by-channel weighting is the input of the next stage module of the DenseNet network after the SE mechanism.

[0095] In this embodiment of the invention, the first feature map pairs of the n_way*k_shot group and the first feature map pairs of the n_way*x_query group, with a size of the first size, are processed by extracting features through the second-stage dense block of the DenseNet network. Then, channel compression and downsampling are performed through the second-stage transition layer to output the initial feature map pairs of the n_way*k_shot group and the initial feature map pairs of the n_way*x_query group, with a size of the second. The initial feature map pairs of the n_way*k_shot group and the initial feature map pairs of the n_way*x_query group, with a size of the second, are compressed into a second vector of 1×1×C1 (number of channels) by the compression module of the SE mechanism using global average pooling. Then, after passing through the first fully connected layer of the Excitation module of the SE mechanism, the vector size changes from 1×1×C1 to 1×1×C1 / R. After passing through the second fully connected layer of the Excitation module of the SE mechanism, the size changes from 1×1×C1 / R to 1×1×C1. The first vector of size 1×1×C1 after using the second fully connected layer is used to assign channel-wise weights to the initial feature map pairs of the n_way*k_shot group and the n_way*x_query group of size 2, thereby obtaining the second feature map pairs of the n_way*k_shot group and the n_way*x_query group of size 2.

[0096] For example, the first feature map pairs of the n_way*k_shot group and the first feature map pairs of the n_way*x_query group, with a size of 64×64×128, are processed by the second-stage dense block of the DenseNet network (containing 12 dense layers, increasing the number of channels to 512). Then, channel compression (to 256) and downsampling are performed through the second-stage transition layer, outputting the initial feature map pairs of the n_way*k_shot group and the initial feature map of the n_way*x_query group with a size of 32×32×256. These initial feature map pairs of the n_way*k_shot group and the initial feature map of the n_way*x_query group, with a size of 32×32×256, are compressed into a 1×1×256 second vector by the compression module of the SE mechanism (with 256 channels) using global average pooling. After passing through the first fully connected layer of the Excitation module of the SE mechanism, the vector size changes from 1×1×256 to 1×1×256 / R. Finally, it passes through the Excitation module of the SE mechanism again. The second fully connected layer of the module has its size changed from 1×1×256 / R to 1×1×256. The first vector, with a size of 1×1×256 after using the second fully connected layer, is weighted channel-wise with the initial feature map pairs n_way*k_shot and n_way*x_query, both with a size of 32×32×256, to obtain the second feature map pairs n_way*k_shot and n_way*x_query, both with a size of 32×32×256.

[0097] Then, the second feature map pairs of the n_way*k_shot group and the second feature map pairs of the n_way*x_query group, with a size of the second, are processed by the third-stage dense block of the DenseNet network to extract features. Next, channel compression and downsampling are performed through the third-stage transition layer, outputting the initial feature map pairs of the n_way*k_shot group and the initial feature map pairs of the n_way*x_query group with a size of the third. The aforementioned initial feature map pairs of the n_way*k_shot group and the initial feature map pairs of the n_way*x_query group with a size of the third are compressed into a 1×1×C1 (number of channels) second vector by the compression module of the SE mechanism using global average pooling. After passing through the first fully connected layer of the Excitation module of the SE mechanism, the vector size changes from 1×1×C1 to 1×1×C1 / R. After passing through the second fully connected layer of the Excitation module of the SE mechanism, the size changes from 1×1×C1 / R to 1×1×C1. The first vector of size 1×1×C1 after using the second fully connected layer is used to assign channel-wise weights to the initial feature map pairs of the n_way*k_shot group and the n_way*x_query group of size 3, thus obtaining the third feature map pairs of the n_way*k_shot group and the n_way*x_query group of size 3.

[0098] For example, the second feature map pairs of the n_way*k_shot group and the n_way*x_query group, with a size of 32×32×256, are processed by the third-stage dense block of the DenseNet network (containing 24 dense layers, increasing the number of channels to 1024). Then, channel compression (to 512) and downsampling are performed through the third-stage transition layer, outputting the initial feature map pairs of the n_way*k_shot group and the n_way*x_query group with a size of 16×16×512. These initial feature map pairs of the n_way*k_shot group and the n_way*x_query group, with a size of 16×16×512, are compressed into a 1×1×512 second vector using a global average pooling operation through the SE mechanism (512 channels). Finally, the vector is excitationd using the SE mechanism. After the first fully connected layer of the module, the vector size changes from 1×1×512 to 1×1×512 / R; then, after passing through the second fully connected layer of the Excitation module with the SE mechanism, the size changes from 1×1×512 / R to 1×1×512. The first vector of size 1×1×512 after the second fully connected layer is used to assign channel-wise weights to the initial feature map pairs of the n_way*k_shot group and the n_way*x_query group, both of size 16×16×512, to obtain the third feature map pairs of the n_way*k_shot group and the n_way*x_query group, both of size 16×16×512.

[0099] Finally, the third feature map pairs of the n_way*k_shot group and the third feature map pairs of the n_way*x_query group, both of size three, are processed by the fourth-stage dense block of the DenseNet network to extract features. Then, through the BatchNorm layer, the output is the initial feature map pairs of the n_way*k_shot group and the initial feature map pairs of the n_way*x_query group, both of size four. The fourth size has the same width and height as the third size, but a different number of channels. These initial feature map pairs of the n_way*k_shot group and the initial feature map pairs of the n_way*x_query group, both of size four, are compressed into a 1×1×C1 (number of channels) second vector by the SE mechanism's compression module using global average pooling. After passing through the first fully connected layer of the SE mechanism's Excitation module, the vector size changes from 1×1×C1 to 1×1×C1 / R. Finally, after passing through the second fully connected layer of the SE mechanism's Excitation module, the size changes from 1×1×C1 / R to 1×1×C1. After using the second fully connected layer, the first vector of size 1×1×C1 is used to assign channel-wise weights to the initial feature map pairs of the n_way*k_shot group and the n_way*x_query group, which are of size 4, to obtain the fourth feature map pairs of the n_way*k_shot group and the n_way*x_query group, which are of size 4.

[0100] For example, the third feature map pairs of the n_way*k_shot group and the third feature map pairs of the n_way*x_query group, with a size of 16×16×512, are processed by the fourth-stage dense block of the DenseNet network (containing 16 dense layers, increasing the number of channels to 1024). Then, through the BatchNorm layer, the initial feature map pairs of the n_way*k_shot group and the initial feature map of the n_way*x_query group, with a size of 16×16×1024, are output. These initial feature map pairs of the n_way*k_shot group and the initial feature map of the n_way*x_query group, with a size of 16×16×1024, are compressed into a 1×1×1024 second vector by the compression module of the SE mechanism (with 1024 channels) using global average pooling. After passing through the first fully connected layer of the Excitation module of the SE mechanism, the vector size changes from 1×1×1024 to 1×1×1024 / R. Finally, it passes through the Excitation module of the SE mechanism again. The second fully connected layer of the module has its size changed from 1×1×1024 / R to 1×1×1024. After using the second fully connected layer, the first vector of size 1×1×1024 is used to assign channel-wise weights to the n_way*k_shot group target feature map pairs and the n_way*x_query group target feature map pairs of size 16×16×1024, resulting in the fourth feature map pairs of n_way*k_shot group and the fourth feature map pair of n_way*x_query group of size 16×16×1024.

[0101] The DenseNet network in this embodiment of the invention, combined with the SE mechanism, can focus on more important features, thereby improving the feature network's sensitivity to target regions and its ability to capture image details.

[0102] Next, a detailed explanation will be given of "the feature map pairs of the n_way*k_shot group at each level are subtracted pixel by pixel to obtain the n_way*k_shot group of M difference feature maps, and the feature map pairs of the n_way*x_query group at each level are subtracted pixel by pixel to obtain the n_way*x_query group of M difference feature maps".

[0103] As one possible implementation, each pair of first feature maps in a group of n_way*k_shot first feature map pairs of size 1 performs pixel-wise subtraction on the two feature maps to obtain n_way*k_shot first difference feature maps; and performs pixel-wise subtraction on the two feature maps in a group of n_way*x_query first feature map pairs of size 1 to obtain n_way*x_query first feature maps; and performs pixel-wise subtraction on the two feature maps in a group of n_way*k_shot second feature map pairs of size 2 to obtain n_way*k_shot second difference feature maps; and performs pixel-wise subtraction on the two feature maps in a group of n_way*x_query second feature map pairs of size 2 to obtain n_way*x_query second difference feature maps. The following steps are performed: First, second, third, and fourth difference feature maps are obtained by subtracting the two feature maps in each of the n_way*k_shot groups of third feature map pairs (size 3), resulting in n_way*k_shot third difference feature maps. Second, the following steps are performed by subtracting the two feature maps in each of the n_way*x_query groups of third feature map pairs (size 3), resulting in n_way*x_query fourth difference feature maps. Finally, the following steps are performed by subtracting the two feature maps in each of the n_way*k_shot groups of fourth feature map pairs (size 4), resulting in n_way*x_query fourth difference feature maps. From the obtained first, second, third, and fourth difference feature maps, we know that M=4. For example, the size of the first difference feature map is 64×64×128, the size of the second difference feature map is 32×32×256, the size of the third difference feature map is 16×16×512, and the size of the fourth difference feature map is 16×16×1024.

[0104] Next, a detailed explanation will be given of "using the Adaptive Weighted Feature Pyramid Network (AWF-FPN) to fuse each group of difference feature maps in the M difference feature maps of the n_way*k_shot group to obtain n_way*k_shot fused difference feature maps, and fusing each group of difference feature maps in the M difference feature maps of the n_way*x_query group to obtain n_way*x_query fused difference feature maps".

[0105] It should be noted that the Adaptive Weighted Feature Pyramid Network (AWF-FPN) innovatively replaces the fixed addition of the traditional FPN by introducing an Adaptive Weight Generation Module (AWGM), and adopts a dual adaptive enhancement strategy of spatial and channel to accurately fuse multi-scale differential feature maps. As one possible implementation, the Adaptive Weighted Feature Pyramid Network (AWF-FPN) is used to fuse the differential feature maps of the M differential feature maps in the n_way*k_shot group to obtain n_way*k_shot fused differential feature maps. Similarly, the fusion of the differential feature maps of the M differential feature maps in the n_way*x_query group can be specifically achieved as follows: Using the AWF-FPN, and combining spatial and channel weights, the differential feature maps of the M differential feature maps in the n_way*k_shot group are fused layer by layer from the top down to obtain n_way*k_shot fused differential feature maps. Furthermore, combining spatial and channel weights, the differential feature maps of the M differential feature maps in the n_way*x_query group are fused layer by layer from the top down to obtain n_way*x_query fused differential feature maps. The result of each fusion layer is then smoothed using a 3×3 convolution to obtain the input for the next layer. The embodiments of the present invention combine spatial weights and channel weights to effectively integrate high-level semantic information and low-level spatial details, thereby improving the accuracy and stability of multi-level differential feature map fusion.

[0106] As one possible implementation, feature mapping and channel alignment are first performed, mapping the M difference feature maps of the n_way*k_shot group and the M difference feature maps of the n_way*x_query group into a unified channel, obtaining the fifth difference feature map corresponding to the M first difference feature maps of the n_way*k_shot group and the M first difference feature maps of the n_way*x_query group, the sixth difference feature map corresponding to the M second difference feature maps of the n_way*k_shot group and the M second difference feature maps of the n_way*x_query group, the seventh difference feature map corresponding to the M third difference feature maps of the n_way*k_shot group and the M third difference feature maps of the n_way*x_query group, and the eighth difference feature map corresponding to the M fourth difference feature maps of the n_way*k_shot group and the M fourth difference feature maps of the n_way*x_query group. For each group's eighth, seventh, sixth, and fifth difference feature maps, the following steps are performed: The eighth difference feature map is upsampled to the width and height dimensions of the seventh difference feature map to obtain the first upsampled feature. This first upsampled feature is concatenated with the seventh difference feature map to obtain the first concatenated feature, resulting in a richer multi-scale feature representation. Then, a spatial attention map is generated: a 1×1 convolution and a sigmoid operation are applied to the first concatenated feature to obtain the spatial attention map W. s1 ∈[0,1] 16×16×1 This indicates a preference for the first upsampled feature, with complementary weights of 1-W. s1 Then, the channel weight vector is generated: the first concatenated feature is input into two MLP layers (512→32→2) after global average pooling, and combined with the Softmax operation to obtain the channel weight vector W. c1 =[w 11 w 21 The first upsampled feature and the current layer feature (i.e., the seventh difference feature map) are respectively broadcast to form a channel weight map. Finally, two weights (spatial weight W) are used to form a channel weight map. s1 With channel weight w 11 w 21 The joint weight (W) is calculated through dot product. 11 and W 21 ): W 11 =W s1 ·w 11 W 21 =(1-W s1 )·w 21 Then, weighted fusion and smoothing are performed by multiplying the joint weights by the features and adding them together to obtain the fusion result F3. Where diff4' represents the eighth difference feature map, diff3' represents the seventh difference feature map, and Up(·) represents upsampling. The fusion result F3 is smoothed by a 3×3 convolution and ReLU (an activation function), outputting a fused difference feature map F3_smooth. The fused difference feature map F3_smooth is upsampled to the width and height dimensions of the sixth difference feature map to obtain the second upsampled feature. The second upsampled feature is concatenated with the sixth difference feature map to obtain the second concatenated feature, resulting in a richer multi-scale feature representation. Then, a 1×1 convolution and Sigmoid (an activation function) operation are applied to the second concatenated feature to obtain the spatial attention map W. s2 ∈[0,1] 32×32×1 , representing the preference for the second upsampled feature, with complementary weights of 1-W. s2 Then, the channel weight vector is generated: the second concatenated feature is input into two MLP layers (512→32→2) after global average pooling, combined with Softmax (an activation function) operation, to obtain the channel weight vector W. c2 =[w 12 w 22 The first two features correspond to the second upsampled feature and the current layer feature (i.e., the sixth difference feature map), respectively. After broadcasting, they form a channel weight map. Finally, the two weights (spatial weight W) are... s2 With channel weight w 12 w 22 The joint weight (W) is calculated through dot product. 12 and W 22 ): W 12 =W s2 ·w 12 W 22 =(1-W s2 )·w 22 Then, weighted fusion and smoothing are performed by multiplying the joint weights by the features and adding them together to obtain the fusion result F2: F2 = W 12 *Up(diff3')+W 22 *diff2'. Here, diff3' represents the seventh difference feature map, and diff2' represents the sixth difference feature map. The fusion result F2 is smoothed by a 3×3 convolution and ReLU, outputting a fused difference feature map F2_smooth. The fused difference feature map F2_smooth is upsampled to the width and height dimensions of the sixth difference feature map to obtain the third upsampled feature. The third upsampled feature is concatenated with the fifth difference feature map to obtain the third concatenated feature, resulting in a richer multi-scale feature representation. Then, a 1×1 convolution and a sigmoid operation are applied to the third concatenated feature to obtain the spatial attention map W. s3 ∈[0,1] 64×64×1 This indicates a preference for the third upsampled feature, with complementary weights of 1-W. s3Then, the channel weight vector is generated: the third concatenated feature is global average pooled and input into a two-layer MLP (Multilayer Perceptron) (512→32→2). Combined with the Softmax operation, the channel weight vector W is obtained. c3 =[w 13 w 23 The first two weights, corresponding to the third upsampled feature and the current layer feature (i.e., the fifth difference feature map), are broadcast to form a channel weight map. Finally, the two weights (spatial weight W) are... s3 With channel weight w 13 w 23 The joint weight (W) is calculated through dot product. 13 and W 23 ): W 13 =W s3 ·w 13 W 23 =(1-W s3 )·w 23 Then, weighted fusion and smoothing are performed by multiplying the joint weights by the features and adding them together to obtain the fusion result F1: F1 = W 13 *Up(diff2')+W 23 *diff1'. Where diff2' represents the sixth difference feature map and diff1' represents the fifth difference feature map. The fusion result F1 is smoothed by a 3×3 convolution and ReLU, outputting the final fused difference feature map F. final _smooth. In this embodiment of the invention, through the above steps, n_way*k_shot fusion difference feature maps and n_way*x_query fusion difference feature maps are obtained.

[0107] The following describes step 103, namely, "obtaining a prototype decision space containing n_way+1 prototype vectors based on n_way*k_shot fusion difference feature maps", in conjunction with an embodiment of the present invention.

[0108] In this embodiment of the invention, a prototype decision space is determined based on n_way*k_shot fused difference feature maps, and the prototype decision space includes n_way+1 prototype vectors.

[0109] As one possible implementation, features of all pixels in the region corresponding to the change category of each of the n_way*k_shot fusion difference feature maps and features of all pixels in the background region are extracted to obtain n_way+1 feature sets. Based on the n_way+1 feature sets, n_way+1 feature vectors are determined to constitute the prototype decision space. F kF is the feature set consisting of the features of all pixels in each of the k_shot fused difference feature maps corresponding to the k-th change category, where k is the index of the change category and f is the feature set F. k One of the features. The prototype decision space P = {P1, P2, ..., P...} k}, k∈[1, n_way+1].

[0110] This invention acquires k_shot support sets and x_query query sets corresponding to each of the n_way change categories from multiple bi-temporal remote sensing images, including n_way change categories. Then, it extracts and fuses differential features from the first bi-temporal remote sensing image in the support set to establish a prototype decision space. Next, it extracts and fuses features from the second bi-temporal remote sensing image in the query set to obtain a fused differential feature map. The cosine similarity between the fused differential feature map and each prototype vector in the prototype decision space is calculated. This invention, by combining prototype learning, improves the interpretability of the remote sensing image change detection algorithm while ensuring detection accuracy.

[0111] The following describes in detail step 104, namely, "determining the distance between each fusion difference feature map in the n_way*x_query fusion difference feature maps and each prototype vector in the prototype decision space," with reference to embodiments of the present invention.

[0112] First, it should be noted that before step 104, each of the n_way*x_query fusion difference feature maps is aligned with the dimension of each prototype vector in the prototype decision space. In this embodiment, the n_way*x_query fusion difference feature maps are denoted as F ∈ [N × C2 × H × W], where: N = n_way × x_query, which is the number of query images; C2 = 512, which is the number of FPN output channels; H and W are the dimensions of the fusion difference feature maps. Each prototype vector has a shape of [1×C2], and the fusion difference feature map F is expanded to [1 × C2 × 1 × 1] through two unsqueeze operations to facilitate distance calculation with each prototype vector.

[0113] In this embodiment of the invention, the distance between each fusion difference feature map in the n_way*x_query fusion difference feature maps and each prototype vector in the prototype decision space is determined.

[0114] As one possible implementation, the distance is determined using the following formula:

[0115]

[0116] Wherein, the simk (x, y) represents any one of the fusion difference feature maps F among n_way*x_query fusion difference feature maps. xy At pixel position (x, y) for P k cosine similarity, P k is any prototype vector in the prototype decision space, α is the scaling factor, and k∈[1, n_way+1].

[0117] In this embodiment of the invention, the change category of the second dual-temporal remote sensing image corresponding to the fused difference feature map is determined by calculating which prototype vector has the smallest distance to the fused difference feature map.

[0118] The following describes step 105, namely "determining the predicted change category of each second dual-temporal remote sensing image based on the distance", in conjunction with an embodiment of the present invention.

[0119] In this embodiment of the invention, any one of the fusion difference feature maps F from the n_way*x_query fusion difference feature maps is used. xy For P at all pixel locations k The cosine similarity values ​​are concatenated along the channels to obtain the first tensor. This first tensor is then upsampled to obtain the second tensor, which has the same size as the bi-temporal remote sensing images. This second tensor represents the response score of each pixel to the prototype vectors of all categories at the original image resolution. Based on this second tensor, the predicted change category for each second bi-temporal remote sensing image is determined. In other words, the predicted change category of the later-phase remote sensing image relative to the earlier-phase remote sensing image is known for each second bi-temporal remote sensing image.

[0120] This invention determines the predicted change category of each second bi-temporal remote sensing image by calculating the response score of each pixel to the prototype vector of all categories at the original image resolution.

[0121] The following describes in detail step 106, namely, "optimizing the feature network based on the predicted change category of each second bi-temporal remote sensing image and the label attached to each second bi-temporal remote sensing image," in conjunction with embodiments of the present invention.

[0122] In this embodiment of the invention, the feature network is optimized based on the predicted change category of each second bi-temporal remote sensing image and the label attached to each second bi-temporal remote sensing image.

[0123] It should be noted that the more identical the predicted change categories of each second bi-temporal remote sensing image are to the labels of each second bi-temporal remote sensing image, the higher the accuracy of the feature network. Conversely, the less identical the predicted change categories are to the labels of each second bi-temporal remote sensing image, the lower the accuracy.

[0124] In this embodiment of the invention, the MIOU (Mean Intersection over Union) value of the predicted change category for each determined second bi-temporal remote sensing image is calculated as an evaluation index. The following is the MIOU calculation formula.

[0125]

[0126] Where: C = n_way + 1, representing the randomly selected change category and background; TP i : The number of true instances in class i; FP i FN: The number of false positives in class i; i : The number of false negative examples in the i-th class. Where, the true positive example count is the number of instances in the second bi-temporal remote sensing image corresponding to the predicted change category of each determined second bi-temporal remote sensing image that are actually positive and are also predicted as positive. The false positive example count is the number of instances in the second bi-temporal remote sensing image corresponding to the predicted change category of each determined second bi-temporal remote sensing image that are actually negative but are predicted as positive. The false negative example count is the number of instances in the second bi-temporal remote sensing image corresponding to the predicted change category of each determined second bi-temporal remote sensing image that are actually positive but are predicted as negative. i∈[1, n_way+1].

[0127] The following describes step 107, namely "using the optimized feature network to detect changes in the dual-temporal remote sensing image of the target", in detail with reference to the embodiments of the present invention.

[0128] In this embodiment of the invention, an optimized feature network is used to detect changes in a target bi-temporal remote sensing image, which is a newly acquired bi-temporal remote sensing image. That is, the optimized feature network in this embodiment can be used to detect the type of change in the later temporal remote sensing image relative to the earlier temporal remote sensing image in a bi-temporal remote sensing image.

[0129] As one possible implementation, for a dual-temporal remote sensing image of a target with a size of the first target, an initial convolutional feature map pair with a size of the third target is extracted using the initial convolutional module of the DenseNet network. The initial convolutional feature map pair with the size of the third target is then processed by the first-stage dense block of the DenseNet network to extract features; subsequently, channel compression and downsampling are performed through the first-stage transition layer, outputting an initial feature map pair with a size of the first target. This initial feature map pair with the first target size is then compressed into a 1×1×C1 second vector using the SE mechanism's compression module and global average pooling. After passing through the first fully connected layer of the SE mechanism's Excitation module, the size of the first vector changes from 1×1×C1 to 1×1×C1 / R; it then passes through the second fully connected layer of the SE mechanism's Excitation module to obtain a 1×1×C1 first vector. The first vector is then used to assign channel-wise weights to the initial feature map pair with the size of the first target, resulting in a first feature map pair with the size of the first target. For a first feature map pair of size 1, the aforementioned steps are repeated using the second-stage dense blocks and SE mechanism of the DenseNet network to obtain a second feature map pair of size 2. For the second feature map pair of size 2, the aforementioned steps are repeated using the third-stage dense blocks and SE mechanism of the DenseNet network to obtain a third feature map pair of size 3. For the third feature map pair of size 3, the aforementioned steps are repeated using the third-stage dense blocks and SE mechanism of the DenseNet network to obtain a fourth feature map pair of size 4. The first feature map pair of size 1, the second feature map pair of size 2, the third feature map pair of size 3, and the fourth feature map pair of size 4 are subtracted pixel-wise at each level to obtain four difference feature maps. The four difference feature maps are fused using the Adaptive Weighted Feature Pyramid Network (AWF-FPN) to obtain a fused difference feature map. The cosine similarity of this fused difference feature map to each prototype vector in P' at each pixel position is calculated, where P' includes n_way prototype vectors corresponding to change categories and the background. In other words, n_way+1 cosine similarities are calculated at each pixel location. The change category index value corresponding to the largest cosine similarity at each pixel location is determined. The change category index value corresponding to the largest cosine similarity at each pixel location constitutes a change category index value map. This change category index value map is upsampled to obtain a change category index value map with a size equal to the size of the first target. The change category index value map with a size equal to the size of the first target is used to characterize the change category to which the dual-temporal remote sensing image of the target with a size equal to the first target belongs.It should be noted that if it is necessary to calculate the cosine similarity of the fused difference feature map at each pixel position to the prototype vectors corresponding to all change categories and the background, it is only necessary to select n_way=N_way when training the feature network. When using the feature network, the k_shot support sets corresponding to each change category in N_way change categories are determined.

[0130] The beneficial effects of the technical solution provided by this invention are as follows: This invention combines prototype learning with remote sensing image change detection, aiming to improve the transparency of the decision-making process for remote sensing image change detection and enhance the trust in its results. Furthermore, to address the challenge of weak feature differences in current change detection methods, a feature network (including DenseNet network, SE mechanism, and Adaptive Weighted Feature Pyramid Network AWF-FPN) is introduced into the model. This network aims to improve the feature differences between two-temporal remote sensing images, thereby enhancing the accuracy and robustness of remote sensing image change detection.

[0131] According to another embodiment, a remote sensing image change detection device is provided. Figure 2 A schematic block diagram of a remote sensing image change detection device according to one embodiment is shown. Figure 2 As shown, the device 200 may include: a remote sensing image acquisition module 201, a processing module 202, a prototype decision space acquisition module 203, a distance determination module 204, a change category prediction and determination module 205, a feature network optimization module 206, and a change detection module 207. The main functions of each component module are as follows:

[0132] The remote sensing image acquisition module 201 is used to acquire n_way*k_shot first bi-temporal remote sensing images and n_way*x_query second bi-temporal remote sensing images. The first bi-temporal remote sensing images and the second bi-temporal remote sensing images are each one of a plurality of bi-temporal remote sensing images containing n_way change categories. The n_way, N_way, k_shot, and x_query are all positive integers, and the n_way ≤ N_way. The bi-temporal remote sensing images are labeled with labels representing change categories.

[0133] Processing module 202 is used to process each first and second dual-temporal remote sensing image through a feature network to obtain n_way*k_shot and n_way*x_query fused difference feature maps;

[0134] The prototype decision space acquisition module 203 is used to obtain a prototype decision space containing n_way+1 prototype vectors based on the n_way*k_shot fusion difference feature maps;

[0135] The distance determination module 204 is used to determine the distance between each fusion difference feature map in the n_way*x_query fusion difference feature maps and each prototype vector in the prototype decision space;

[0136] The predicted change category determination module 205 is used to determine the predicted change category of each second dual-temporal remote sensing image based on the distance;

[0137] The feature network optimization module 206 is used to optimize the feature network based on the predicted change category of each second bi-temporal remote sensing image and the label attached to each second bi-temporal remote sensing image.

[0138] The change detection module 207 is used to perform change detection on the target dual-temporal remote sensing image using an optimized feature network.

[0139] In one possible implementation, the feature network includes: a DenseNet network, an SE mechanism, and an adaptive weighted feature pyramid network AWF-FPN;

[0140] The processing module 202 includes a feature pair extraction module, a pixel subtraction module, and a feature map fusion module;

[0141] The feature pair extraction module is used to extract n_way*k_shot groups of M-level feature map pairs and n_way*x_query groups of M-level feature map pairs for each first dual-temporal remote sensing image and each second dual-temporal remote sensing image through the DenseNet network and SE mechanism;

[0142] The pixel subtraction module is used to perform a pixel-by-pixel subtraction operation on the feature map pairs of the n_way*k_shot group at each level to obtain the n_way*k_shot group of M difference feature maps, and the feature map pairs of the n_way*x_query group at each level are also subjected to a pixel-by-pixel subtraction operation to obtain the n_way*x_query group of M difference feature maps.

[0143] The feature map fusion module is used to fuse each group of differential feature maps in the n_way*k_shot group of M differential feature maps using the Adaptive Weighted Feature Pyramid Network (AWF-FPN) to obtain n_way*k_shot fused differential feature maps, and to fuse each group of differential feature maps in the n_way*x_query group of M differential feature maps to obtain n_way*x_query fused differential feature maps.

[0144] In one possible implementation, the feature extraction module includes a first module, a second module, a third module, a fourth module, and a fifth module;

[0145] The first module is used to extract n_way*k_shot and n_way*x_query initial convolutional feature map pairs with a size of the second target size from the initial convolutional module of the DenseNet network for each first and second dual-temporal remote sensing image.

[0146] The second module is used to process the initial convolutional feature map pairs of n_way*k_shot and n_way*x_query respectively through the first stage module of the DenseNet network, and output the initial feature map pairs of n_way*k_shot and n_way*x_query with the first size;

[0147] The third module is used to process the initial feature map pairs of the n_way*k_shot group and the n_way*x_query group with the first size through the SE mechanism to obtain a first vector of 1×1×C1, where C1 is the number of channels;

[0148] The fourth module is used to assign weights to each channel of the initial feature maps of the n_way*k_shot group and the n_way*x_query group with the first vector as the first size, and obtain the first feature map pairs of the n_way*k_shot group and the n_way*x_query group with the first size respectively.

[0149] The fifth module is used to repeat the functions of the second to fourth modules by applying the M-1th feature map pairs of the n_way*k_shot group and the n_way*x_query group of size M-1 through the M-stage module of the DenseNet network and the SE mechanism, to obtain the Mth feature map pairs of the n_way*k_shot group and the n_way*x_query group of size M. Thus, M layers of feature map pairs of the n_way*k_shot group and the n_way*x_query group are obtained.

[0150] In one possible implementation, the feature map fusion module is specifically used to utilize the Adaptive Weighted Feature Pyramid Network (AWF-FPN), combining spatial weights and channel weights, to fuse each group of differential feature maps in the n_way*k_shot group of M differential feature maps layer by layer from the top layer down, obtaining n_way*k_shot fused differential feature maps. Furthermore, combining spatial weights and channel weights, it fuses each group of differential feature maps in the n_way*x_query group of M differential feature maps layer by layer from the top layer down, obtaining n_way*x_query fused differential feature maps. The result of each fusion layer is then smoothed using a 3×3 convolution to obtain the input for the next layer.

[0151] In one possible implementation, the prototype decision space acquisition module 203 is specifically used to extract the features of all pixels in the region corresponding to the change category of each fusion difference feature map in the n_way*k_shot fusion difference feature maps and the features of all pixels in the background region to obtain n_way+1 feature sets; and to determine the prototype decision space containing n_way+1 prototype vectors based on the n_way+1 feature sets.

[0152] In one possible implementation, the distance determination module 204 is specifically used to determine the distance according to the following formula:

[0153]

[0154] Wherein, the sim k (x, y) represents any one of the fusion difference feature maps F from n_way*x_query fusion difference feature maps. xy At pixel position (x, y) for P k cosine similarity, P k is any prototype vector in the prototype decision space, α is the scaling factor, and k∈[1, n_way+1].

[0155] In one possible implementation, the predicted change category determination module 205 is specifically used to select any one of the n_way*x_query fused difference feature maps F xy For P at all pixel locations k The cosine similarity values ​​are concatenated along the channels to obtain the first tensor; the first tensor is upsampled to obtain the second tensor, the size of which is the same as the size of the dual-temporal remote sensing image, and the second tensor represents the response score of each pixel to the prototype vector of all categories at the original image resolution; based on the second tensor, the predicted change category of each second dual-temporal remote sensing image is determined.

[0156] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0157] In addition, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0158] And an electronic device, comprising:

[0159] One or more processors; and

[0160] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0161] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0162] in, Figure 3 The architecture of an electronic device is illustrated by way of example, which may include a processor 310, a video display adapter 311, a disk drive 312, an input / output interface 313, a network interface 314, and a memory 320. The processor 310, video display adapter 311, disk drive 312, input / output interface 313, network interface 314, and memory 320 can communicate with each other via a communication bus 330.

[0163] The processor 310 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0164] The memory 320 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 320 can store the operating system 321 for controlling the operation of the electronic device 300, and the basic input / output system (BIOS) 322 for controlling the low-level operations of the electronic device 300. Additionally, it can store a web browser 323, a data storage management system 324, and a remote sensing image change detection device 325, etc. The aforementioned remote sensing image change detection device 325 can be the application program that specifically implements the aforementioned steps in this embodiment of the invention. In summary, when the technical solution provided in this embodiment of the invention is implemented through software or firmware, the relevant program code is stored in the memory 320 and is called and executed by the processor 310.

[0165] Input / output interface 313 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0166] Network interface 314 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0167] Bus 330 includes a pathway for transmitting information between various components of the device, such as processor 310, video display adapter 311, disk drive 312, input / output interface 313, network interface 314, and memory 320.

[0168] It should be noted that although the above-described device only shows the processor 310, video display adapter 311, disk drive 312, input / output interface 313, network interface 314, memory 320, bus 330, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the embodiments of the present invention have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for change detection in remote sensing images, characterized in that, The method comprises the following steps: acquiring n_way*k_shot first dual-time-phase remote sensing images and n_way*x_query second dual-time-phase remote sensing images, wherein the first dual-time-phase remote sensing images and the second dual-time-phase remote sensing images are respectively one of a plurality of dual-time-phase remote sensing images containing n_way different categories, N_way is all different categories of the dual-time-phase remote sensing images, n_way, N_way, k_shot and x_query are positive integers, n_way≤N_way, and the dual-time-phase remote sensing images are labeled to represent different categories; processing each first and second dual-time-phase remote sensing image through a feature network to obtain n_way*k_shot and n_way*x_query fusion difference feature maps; obtaining a prototype decision space containing n_way+1 prototype vectors according to the n_way*k_shot fusion difference feature maps; determining the distance between each fusion difference feature map in the n_way*x_query fusion difference feature maps and each prototype vector in the prototype decision space; determining the predicted different category of each second dual-time-phase remote sensing image according to the distance; optimizing the feature network according to the predicted different category of each second dual-time-phase remote sensing image and the label of each second dual-time-phase remote sensing image; detecting the change of a target dual-time-phase remote sensing image using the optimized feature network; wherein the feature network comprises a DenseNet network, an SE mechanism and an adaptive weighted feature pyramid network AWF-FPN; the processing of each first and second dual-time-phase remote sensing image through the feature network to obtain n_way*k_shot and n_way*x_query fusion difference feature maps comprises: extracting n_way*k_shot sets of M-level feature map pairs and n_way*x_query sets of M-level feature map pairs through the DenseNet network and the SE mechanism for each first dual-time-phase remote sensing image and each second dual-time-phase remote sensing image; performing pixel-by-pixel subtraction operation on each set of n_way*k_shot M-level feature map pairs to obtain n_way*k_shot M-difference feature maps, and performing pixel-by-pixel subtraction operation on each set of n_way*x_query M-level feature map pairs to obtain n_way*x_query M-difference feature maps; using the adaptive weighted feature pyramid network AWF-FPN to fuse each set of n_way*k_shot M-difference feature maps to obtain n_way*k_shot fusion difference feature maps, and to fuse each set of n_way*x_query M-difference feature maps to obtain n_way*x_query fusion difference feature maps.

2. The method of claim 1, wherein, The size of the first and second dual-time-phase remote sensing images is a first target size. The n_way*k_shot group M-level feature map pairs and the n_way*x_query group M-level feature map pairs are extracted for each first dual-time phase remote sensing image and each second dual-time phase remote sensing image through the DenseNet network and the SE mechanism, including: Step one, for each first and second dual-time phase remote sensing image, the n_way*k_shot and n_way*x_query initial convolution feature map pairs with a second target size are extracted through the initial convolution module of the DenseNet network; Step two, for the n_way*k_shot and n_way*x_query initial convolution feature map pairs, the first stage module of the DenseNet network is used to output the n_way*k_shot group and n_way*x_query group initial feature map pairs with a first size; Step three, the n_way*k_shot group and n_way*x_query group initial feature map pairs with the first size are subjected to the SE mechanism to obtain a first vector of 1×1×C1, wherein C1 is the number of channels; Step four, the first vector is used to assign weights to the n_way*k_shot group and n_way*x_query group initial feature map pairs with the first size respectively, and the n_way*k_shot group and n_way*x_query group first feature map pairs with the first size are obtained respectively; Step five, by analogy, for the n_way*k_shot group and n_way*x_query group M-1 feature map pairs with the M-1 size, the steps two to four are repeated through the Mth stage module of the DenseNet network and the SE mechanism to obtain the n_way*k_shot group and n_way*x_query group M feature map pairs with the M size; thus, the n_way*k_shot group and n_way*x_query group M-level feature map pairs are obtained.

3. The method of claim 1, wherein, The adaptive weighted feature pyramid network AWF-FPN is used to fuse each group of the n_way*k_shot group M difference feature maps to obtain n_way*k_shot fused difference feature maps, and each group of the n_way*x_query group M difference feature maps is fused to obtain n_way*x_query fused difference feature maps, including: The n_way*k_shot groups of M difference feature maps are fused from the top layer to the lower layer in combination with the spatial weight and the channel weight, to obtain n_way*k_shot fusion difference feature maps, and the n_way*x_query groups of M difference feature maps are fused from the top layer to the lower layer in combination with the spatial weight and the channel weight, to obtain n_way*x_query fusion difference feature maps; wherein the fusion result of each layer is smoothed by 3*3 convolution and ReLU to obtain the input of the next layer.

4. The method of claim 1, wherein, The prototype decision space containing n_way+1 prototype vectors is obtained according to the n_way*k_shot fusion difference feature maps, including: The features of all pixels in the region corresponding to the change class and the features of all pixels in the background region of each of the n_way*k_shot fusion difference feature maps are extracted, to obtain n_way+1 feature sets; The prototype decision space containing n_way+1 prototype vectors is determined according to the n_way+1 feature sets.

5. The method of claim 1, wherein, The distance between each of the n_way*x_query fusion difference feature maps and each of the prototype vectors in the prototype decision space is determined, including: Wherein, the sim k (x, y) represents any one of the n_way * x_query fusion difference feature maps F xy The cosine similarity of the pixel position (x, y) to P k The cosine similarity of the pixel position (x, y) to P k Is any one of the prototype vectors in the prototype decision space, and alpha is a scaling factor, k∈[1, n_way+1].

6. The method of claim 5, wherein, The predicted change class of each second dual-time-phase remote sensing image is determined according to the distance, including: Any one of the n_way x x_query fusion difference feature maps F xy The cosine similarity of all pixel position pairs P k is spliced according to the channel to obtain the first tensor; The first tensor is up-sampled to obtain a second tensor, the size of the second tensor is the same as that of the dual-time-phase remote sensing image, and the second tensor represents the response score of each pixel point to all prototype vectors of all classes at the original resolution; The predicted change class of each second dual-time-phase remote sensing image is determined according to the second tensor.

7. A remote sensing image change detection apparatus, characterized by, Including: The remote sensing image acquisition module is configured to acquire n_way*k_shot first dual-time-phase remote sensing images and n_way*x_query second dual-time-phase remote sensing images, the first dual-time-phase remote sensing images and the second dual-time-phase remote sensing images are one of a plurality of dual-time-phase remote sensing images containing n_way change classes, N_way is all change classes of the dual-time-phase remote sensing image, the n_way, N_way, k_shot and x_query are positive integers, the n_way is less than or equal to the N_way, and the dual-time-phase remote sensing image has a label representing a change class; The processing module is configured to process each first and second dual-time-phase remote sensing image through a feature network, to obtain n_way*k_shot and n_way*x_query fusion difference feature maps; The prototype decision space obtaining module is configured to obtain a prototype decision space containing n_way+1 prototype vectors according to the n_way*k_shot fusion difference feature maps; The distance determination module is configured to determine the distance between each of the n_way*x_query fusion difference feature maps and each of the prototype vectors in the prototype decision space. The predicted change category determination module is configured to determine the predicted change category of each of the second dual-time-phase remote sensing images according to the distance. The feature network optimization module is configured to optimize the feature network according to the determined predicted change category of each of the second dual-time-phase remote sensing images and the label carried by each of the second dual-time-phase remote sensing images. The change detection module is configured to perform change detection on the target dual-time-phase remote sensing image by using the optimized feature network. The feature network comprises a DenseNet network, an SE mechanism and an adaptive weighted feature pyramid network AWF-FPN. The processing module comprises a feature pair extraction module, a pixel subtraction module and a feature map fusion module. The feature pair extraction module is configured to extract n_way*k_shot sets of M-level feature map pairs and n_way*x_query sets of M-level feature map pairs for each of the first dual-time-phase remote sensing images and each of the second dual-time-phase remote sensing images by using the DenseNet network and the SE mechanism. The pixel subtraction module is configured to perform pixel-by-pixel subtraction on the n_way*k_shot sets of M-level feature map pairs at each level to obtain n_way*k_shot sets of M difference feature maps, and perform pixel-by-pixel subtraction on the n_way*x_query sets of M-level feature map pairs at each level to obtain n_way*x_query sets of M difference feature maps. The feature map fusion module is configured to fuse each of the n_way*k_shot sets of difference feature maps by using the adaptive weighted feature pyramid network AWF-FPN to obtain n_way*k_shot fusion difference feature maps, and fuse each of the n_way*x_query sets of difference feature maps to obtain n_way*x_query fusion difference feature maps.

8. An electronic device, comprising: It comprises: a memory and a processor, which communicate with each other through a bus; the memory stores program instructions executable by the processor, and the processor calling the program instructions can execute the method of any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-6. The computer program is executed by the processor to implement the method of any one of claims 1-6.

Images