Remote sensing image change detection method, device, equipment and medium

Abstract

The present application relates to the field of image recognition technology and discloses a remote sensing image change detection method, apparatus, device and medium, including: obtaining image change features and first depth features and second depth features at each feature size; performing fusion decoding and key feature fusion based on the image change features, first depth features and second depth features at the second feature size to obtain spatiotemporal fusion features at the second feature size; performing fusion decoding and key feature fusion based on the spatiotemporal fusion features, first depth features and second depth features at the first feature size to obtain spatiotemporal fusion features at the first feature size; performing change classification detection based on the spatiotemporal fusion features at the first feature size to obtain change detection results. The beneficial effect is that the degree of feature interaction between scales is enhanced, mutual feature interference between different scales is overcome, and the accuracy of change detection results is significantly improved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a remote sensing image change detection method, device, equipment and medium. Background Art

[0002] Remote sensing technology, a non-contact method for measuring the surface of a specific geographic area, has become a crucial tool for acquiring modern geographic information, primarily used in fields such as environmental monitoring and resource surveys. Change detection, a key technology in remote sensing tasks, has been widely applied in areas such as natural disaster assessment, urban planning, environmental surveys, and land monitoring. The basic process of change detection involves acquiring remote sensing images of a geographic area at different times and identifying differences between them to determine whether locations within the area have changed. However, the accuracy of change detection results in related technologies still needs to be improved. Summary of the Invention

[0003] The present application provides a remote sensing image change detection method, apparatus, device and medium. By fusing image features at different times and levels, the degree of feature interaction between scales is enhanced, mutual feature interference between scales is overcome, and the accuracy of change detection results is significantly improved.

[0004] In order to achieve the above objectives, the main technical solutions adopted in this application include:

[0005] In a first aspect, an embodiment of the present application provides a remote sensing image change detection method, the method comprising:

[0006] Acquire first depth features of a first remote sensing image at each characteristic size, second depth features of a second remote sensing image at each characteristic size, and image change features between the first remote sensing image and the second remote sensing image; wherein the first remote sensing image is acquired before the second remote sensing image, the characteristic sizes include at least a first characteristic size and a second characteristic size, and the first characteristic size is greater than the second characteristic size;

[0007] At the second feature size, performing fusion decoding and key feature fusion according to the image change feature, the first depth feature, and the second depth feature to obtain a spatiotemporal fusion feature at the second feature size;

[0008] On the first feature size, performing fusion decoding and key feature fusion based on the spatiotemporal fusion features, the first depth features, and the second depth features on the second feature size to obtain the spatiotemporal fusion features on the first feature size;

[0009] Change classification detection is performed based on the spatiotemporal fusion features on the first feature size to obtain a change detection result.

[0010] The remote sensing image change detection method proposed in the embodiment of the present application performs dimension-by-dimension fusion decoding and key feature fusion on the change features between the first remote sensing image and the second remote sensing image, as well as the depth features of the first remote sensing image and the second remote sensing image at multiple feature scales, thereby realizing the interaction of image features at the depth scale and time scale, enhancing the degree of interaction of image features at the spatiotemporal scale, and providing an accurate feature basis for change classification detection. Compared with related technologies, the present application overcomes the mutual interference between image features at different scales by strengthening the interaction of image features at the depth scale and time scale, improving the feature information richness and feature detail accuracy in the spatiotemporal fusion features, and thus significantly improving the accuracy of change detection results.

[0011] In a second aspect, an embodiment of the present application provides a remote sensing image change detection device, the device comprising:

[0012] a multi-layer feature extraction module, configured to obtain first depth features of a first remote sensing image at each characteristic dimension, second depth features of a second remote sensing image at each characteristic dimension, and image change features between the first remote sensing image and the second remote sensing image; wherein the first remote sensing image is acquired before the second remote sensing image, the characteristic dimensions include at least a first characteristic dimension and a second characteristic dimension, and the first characteristic dimension is greater than the second characteristic dimension;

[0013] A second feature fusion module is configured to perform fusion decoding and key feature fusion on the second feature size based on the image change feature, the first depth feature, and the second depth feature to obtain a spatiotemporal fusion feature on the second feature size;

[0014] A first feature fusion module is configured to perform fusion decoding and key feature fusion on the first feature size based on the spatiotemporal fusion features, the first depth features, and the second depth features on the second feature size to obtain the spatiotemporal fusion features on the first feature size;

[0015] The image change detection module is used to perform change classification detection based on the spatiotemporal fusion features on the first feature size to obtain a change detection result.

[0016] In a third aspect, an embodiment of the present application provides a computer device comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the method described in any one of the above embodiments by executing the computer instructions.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute any one of the methods in the above embodiments.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute any one of the methods described in the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 A diagram showing the steps of a remote sensing image change detection method provided in an embodiment of the present application;

[0021] Figure 2a This is a processing flow chart of the decoding network in an embodiment of the present application;

[0022] Figure 2b This is a schematic diagram of the first architecture of the decoding network in an embodiment of the present application;

[0023] Figure 3 A diagram showing the steps of integrating key features in an embodiment of the present application;

[0024] Figure 4 This is a schematic structural diagram of the spatial cross-scale change module in an embodiment of the present application;

[0025] Figure 5 This is a step diagram of cross attention fusion in an embodiment of the present application;

[0026] Figure 6 This is a diagram of the steps of fusion decoding in an embodiment of the present application;

[0027] Figure 7 This is a schematic diagram of the structure of the Transformer decoder in an embodiment of the present application;

[0028] Figure 8 This is a diagram showing the steps for obtaining spatiotemporal fusion features at multiple feature sizes in an embodiment of the present application;

[0029] Figure 9 This is a schematic diagram of a second architecture of a decoding network in an embodiment of the present application;

[0030] Figure 10A diagram showing the steps for obtaining image change characteristics in an embodiment of the present application;

[0031] Figure 11a This is a processing flow chart of the coding network in an embodiment of the present application;

[0032] Figure 11b Schematic diagram of the coding network architecture in an embodiment of the present application;

[0033] Figure 11c This is a schematic diagram of the structure of the CNN encoder in the embodiment of the present application;

[0034] Figure 11d This is a schematic diagram of the structure of a multi-layer perceptron in an embodiment of the present application;

[0035] Figure 12 A diagram showing the steps for obtaining the time-phase fusion feature in an embodiment of the present application;

[0036] Figure 13 This is a structural diagram of the phase difference mixing module in the embodiment of the present application;

[0037] Figure 14a This is a schematic diagram of the architecture of a complete encoding and decoding neural network in an embodiment of the present application;

[0038] Figure 14b Schematic diagram of backbone network and channel transformation in an embodiment of the present application;

[0039] Figure 15a This is a comparison chart of the change detection results obtained on the WHU-CD dataset in the embodiments of this application;

[0040] Figure 15b This is a comparison chart of the change detection results obtained on the LEVIR-CD dataset in the embodiments of this application;

[0041] Figure 16 A module diagram of a remote sensing image change detection device provided in an embodiment of the present application;

[0042] Figure 17 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] To make the purpose, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.

[0044] Remote sensing technology, a non-contact method for measuring the surface of a specific geographic area, has become a crucial tool for acquiring modern geographic information, primarily used in fields such as environmental monitoring and resource surveys. Change detection, a key technology in remote sensing, has been widely applied in areas such as natural disaster assessment, urban planning, environmental surveys, and land monitoring. The basic process of change detection involves acquiring remote sensing images of a geographic area at different times and identifying differences between them to determine whether locations within the area have changed.

[0045] In the related art, early change detection methods mainly utilized machine learning algorithms and were based on relevant experts' understanding of changes in remote sensing images and reverse change detection. With the development of computer technology, the related art mainly adopts deep learning-based change detection methods for change detection. This method is mainly implemented using an encoder-decoder architecture. In the encoder stage, a neural network is used to compress the feature map and extract the key semantic features contained therein. In the decoder stage, the feature map is dimensionalized by a neural network to obtain a change feature map. The change feature map is then input into a classifier to obtain the corresponding change mask.

[0046] However, in real-world scenarios, due to the influence of various factors, in addition to target changes, remote sensing images also contain "non-semantic changes," "background semantic changes," and "non-target semantic changes" that interfere with change detection results. "Non-semantic changes" refer to image changes in remote sensing images that are unrelated to target changes. These changes are primarily caused by conditions such as camera motion, sensor noise, or illumination changes. They do not involve actual changes in the target object and do not have a direct impact on the target object, but they can interfere with the detection of target changes. "Background semantic changes" refer to image changes in the background portion of a remote sensing image. These changes are primarily caused by changes in environmental factors, such as seasonal changes or plant growth, which can interfere with the accuracy of target change detection. "Non-target semantic changes" refer to image changes in non-target objects other than the target object in remote sensing images. Since change detection algorithms are primarily designed and optimized for target objects, these algorithms are insensitive to changes in non-target objects and are unable to accurately detect these changes, increasing the difficulty of detecting changes in non-target objects.

[0047] The change detection algorithms in related technologies are not yet able to effectively handle the aforementioned multiple types of changes and are subject to significant interference during the actual detection process, resulting in limited change detection accuracy. Furthermore, the network architecture design in related technologies is overly simple. In a change detection network with a conventional codec architecture, the encoder typically uses simple image differences and channel splicing to extract features from remote sensing images, but this method cannot fully extract deep features from remote sensing images. The decoder typically decodes the image features output by the encoder using upsampling as the primary method, but this method cannot fully utilize the spatial structural information in the image features, thereby affecting the edge integrity of the change mask in the change detection results.

[0048] Based on the above problems, the present application provides a remote sensing image change detection method, device, equipment and medium, the method including: obtaining image change features and first depth features of the first remote sensing image and second depth features of the second remote sensing image at each feature size; at the second feature size, performing fusion decoding and key feature fusion according to the image change features, the first depth features and the second depth features to obtain spatiotemporal fusion features at the second feature size; at the first feature size, performing fusion decoding and key feature fusion according to the spatiotemporal fusion features, the first depth features and the second depth features at the second feature size to obtain spatiotemporal fusion features at the first feature size; performing change classification detection based on the spatiotemporal fusion features at the first feature size to obtain change detection results.

[0049] The remote sensing image change detection method provided in the present application performs dimension-by-dimension fusion decoding and key feature fusion on the change features between the first remote sensing image and the second remote sensing image, as well as the depth features of the first remote sensing image and the second remote sensing image at multiple feature sizes, thereby realizing the interaction of image features at the depth scale and the time scale, enhancing the degree of interaction of image features at the spatiotemporal scale, and providing an accurate feature basis for change classification detection.

[0050] Compared with related technologies, this application overcomes the mutual interference between image features at different scales by strengthening the interaction of image features at depth scale and time scale, improves the feature information richness and feature detail accuracy in spatiotemporal fusion features, and thus significantly improves the accuracy of change detection results.

[0051] According to an embodiment of the present application, an embodiment of a remote sensing image change detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] In this embodiment, a remote sensing image change detection method is provided, which can be used to detect changes in remote sensing images in tasks such as natural disaster assessment, urban planning, environmental surveys, and land monitoring. Figure 1 As shown, the method includes:

[0053] S100. Obtain first depth features of the first remote sensing image at each characteristic size, second depth features of the second remote sensing image at each characteristic size, and image change features between the first remote sensing image and the second remote sensing image; wherein, the acquisition time of the first remote sensing image is before the acquisition time of the second remote sensing image, each characteristic size includes at least a first characteristic size and a second characteristic size, and the first characteristic size is larger than the second characteristic size.

[0054] S200. At the second feature size, perform fusion decoding and key feature fusion according to the image change feature, the first depth feature, and the second depth feature to obtain a spatiotemporal fusion feature at the second feature size.

[0055] S300. On the first feature size, perform fusion decoding and key feature fusion based on the spatiotemporal fusion features, the first depth features, and the second depth features on the second feature size to obtain the spatiotemporal fusion features on the first feature size.

[0056] S400. Perform change classification detection based on the spatiotemporal fusion features at the first feature size to obtain a change detection result.

[0057] Among them, the feature size can be the size of the depth feature in the corresponding level, which can be represented according to the resolution of the feature map. The more feature sizes there are, the more levels of depth feature extraction are performed on the remote sensing image, thereby being able to extract deep-level change features in the remote sensing image.

[0058] Specifically, remote sensing images are acquired for the target area requiring change detection. A first remote sensing image is acquired at a first acquisition time, and a second remote sensing image is acquired at a second acquisition time, where the first acquisition time is before the second acquisition time. The first remote sensing image and the second remote sensing image are cropped and registered so that they are of the same size and spatially aligned. Scale-by-scale feature extraction is performed on the first remote sensing image at each scale to obtain first depth features for each feature size of the first remote sensing image. Scale-by-scale feature extraction is also performed on the second remote sensing image at each scale to obtain second depth features for each feature size of the second remote sensing image. Image change features are obtained based on the image features of the first and second remote sensing images. The size of the image change features is the same as the smallest feature size among the feature sizes. It is understood that the image change features can represent the feature differences between the first and second remote sensing images for use in change detection.

[0059] Reference Figure 2a As shown, in the processing flow of the decoding network, first, according to the image change feature and the first depth feature on each feature size, a size-by-size fusion decoding is performed to obtain the first decoding feature on each feature size. It should be noted that, except for the smallest feature size, the first decoding feature on any feature size is obtained based on the first decoding feature on the previous feature size and the first depth feature on any feature size. Similarly, according to the image change feature and the second depth feature on each feature size, a size-by-size fusion decoding is performed to obtain the second decoding feature on each feature size. It should be noted that, except for the smallest feature size, the second decoding feature on any feature size is obtained based on the second decoding feature on the previous feature size and the second depth feature on any feature size.

[0060] Next, we perform a size-by-size key feature fusion on the image change features and the first and second decoded features at each feature size to obtain the spatiotemporal fusion features at each feature size. It should be noted that, except for the smallest feature size, the spatiotemporal fusion features at any feature size are derived from the spatiotemporal fusion features at the previous feature size and the first and second decoded features at that feature size. Finally, we perform change detection based on the spatiotemporal fusion features at the largest feature size to obtain the change detection result.

[0061] Reference Figure 2b In the case where the feature size includes a first feature size and a second feature size, and the first feature size is larger than the second feature size, the decoding network architecture in this embodiment is shown in the figure, where the size of the image change feature is the same as the second feature size, the size of Transformer decoder 1 is the same as the second feature size, and the size of Transformer decoder 2 is the same as the first feature size.

[0062] First, the image change feature and the first depth feature at the second feature size are input into the Transformer decoder 1 for fusion decoding to obtain the first decoded feature at the second feature size. Similarly, the image change feature and the second depth feature at the second feature size are input into the Transformer decoder 1 of another branch for fusion decoding to obtain the second decoded feature at the second feature size. It can be understood that the Transformer decoder is also provided with an upsampling module for upsampling the decoded features, increasing the resolution of the decoded features, and adapting to the latter feature size.

[0063] It should be noted that when the image change feature is obtained based on the first depth feature and the second depth feature at each feature scale, the image feature at the smallest feature size has a high similarity with the first depth feature and the second depth feature at the smallest feature size. In this case, the fusion decoding only improves the feature information richness of the first decoded feature and the second decoded feature to a limited extent. Therefore, the step of fusion decoding the image change feature, the first depth feature, and the second depth feature at the smallest feature size can be omitted. Only the image change feature can be decoded to obtain the first decoded feature and the second decoded feature at the smallest feature size.

[0064] Secondly, the image change features, the first decoding features on the second feature size, and the second decoding features are input into the spatial cross-scale change module on the second feature size, and key features are fused to obtain the spatiotemporal fusion features on the second feature size. It is understandable that an upsampling module is also provided after the spatial cross-scale change module to upsample the spatiotemporal fusion features, increase the resolution of the spatiotemporal fusion features, and adapt to the latter feature size. Compared with related technologies, the spatiotemporal fusion features fuse image features at different acquisition times, and also fuse the image features of the first remote sensing image and the second remote sensing image at different depth scales, effectively enhancing the interaction of image features at the depth scale and time scale, and improving the richness of feature information and the accuracy of feature details.

[0065] Then, similar to the second feature size, the first decoded features on the second feature size and the first depth features on the first feature size are input into the Transformer decoder 2 for fusion decoding to obtain the first decoded features on the first feature size. Similarly, the second decoded features on the second feature size and the second depth features on the first feature size are input into the Transformer decoder 2 of another branch for fusion decoding to obtain the second decoded features on the first feature size. The spatiotemporal fusion features on the second feature size and the first and second decoded features on the first feature size are input into the spatial cross-scale variation module on the first feature size, and key features are fused to obtain the spatiotemporal fusion features on the first feature size.

[0066] Finally, the spatiotemporal fusion features at the first feature scale are input into the change classification module for change detection, resulting in change detection results between the first and second remote sensing images. It can be understood that the spatiotemporal fusion features at the first feature scale incorporate image features of the first and second remote sensing images at other depth scales, further enhancing the richness of feature information and the precision of feature detail based on the spatiotemporal fusion features at the second feature scale, thereby improving the accuracy of change detection results.

[0067] The remote sensing image change detection method provided in this embodiment performs dimension-by-dimension fusion decoding and key feature fusion on the change features between the first remote sensing image and the second remote sensing image, as well as the depth features of the first remote sensing image and the second remote sensing image at multiple feature sizes, thereby realizing image feature interaction at the depth scale and the time scale, enhancing the degree of interaction of image features at the spatiotemporal scale, and providing an accurate feature basis for change classification detection.

[0068] Compared with related technologies, this application overcomes the mutual interference between image features at different scales by strengthening the interaction of image features at depth scale and time scale, improves the feature information richness and feature detail accuracy in spatiotemporal fusion features, and thus significantly improves the accuracy of change detection results.

[0069] Reference Figure 3 As shown, as an embodiment of the present application, the method of performing key feature fusion on the second feature size is the same as the method of performing key feature fusion on the first feature size; on the first feature size, key feature fusion is performed in the following manner:

[0070] S310. Perform feature fusion on the spatiotemporal fusion feature on the second feature size and the first decoding feature on the first feature size to obtain a first composite feature.

[0071] S320. Perform feature fusion on the spatiotemporal fusion feature on the second feature size and the second decoding feature on the first feature size to obtain a second composite feature.

[0072] S330. Perform cross-attention fusion based on the first composite feature and the second composite feature to obtain a spatiotemporal fusion feature at the first feature size.

[0073] Reference Figure 4As shown, this embodiment is manifested as a spatial cross-scale variation module in the decoding network architecture. Taking the spatial cross-scale variation module on the first feature size as an example, after the features are jointly input into the spatial cross-scale variation module, the spatiotemporal fusion features on the second feature size and the first decoded features on the first feature size are subjected to pixel-level image fusion to obtain a first composite feature; correspondingly, the spatiotemporal fusion features on the second feature size and the second decoded features on the first feature size are subjected to pixel-level image fusion to obtain a second composite feature. It should be noted that the first composite feature and the second composite feature are only used for key feature fusion on the current feature size and are unrelated to the key feature fusion on other feature sizes. Each feature size has its own corresponding first composite feature and second composite feature. The first composite feature fuses the image change features between the first remote sensing image and the second remote sensing image, and uses the image change features to enhance the first depth features of the first remote sensing image at the corresponding size, thereby improving the first decoded feature's ability to represent the changes between the first remote sensing image and the second remote sensing image.

[0074] Similarly, in the spatial cross-scale variation module at the second feature size, pixel-level image fusion is performed on the image variation feature and the first decoded feature at the first feature size to obtain a first composite feature at the second feature size. Correspondingly, pixel-level image fusion is performed on the image variation feature and the second decoded feature at the first feature size to obtain a second composite feature at the second feature size. It can be understood that the second composite feature also improves the second decoded feature's ability to represent the changes between the first and second remote sensing images.

[0075] Furthermore, the first composite feature and the second composite feature on the first feature size are cross-attended and fused to obtain the spatiotemporal fusion feature on the first feature size. It is understandable that each feature size corresponds to its own first composite feature and second composite feature, and the first composite feature and the second composite feature on any feature size are cross-attended and fused to obtain the spatiotemporal fusion feature on that feature size. The cross-attention mechanism can effectively enhance the correlation between image features, improve the degree of interaction between image features on the depth scale and time scale, and improve the feature information richness and feature detail accuracy in the spatiotemporal fusion feature. By fusing key features size by size on multiple feature sizes, the degree of fusion of multi-scale features is preliminarily improved, and the details of the changing features in the spatiotemporal fusion feature are enhanced, thereby improving the accuracy of the change detection results.

[0076] Reference Figure 5 As shown, as an embodiment of the present application, cross-attention fusion is performed according to the first composite feature and the second composite feature to obtain a spatiotemporal fusion feature on the first feature size, including:

[0077] S332. Evaluate the key feature weights based on the first composite feature to obtain a feature fusion weight.

[0078] S334. Extract key feature information based on the second composite feature to obtain key feature information.

[0079] S336. Perform weighted feature fusion on the key feature information according to the feature fusion weight to obtain the spatiotemporal fusion feature on the first feature size.

[0080] Specifically, the structural correlation of image features is calculated based on the first composite feature, focusing on key region features in the first remote sensing image. Feature fusion weights are then evaluated through key feature weighting. Using the first composite feature as prior information, information is extracted from the second composite feature based on key regions in the first remote sensing image to obtain key feature information. Weighted feature fusion is then performed on the key feature information based on the feature fusion weights to obtain spatiotemporal fusion features at the first feature scale, thereby achieving the fusion of the first composite feature and the second composite feature.

[0081] Reference Figure 4 As shown in the figure, during the cross-attention synthesis process, the attention vector is calculated for the first composite feature to obtain the Query vector and Key vector. After the Query vector and Key vector are shaped, the dot product is calculated on the Query vector and the Key vector, and the feature fusion weight is obtained after the softmax operation. The attention vector is calculated for the second composite feature to obtain the Value vector, which is then shaped and used as the key feature information. Based on the feature fusion weight, the Value vector, which serves as the key feature information, is subjected to weighted attention calculation. The weighted calculation result is further extracted through the convolution layer to obtain the spatiotemporal fusion feature at the corresponding feature size.

[0082] Furthermore, the form of spatiotemporal fusion features is as follows:

[0083]

[0084] in, is the spatiotemporal fusion feature output by the i-th spatial cross-scale change module; is the total number of Value vectors; is the jth Value vector, , is the number of pixels, is the number of channels; is the learnable parameter matrix; is the feature fusion weight of the j-th Value vector in the i-th spatial cross-scale change module, and its form is as follows:

[0085]

[0086] in, To scale the dot product attention, its form is as follows:

[0087]

[0088] in, is the i-th Query vector, ; is the jth Key vector, ; The number of channels of Query vector, Key vector and Value vector; and is the learnable parameter matrix.

[0089] It should be noted that, in the process of cross-attention fusion, the present application obtains the structural correlation features of the first remote sensing image according to the first composite feature, and generates corresponding feature fusion weights based on this to perform weighted feature fusion on the key feature information, and obtains the spatiotemporal fusion features on the first feature size, thereby suppressing the image background interference while enhancing the detail representation of the target object, improving the feature information richness and feature detail accuracy in the spatiotemporal fusion features, and thus significantly improving the accuracy of the change detection results.

[0090] For example, in a natural disaster assessment, the first remote sensing image can be a pre-disaster image, and the second remote sensing image can be a post-disaster image. This application can identify the area of ​​a target object (e.g., a building) based on the first remote sensing image and extract image features of the corresponding location in the second remote sensing image based on the recognition results of the first remote sensing image, thereby enabling targeted comparison and improving the accuracy of change detection.

[0091] Reference Figure 6 As shown, as an embodiment of the present application, the method of performing fusion decoding on the second feature size is the same as the method of performing fusion decoding on the first feature size, and the method of obtaining the second decoding feature is the same as the method of obtaining the first decoding feature; on the first feature size, the first decoding feature is obtained by the following method:

[0092] S302. Perform channel stitching on the first decoding feature at the second feature size and the first depth feature at the first feature size to obtain a first stitched mixed feature.

[0093] S304. Perform deep decoding on the first spliced ​​mixed feature to obtain a first decoded feature at a first feature size.

[0094] Specifically, in the branch that processes the first remote sensing image, the first depth feature is obtained by directly extracting the features of the first remote sensing image, and contains the shallow features of the first remote sensing image. The shallow features retain more spatial structure information. The first decoding feature is obtained by reconstructing the image features. Compared with the first depth feature, it has undergone more image processing and lost some spatial structure information. Through the jump connection between the first depth feature and the first decoding feature, the spatial structure features in the image features of a single phase are effectively refined, and the accuracy of the feature details of the first remote sensing image is improved. Similarly, in the branch that processes the second remote sensing image, the jump connection between the second depth feature and the second decoding feature can also improve the accuracy of the feature details of the second remote sensing image.

[0095] Furthermore, the structure in the Transformer decoder refers to Figure 7 As shown, in the branch that processes the first remote sensing image, the corresponding first decoding feature and the first depth feature are channel-spliced ​​to obtain a first spliced ​​mixed feature, which is in the following form:

[0096]

[0097] in, is the first splicing mixed feature; is the first decoding feature; is the first depth feature; Represents the concatenation calculation of the feature map along the channel.

[0098] Furthermore, in the branch that processes the first remote sensing image, operations such as image block embedding and sequence reduction are performed on the first spliced ​​mixed features in sequence, and the processed first spliced ​​mixed features are input into the multi-head attention to perform self-attention calculation on the first spliced ​​mixed features. The output of the multi-head attention is as follows:

[0099]

[0100]

[0101] in, is the scaling factor; It should be noted that in the multi-head attention of the Transformer decoder, the query vector, key vector, and value vector are all for the first concatenated mixed feature.

[0102] Furthermore, the first concatenated mixed feature after multi-head attention processing is processed by two layers of multi-layer perceptrons and one layer of deep convolution to obtain the first decoding feature, which is in the following form:

[0103]

[0104] in, Represents the multi-layer perceptron operation; Represents depthwise convolution computation.

[0105] Similarly, in the branch that processes the second remote sensing image, the second decoding feature and the second depth feature are channel-spliced ​​to obtain a second spliced ​​mixed feature, which is in the following form:

[0106]

[0107] in, is the second splicing and mixing feature; is the second decoding feature; The second deep feature is processed by image block embedding and sequence reduction, and then input into the multi-head attention to perform self-attention calculation on the second spliced ​​mixed feature. After the multi-head attention processing, the second spliced ​​mixed feature is processed by two layers of multi-layer perceptron and one layer of deep convolution to obtain the second decoding feature, which is as follows:

[0108] .

[0109] Reference Figure 8 As shown, as an embodiment of the present application, each characteristic size further includes a third characteristic size and a fourth characteristic size, the third characteristic size is smaller than the second characteristic size, and the fourth characteristic size is smaller than the third characteristic size; the method further includes:

[0110] S510. At the fourth feature size, decoding and key feature fusion are performed according to the image change features to obtain spatiotemporal fusion features at the fourth feature size.

[0111] S520. At the third feature size, perform fusion decoding and key feature fusion based on the spatiotemporal fusion features, the first depth features, and the second depth features at the fourth feature size to obtain the spatiotemporal fusion features at the third feature size.

[0112] S530. At the second feature size, perform fusion decoding and key feature fusion based on the spatiotemporal fusion features, the first depth features, and the second depth features at the third feature size to obtain the spatiotemporal fusion features at the second feature size.

[0113] S540. On the first feature size, perform fusion decoding and key feature fusion based on the spatiotemporal fusion features, the first depth features, and the second depth features on the second feature size to obtain the spatiotemporal fusion features on the first feature size.

[0114] Specifically, in this embodiment, the image change feature is derived based on the first and second depth features at various feature scales. At this point, the image change feature has a high degree of similarity to the first and second depth features at the fourth feature size, and fusion decoding only improves the feature information richness of the first and second decoded features. Therefore, the step of fusion decoding the image change feature, the first and second depth features at the fourth feature size can be omitted, and only the image change feature can be decoded.

[0115] Reference Figure 9 The decoding network architecture of this embodiment is shown in the figure, where the size of the image change feature is the same as the fourth feature size, the size of Transformer decoder 1 is the same as the fourth feature size, the size of Transformer decoder 2 is the same as the third feature size, the size of Transformer decoder 3 is the same as the second feature size, and the size of Transformer decoder 4 is the same as the first feature size. At the fourth feature size, the image change feature is input into the Transformer decoders of different branches for decoding, thereby obtaining the first decoding feature and the second decoding feature at the fourth feature size. The image change feature and the first decoding feature and the second decoding feature at the fourth feature size are then subjected to key feature fusion to obtain the spatiotemporal fusion feature at the fourth feature size.

[0116] It is understood that the image processing performed from the third feature size to the first feature size is the same as the aforementioned steps. By decoding image features at multiple feature sizes, the similarity between the decoded features and the remote sensing image is improved, thereby enhancing the change classification module's ability to capture changing features in the spatiotemporal fusion features and improving the accuracy of change detection results.

[0117] Reference Figure 10 As shown, as an embodiment of the present application, the image change feature is obtained by the following method:

[0118] S110. Perform size-by-size temporal feature enhancement fusion on the first depth features of the first remote sensing image at each characteristic size and the second depth features of the second remote sensing image at each characteristic size to obtain temporal fusion features at each characteristic size.

[0119] S120. Perform feature fusion on the time-phase fusion features at each feature size to obtain image change features.

[0120] Specifically, a first remote sensing image and a second remote sensing image are obtained, cropped, registered, and channel transformed so that the number of channels of the first remote sensing image and the second remote sensing image is changed from three channels to multiple channels, and the images are input into an encoding network.

[0121] Reference Figure 11a As shown, in the processing flow of the encoding network, first, scale-by-scale feature extraction is performed on the first remote sensing image at each scale to obtain the first depth features of the first remote sensing image at each feature size. Scale-by-scale feature extraction is performed on the second remote sensing image at each scale to obtain the second depth features of the second remote sensing image at each feature size. Secondly, at any feature size, the first depth features of the first remote sensing image and the second depth features of the second remote sensing image are subjected to temporal feature enhancement and fusion to obtain the temporal fusion features at that feature size, thereby obtaining the temporal fusion features at each feature size. Finally, feature fusion is performed on the temporal fusion features at all feature sizes to obtain the image change features between the first remote sensing image and the second remote sensing image.

[0122] Reference Figure 11b The encoding network architecture of this embodiment is shown in the figure, where the size of the image change feature is the same as the second feature size, the size of CNN encoder 1 is the same as the first feature size, the size of CNN encoder 2 is the same as the second feature size, and the sizes of other CNN encoders are similar. The structure of the CNN encoder is referenced. Figure 11c As shown, the number of feature sizes corresponding to the CNN encoder is the same as the number of feature sizes corresponding to the Transformer decoder, and the convolutional neural networks of the CNN encoders on different branches share weights. For example, when the feature size also includes a third feature size and a fourth feature size, the size of CNN encoder 3 is the same as the third feature size, and the size of CNN encoder 4 is the same as the fourth feature size.

[0123] First, the first remote sensing image is input into the CNN encoder 1 for feature extraction to obtain the first depth feature of the first remote sensing image at the first feature size. Similarly, the second remote sensing image is input into the CNN encoder 1 of another branch for feature extraction to obtain the second depth feature of the second remote sensing image at the first feature size. The first depth feature and the second depth feature at the first feature size are input into the phase difference mixing module together, and the phase feature enhancement fusion is performed to obtain the phase fusion feature at the first feature size. It can be understood that the CNN decoder is also provided with a downsampling module for downsampling the depth feature at the previous feature scale, increasing the depth of the depth feature, improving the feature representation capability of the depth feature, and adapting to the subsequent feature size.

[0124] Secondly, similar to the first feature size, the first depth feature of the first remote sensing image on the first feature size is input into the CNN encoder 2 for feature extraction to obtain the first depth feature of the first remote sensing image on the second feature size. Similarly, the second remote sensing image is input into the CNN encoder 2 of another branch for feature extraction to obtain the second depth feature of the second remote sensing image on the second feature size. The first depth feature and the second depth feature on the second feature size are input into the phase differential mixing module together, and the phase feature enhancement fusion is performed to obtain the phase fusion feature on the second feature size. In a similar manner, the phase fusion features on multiple feature sizes can be obtained until the number of phase fusion features reaches a preset threshold. It can be understood that the preset threshold of the number of phase fusion features is the same as the number of feature sizes.

[0125] Finally, the time-phase fusion features on all feature sizes are input into the multi-layer perceptron, and the time-phase fusion features on all feature sizes are fused to obtain the image change features. Figure 11d As shown in the figure, the multi-layer perceptron performs channel sorting on the time-phase fusion features at each feature size to obtain channel alignment features. Channel alignment features are multiple time-phase fusion features corresponding to each feature size. A neural network is used to perform multi-depth feature fusion on the channel alignment features. The final difference feature is obtained based on the time-phase fusion features at each feature size as the image change feature. The image change feature can be expressed as follows:

[0126]

[0127] in, is the image change feature; are the time-phase fusion features on any feature size, The preset threshold for the number of temporal fusion features.

[0128] It can be understood that as the feature size gradually decreases, the resolution of the deep features and the temporal fusion features gradually decreases, and the corresponding channel dimension gradually increases, so that the semantic representation of the remote sensing image in the deep space of the image can be extracted, so that the deep features and the temporal fusion features can represent the multi-level deep image features of the remote sensing image, and enhance the ability of the neural network to capture the key features in the remote sensing image.

[0129] Reference Figure 12 As shown, as an embodiment of the present application, the time-phase fusion feature at any feature size is obtained by the following method:

[0130] S112. Perform feature difference analysis on the first depth feature and the second depth feature at any feature size to obtain a differential feature at any feature size.

[0131] S114. Perform differential enhancement on the first depth feature at any feature size according to the differential feature to obtain a first differential enhanced feature; perform differential enhancement on the second depth feature at any feature size according to the differential feature to obtain a second differential enhanced feature.

[0132] S116. Perform feature extraction on the first differential enhancement feature and the second differential enhancement feature respectively, and perform channel mixing and splicing on the feature extraction results to obtain a time-phase fusion feature at any feature size.

[0133] Reference Figure 13 As shown, this embodiment is manifested as a time-phase differential mixing module in the encoding network architecture. For the time-phase differential mixing module on any feature size, in the feature difference stage, the first depth feature and the second depth feature on any feature size are first input into the time-phase differential mixing module, and the pixel-level image difference operation is performed on the input first depth feature and the second depth feature to obtain the differential feature on any feature size. The form of the differential feature is as follows:

[0134]

[0135] in, is the differential feature; is the first depth feature on any feature size; is a second depth feature on any feature size; Represents a pixel-level image difference operation. The differential feature can characterize the image difference between the first remote sensing image and the second remote sensing image at any feature size, representing some easily captured change features.

[0136] Secondly, pixel-level image fusion is performed on the differential feature and the first depth feature at any feature size to perform difference enhancement on the first depth feature at any feature size to obtain a first differential enhancement feature. The form of the first differential enhancement feature is as follows:

[0137]

[0138] in, is the first differential enhancement feature; Represents pixel-level image fusion calculation. Similarly, pixel-level image fusion is performed on the differential feature and the second depth feature at any feature size to perform difference enhancement on the second depth feature at any feature size to obtain a second differential enhancement feature. The second differential enhancement feature is in the following form:

[0139]

[0140] in, It should be noted that the first and second differential enhancement features are only used to obtain the time-phase fusion feature at the current feature size and have nothing to do with operations on other feature sizes. Each feature size has its own corresponding first and second differential enhancement features.

[0141] It can be understood that the first differential enhancement feature is the result of differential enhancement of the first depth feature. For the non-changing area in the first depth feature that has not changed compared to the second depth feature, the differential enhancement can increase the residual of the non-changing area, which is conducive to deeper iterative updates of the neural network. For the changing area in the first depth feature that has changed compared to the second depth feature, the differential enhancement can strengthen the representation ability of the changing area, making the changing area more prominent in the first deep feature, thereby improving the neural network's ability to capture the difference between the first remote sensing image and the second remote sensing image, and thus improving the accuracy of the change detection results. Similarly, the second differential enhancement feature has a similar effect on the second depth feature.

[0142] Next, a convolution operation is performed on the first differential enhancement feature to obtain a feature extraction result of the first differential enhancement feature through feature extraction; a convolution operation is performed on the second differential enhancement feature to obtain a feature extraction result of the second differential enhancement feature through feature extraction. In the feature mixing stage, the feature extraction results of the first differential enhancement feature and the second differential enhancement feature are respectively spliced ​​at the channel level to obtain a temporal fusion feature at any feature size, which is in the following form:

[0143]

[0144] in, is the temporal fusion feature on the i-th feature size; Indicates channel mixing and splicing; Represents a convolution operation.

[0145] It should be noted that, in this embodiment, the feature extraction results of the first differential enhancement feature and the second differential enhancement feature are mixed and spliced ​​through channels instead of directly splicing the two, thereby reducing the redundant information in the time fusion feature, effectively combining the important information in the first differential enhancement feature and the second differential enhancement feature, and improving the ability of the neural network to capture key features, thereby improving the accuracy of the change detection results.

[0146] Based on the above embodiments, the complete codec neural network architecture provided by this application is referenced as follows: Figure 14aAs shown, the input of the encoding and decoding neural network is the first remote sensing image acquired at time T1 and the second remote sensing image acquired at time T2, and time T1 is before time T2. The encoding and decoding neural network includes an encoding network and a decoding network. The encoding network and the decoding network each include two branches, corresponding to the first remote sensing image and the second remote sensing image, respectively. In this embodiment, the number of feature sizes is four, including a first feature size, a second feature size, a third feature size, and a fourth feature size that decrease in sequence. The encoding network and the decoding network in the encoding and decoding neural network both use the above feature sizes. The feature size corresponding to the CNN encoder in the encoding network gradually decreases, and the feature size corresponding to the Transformer decoder in the decoding network gradually increases.

[0147] Furthermore, the CNN encoder in the encoding network uses a convolutional neural network to extract scale-by-scale features from remote sensing images through a cascade of multiple downsampling and convolutional layers. It then fuses the extracted deep features with temporal features using a temporal differential hybrid module. Finally, a multi-layer perceptron is used to determine the image change characteristics between the first and second remote sensing images. This encoding network effectively fuses dual-temporal features at different feature sizes, enriching the information content of the image change characteristics and improving the neural network's ability to classify remote sensing image changes.

[0148] Furthermore, the decoding network, primarily based on a Transformer structure, decodes image change features and deep features of the first and second remote sensing images at different feature scales through cascaded multi-layer Transformer modules and upsampling layers. The decoding results are then fused with key features scale by scale using a spatial cross-scale change module, yielding spatiotemporal fusion features representing the difference in changes between the first and second remote sensing images for use in change detection. The decoding network effectively integrates the representation of change features at different depth scales and enhances the interaction of change features across spatial scales, thereby improving the neural network's ability to identify change features and restore spatial structure.

[0149] Furthermore, the codec neural network is connected to the change classification module, which can include a convolution block and a classification head. After the spatial cross-scale change module at the largest feature size outputs the final spatiotemporal fusion features, they are input into the change classification module, which outputs a binary change mask image as the change detection result. In the binary change mask image, white areas indicate changed areas, indicating areas in the second remote sensing image that have changed compared to the first remote sensing image.

[0150] Furthermore, the loss function of the encoding and decoding neural network in this embodiment includes binary cross entropy loss and dice loss, wherein the binary cross entropy loss is used for iteration during network parameter optimization, and its form is as follows:

[0151]

[0152] in, is the binary cross entropy loss; is the true label, is the predicted value; is the width of the remote sensing image, is the height of the remote sensing image. When the encoding and decoding neural network has problems with difficult or few training samples, such as poor learning, the binary cross entropy loss can be expanded to the Focal loss to mitigate the impact of severe imbalance in training samples on the network, thereby obtaining more accurate and complete change detection results.

[0153] The dice loss is used to maximize the similarity between the segmentation prediction results and the true labels to reflect the overlap between the predicted masks and the true labels. It has the following form:

[0154]

[0155] in, is the dice loss. Therefore, the total loss of the encoder-decoder neural network is as follows:

[0156]

[0157] in, For the total loss.

[0158] Reference Figure 14b As shown, in this embodiment, the first remote sensing image and the second remote sensing image are both three-channel color remote sensing images with a size of (w, h, 3). Before entering the encoding and decoding neural network, both are convolutionally processed through a convolution layer with a convolution kernel size of 7×7, thereby converting both from three-channel color remote sensing images into feature maps with a channel number of 32. At the same time, by setting the convolution step size, both are downsampled so that the resolution of the feature map is 1 / 4 of the color remote sensing image. At this time, the size of the feature map is (w / 4, h / 4, 32).

[0159] Furthermore, the feature map corresponding to the first remote sensing image is input into the first branch of the encoding network, and the feature map corresponding to the second remote sensing image is input into the second branch of the encoding network. In each branch, feature extraction and temporal feature enhancement fusion are performed on the two feature maps on a scale-by-scale basis, generating temporal fusion features at each scale. The feature map size in CNN encoder 1 is (w / 4, h / 4, 32), and the corresponding temporal fusion features in encoder 1 corresponding to the temporal differential mixing module are of size (w / 4, h / 4, 64). The feature map size in CNN encoder 2 is (w / 8, h / 8, 64), and the corresponding temporal fusion features in encoder 2 corresponding to the temporal differential mixing module are of size (w / 8, h / 8, 128). The feature map size in CNN encoder 3 is (w / 16, h / 16, 128), and the corresponding temporal fusion features in encoder 3 corresponding to the temporal differential mixing module are of size (w / 16, h / 16, 256). The feature map size in CNN encoder 4 is (w / 32, h / 32, 256). The corresponding time-phase fusion feature size in encoder 4 corresponding to the time-phase differential mixing module is (w / 32, h / 32, 512). A multi-layer perceptron is used to perform multi-depth feature fusion on the multiple time-phase fusion features output by encoders 1 to 4 to obtain image change features with a size of (w / 32, h / 32, 256).

[0160] Furthermore, the image variation features are fed into multiple branches of the decoding network and combined with the deep features output from the encoding network for scale-by-scale fusion decoding and key feature fusion, resulting in spatiotemporal fusion features at each feature scale. The dimensions of decoder 1, corresponding to the first-level Transformer decoder and the spatial cross-scale variation module, are both (w / 32, h / 32, 256); the dimensions of decoder 2, corresponding to the second-level Transformer decoder and the spatial cross-scale variation module, are both (w / 16, h / 16, 128); the dimensions of decoder 3, corresponding to the third-level Transformer decoder and the spatial cross-scale variation module, are both (w / 8, h / 8, 64); and the dimensions of decoder 4, corresponding to the fourth-level Transformer decoder and the spatial cross-scale variation module, are both (w / 4, h / 4, 32). It can be understood that the final spatiotemporal fusion feature output by the decoding network is of size (w / 4, h / 4, 32).

[0161] Furthermore, the final spatiotemporal fusion features are input into the classification head for change classification detection to obtain the change detection result. The change detection result can be a binary change mask image with a size of (w, h, 2).

[0162] Based on the remote sensing image change detection method and codec neural network provided in this application, the following example describes the experimental test results of this application. This example uses the WHU-CD dataset and the LEVIR-CD dataset for experimental testing. Both are public datasets widely used in the field of remote sensing image change detection. It should be noted that for the WHU-CD dataset, this example follows a widely used dataset preprocessing method, cropping the images in the dataset into non-overlapping image blocks of 256×256 size and randomly partitioning the dataset in an 8:1:1 ratio, resulting in a training set of 6096 images, a validation set of 762 images, and a test set of 762 images. For the LEVIR-CD dataset, this example also follows a widely used dataset preprocessing method, cropping the images in the dataset into non-overlapping image blocks of 256×256 size and randomly partitioning the dataset in a 7:1:2 ratio, resulting in a training set of 7120 images, a validation set of 1024 images, and a test set of 2048 images.

[0163] In this embodiment, qualitative analysis and quantitative analysis are used to compare the experimental results. Among them, qualitative analysis is to compare the change detection results obtained according to this application with the true value labels provided by the data set and the results of other methods to observe whether the mask is clear and complete, whether there is serious mis-segmentation, etc. Quantitative analysis is to compare the change detection results obtained in this application with the results of other methods using objective evaluation indicators widely used in the field of change detection. The objective evaluation indicators are calculated by a computer based on the input data and the generated binary change mask image. The objective evaluation indicators used in this embodiment include the precision (Precision) of the change category, the recall rate (Recall) of the change category, the F1 score (F1-Score) of the change category, the intersection over union (IoU) of the change category and the overall accuracy (OA) of all categories, where the precision of the change category is in the following form:

[0164]

[0165] The recall of the changing category is of the following form:

[0166]

[0167] The F1 score for the change category is in the following form:

[0168]

[0169] The form of the intersection-over-union ratio of the change categories is as follows:

[0170]

[0171] The overall accuracy for all classes is of the form:

[0172]

[0173] in, is a true positive; is a true negative; It is a false positive; It is a false negative.

[0174] In this embodiment, other change detection methods used for comparison include the FC-EF method, the FC-Siam-Conc method, the FC-Siam-Diff method, the IFN method, and the BIT method. During the experiment, an NVIDIA Telsa A40 GPU (48GB of video memory) was used for training and inference. The Adam optimizer was used to optimize network parameters, with momentum set to 0.9, weight decay set to 0.0001, initial learning rate set to 0.001, and batch size set to 8. Training was performed on the training set for 100 epochs. At the end of each 10 epochs, inference was performed on the validation set to evaluate the training performance. After training, the trained weight model was used to perform inference on the test set and calculate evaluation metrics to assess the overall experimental performance. During the training phase, data augmentation techniques widely used by other methods were also applied to the dataset, including random flipping, random scaling (0.8 to 1.2), random cropping, Gaussian blurring, and random color jittering.

[0175] Qualitative analysis results of the WHU-CD dataset refer to Figure 15a As shown, it can be seen that compared with other change detection methods, the mask in the change detection result of this embodiment is clearer and more complete, the boundary contour of the mask is neater, and there are fewer mixed mis-segmented pixels. Figure 15a This embodiment also accurately outputs the correct mask results for the first row of samples, achieving good segmentation results overall. The quantitative analysis results of the WHU-CD dataset are shown in Table 1, where P represents the precision of the change category, R represents the recall of the change category, and F1 represents the F1 score of the change category.

[0176] Table 1 Quantitative analysis results of the WHU-CD dataset

[0177]

[0178] As shown in Table 1, the experimental results of this example achieved the highest values ​​in terms of F1 score, intersection-over-union (IoU), and overall accuracy, and the second highest values ​​in terms of precision and recall, second only to the IFN method and the BIT method, respectively. Specifically, this example improved the F1 score by 0.46% and the IoU by 0.76%.

[0179] Qualitative analysis results of the LEVIR-CD dataset refer to Figure 15b As shown in the figure, compared with other change detection methods, the change detection results of this embodiment produce clearer and more complete masks, with neater edge contours. It can also accurately segment a large number of buildings, reaching an overall state-of-the-art level. The quantitative analysis results of the LEVIR-CD dataset are shown in Table 2.

[0180] Table 2 Quantitative analysis results of the LEVIR-CD dataset

[0181]

[0182] As shown in Table 2, the experimental results of this example achieved the highest values ​​in all four indicators: recall, F1 score, intersection over union (IoU), and overall accuracy, and the second highest value in precision, second only to the IFN method. Specifically, the F1 score was improved by 1.6%, and the IoU was improved by 2.66%.

[0183] Accordingly, please refer to Figure 16 , an embodiment of the present application provides a remote sensing image change detection device, the device comprising:

[0184] The multi-layer feature extraction module 1610 is used to obtain the first depth features of the first remote sensing image at each feature size, the second depth features of the second remote sensing image at each feature size, and the image change features between the first remote sensing image and the second remote sensing image; wherein the acquisition time of the first remote sensing image is before the acquisition time of the second remote sensing image, each feature size includes at least a first feature size and a second feature size, and the first feature size is larger than the second feature size.

[0185] The second feature fusion module 1620 is used to perform fusion decoding and key feature fusion on the second feature size according to the image change feature, the first depth feature and the second depth feature to obtain the spatiotemporal fusion feature on the second feature size.

[0186] The first feature fusion module 1630 is used to perform fusion decoding and key feature fusion on the first feature size based on the spatiotemporal fusion features, the first depth features, and the second depth features on the second feature size to obtain the spatiotemporal fusion features on the first feature size.

[0187] The image change detection module 1640 is configured to perform change classification detection based on the spatiotemporal fusion features at the first feature size to obtain a change detection result.

[0188] In some optional implementations, the first feature fusion module 1630 includes:

[0189] The first feature compounding unit is used to perform feature fusion on the spatiotemporal fusion feature on the second feature size and the first decoding feature on the first feature size to obtain a first compound feature.

[0190] The second feature compounding unit is used to perform feature fusion on the spatiotemporal fusion feature on the second feature size and the second decoding feature on the first feature size to obtain a second compound feature.

[0191] The cross-attention fusion unit is used to perform cross-attention fusion based on the first composite feature and the second composite feature to obtain a spatiotemporal fusion feature on the first feature size.

[0192] In some optional embodiments, the cross attention fusion unit includes:

[0193] The feature weight evaluation subunit is used to evaluate the key feature weight according to the first composite feature to obtain the feature fusion weight.

[0194] The feature information extraction subunit is used to extract key feature information according to the second composite feature to obtain key feature information.

[0195] The weighted feature fusion subunit is used to perform weighted feature fusion on key feature information according to the feature fusion weight to obtain spatiotemporal fusion features on the first feature size.

[0196] In some optional implementations, the first feature fusion module 1630 further includes a fusion decoding unit, including:

[0197] The feature stitching subunit is used to perform channel stitching on the first decoding feature on the second feature size and the first depth feature on the first feature size to obtain a first stitching mixed feature.

[0198] The feature decoding subunit is used to perform depth decoding on the first spliced ​​mixed feature to obtain a first decoded feature at a first feature size.

[0199] In some optional embodiments, the device further includes a multi-layer spatiotemporal fusion module, including:

[0200] The fourth spatiotemporal fusion unit is used to perform decoding and key feature fusion according to the image change characteristics at a fourth characteristic size to obtain spatiotemporal fusion features at the fourth characteristic size.

[0201] The third spatiotemporal fusion unit is used to perform fusion decoding and key feature fusion on the third feature size based on the spatiotemporal fusion features, the first depth features and the second depth features on the fourth feature size to obtain the spatiotemporal fusion features on the third feature size.

[0202] The second spatiotemporal fusion unit is used to perform fusion decoding and key feature fusion on the second feature size based on the spatiotemporal fusion features, the first depth features and the second depth features on the third feature size to obtain the spatiotemporal fusion features on the second feature size.

[0203] The first spatiotemporal fusion unit is used to perform fusion decoding and key feature fusion on the first feature size based on the spatiotemporal fusion features, the first depth features and the second depth features on the second feature size to obtain the spatiotemporal fusion features on the first feature size.

[0204] In some optional implementations, the multi-layer feature extraction module 1610 includes:

[0205] The temporal enhancement fusion unit is used to perform size-by-size temporal feature enhancement fusion on the first depth features of the first remote sensing image at each feature size and the second depth features of the second remote sensing image at each feature size to obtain the temporal fusion features at each feature size.

[0206] The multi-scale feature fusion unit is used to fuse the temporal fusion features at each feature size to obtain image change features.

[0207] In some optional embodiments, the phase enhancement fusion unit includes:

[0208] The feature difference analysis subunit is used to perform feature difference analysis on the first depth feature and the second depth feature at any feature size to obtain a differential feature at any feature size.

[0209] The difference feature enhancement subunit is used to perform difference enhancement on the first depth feature on any feature size according to the differential feature to obtain a first differential enhanced feature; and to perform difference enhancement on the second depth feature on any feature size according to the differential feature to obtain a second differential enhanced feature.

[0210] The feature extraction mixing subunit is used to extract the first differential enhancement feature and the second differential enhancement feature respectively, and perform channel mixing and splicing on the feature extraction results to obtain the time-phase fusion feature at any feature size.

[0211] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0212] The remote sensing image change detection device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0213] See also Figure 17 , Figure 17 1 is a structural diagram of a computer device provided by an embodiment of the present application. As shown in the figure, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 17 A processor 10 is taken as an example.

[0214] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0215] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0216] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0217] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0218] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0219] The embodiments of the present application also provide a computer-readable storage medium. The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0220] An embodiment of the present application provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method according to any embodiment of the present application.

[0221] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0222] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0223] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0224] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0225] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0226] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0227] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0228] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0229] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

[0230] Although the embodiments of the present application have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations shall fall within the scope defined by the appended claims.

Claims

1. A remote sensing image change detection method, characterized in that: The method comprises: Acquire first depth features of a first remote sensing image at each characteristic size, second depth features of a second remote sensing image at each characteristic size, and image change features between the first remote sensing image and the second remote sensing image; specifically, perform feature fusion on the temporal fusion features at each characteristic size to obtain the image change features; a method for obtaining the temporal fusion features includes: performing feature difference analysis on the first depth features and the second depth features at any characteristic size to obtain differential features at any characteristic size; performing differential enhancement on the first depth features at any characteristic size based on the differential features to obtain first differential enhancement features; performing differential enhancement on the second depth features at any characteristic size based on the differential features to obtain second differential enhancement features; performing feature extraction on the first differential enhancement features and the second differential enhancement features respectively, and performing channel mixing and splicing on the feature extraction results to obtain the temporal fusion features at any characteristic size; wherein the acquisition time of the first remote sensing image is before the acquisition time of the second remote sensing image, the respective characteristic sizes include at least a first characteristic size and a second characteristic size, and the first characteristic size is larger than the second characteristic size; At the second feature size, performing fusion decoding and key feature fusion according to the image change feature, the first depth feature, and the second depth feature to obtain a spatiotemporal fusion feature at the second feature size; On the first feature size, fusion decoding and key feature fusion are performed based on the spatiotemporal fusion feature, the first depth feature and the second depth feature on the second feature size to obtain the spatiotemporal fusion feature on the first feature size; specifically, feature fusion is performed on the spatiotemporal fusion feature on the second feature size and the first decoded feature on the first feature size to obtain a first composite feature; feature fusion is performed on the spatiotemporal fusion feature on the second feature size and the second decoded feature on the first feature size to obtain a second composite feature; cross-attention fusion is performed on the first composite feature and the second composite feature to obtain the spatiotemporal fusion feature on the first feature size; wherein the first decoded feature on the first feature size is obtained by fusion decoding based on the image change feature and the first depth feature on the first feature size, and the second decoded feature on the first feature size is obtained by fusion decoding based on the image change feature and the second depth feature on the first feature size; the cross-attention fusion includes: performing key feature weight evaluation based on the first composite feature to obtain feature fusion weight; performing key feature information extraction based on the second composite feature to obtain key feature information; and performing weighted feature fusion on the key feature information based on the feature fusion weight to obtain the spatiotemporal fusion feature on the first feature size; Change classification detection is performed based on the spatiotemporal fusion features on the first feature size to obtain a change detection result.

2. The method according to claim 1, characterized in that The manner of performing key feature fusion on the second feature size is the same as the manner of performing key feature fusion on the first feature size.

3. The method according to claim 1, characterized in that The manner of performing fusion decoding on the second feature size is the same as the manner of performing fusion decoding on the first feature size, and the manner of obtaining the second decoding feature is the same as the manner of obtaining the first decoding feature; on the first feature size, the first decoding feature is obtained by: Performing channel stitching on the first decoding feature at the second feature size and the first depth feature at the first feature size to obtain a first stitched mixed feature; The first spliced ​​mixed feature is deeply decoded to obtain a first decoded feature at the first feature size.

4. The method according to claim 1, wherein The respective characteristic sizes further include a third characteristic size and a fourth characteristic size, the third characteristic size is smaller than the second characteristic size, and the fourth characteristic size is smaller than the third characteristic size; the method further includes: At the fourth characteristic size, decoding and key feature fusion are performed according to the image change feature to obtain a spatiotemporal fusion feature at the fourth characteristic size; At the third characteristic size, performing fusion decoding and key feature fusion according to the spatiotemporal fusion feature, the first depth feature, and the second depth feature at the fourth characteristic size to obtain the spatiotemporal fusion feature at the third characteristic size; On the second feature size, performing fusion decoding and key feature fusion according to the spatiotemporal fusion features, the first depth features, and the second depth features on the third feature size to obtain spatiotemporal fusion features on the second feature size; On the first feature size, fusion decoding and key feature fusion are performed based on the spatiotemporal fusion features, the first depth features, and the second depth features on the second feature size to obtain the spatiotemporal fusion features on the first feature size.

5. A remote sensing image change detection device, characterized in that: The device comprises: A multi-layer feature extraction module is used to obtain first depth features of the first remote sensing image at each feature size, second depth features of the second remote sensing image at each feature size, and image change features between the first remote sensing image and the second remote sensing image; specifically, feature fusion is performed on the time-phase fusion features at each feature size to obtain the image change features; a method for obtaining the time-phase fusion features includes: performing feature difference analysis on the first depth features and the second depth features at any feature size to obtain differential features at any feature size; performing difference enhancement on the first depth features at any feature size based on the differential features to obtain first differential enhancement features; performing difference enhancement on the second depth features at any feature size based on the differential features to obtain second differential enhancement features; performing feature extraction on the first differential enhancement features and the second differential enhancement features respectively, and performing channel mixing and splicing on the feature extraction results to obtain the time-phase fusion features at any feature size; wherein the acquisition time of the first remote sensing image is before the acquisition time of the second remote sensing image, the various feature sizes include at least a first feature size and a second feature size, and the first feature size is larger than the second feature size; A second feature fusion module is configured to perform fusion decoding and key feature fusion on the second feature size based on the image change feature, the first depth feature, and the second depth feature to obtain a spatiotemporal fusion feature on the second feature size; The first feature fusion module is configured to perform fusion decoding and key feature fusion on the first feature size based on the spatiotemporal fusion feature, the first depth feature, and the second depth feature on the second feature size to obtain a spatiotemporal fusion feature on the first feature size. Specifically, the spatiotemporal fusion feature on the second feature size and the first decoded feature on the first feature size are fused to obtain a first composite feature. The spatiotemporal fusion feature on the second feature size and the second decoded feature on the first feature size are fused to obtain a second composite feature. The first composite feature and the second composite feature are cross-attention fusion to obtain a spatiotemporal fusion feature on the first feature size. The first decoded feature on the first feature size is obtained by fusion decoding based on the image change feature and the first depth feature on the first feature size, and the second decoded feature on the first feature size is obtained by fusion decoding based on the image change feature and the second depth feature on the first feature size. The cross-attention fusion includes: performing key feature weight evaluation based on the first composite feature to obtain a feature fusion weight; performing key feature information extraction based on the second composite feature to obtain key feature information; and performing weighted feature fusion on the key feature information based on the feature fusion weight to obtain a spatiotemporal fusion feature on the first feature size. The image change detection module is used to perform change classification detection based on the spatiotemporal fusion features on the first feature size to obtain a change detection result.

6. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 4 by executing the computer instructions.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 4.

Images