Remote sensing image change description method based on multi-scale space-time alignment and significance driving

Abstract

The invention particularly relates to a remote sensing image change description method based on multi-scale space-time alignment and saliency driving, and the method comprises the steps: carrying out the multi-scale feature extraction of a dual-time-phase remote sensing image pair, constructing a differential representation at each scale through deformable cross attention, and obtaining an image fusion feature through hierarchical fusion and Transform coding. A saliency feature map is generated through a saliency prediction head, image fusion features are decomposed into image local features and image global features based on the saliency feature map, and the image local features and the image global features are tokenized. A compact token representation is generated by dimensionally reducing the sequence of tokens based on a learnable token compressor. Based on the compact token representation, a main decoder in the double-branch decoder is used for generating change description through autoregression, and an auxiliary decoder is used for reconstructing a change difference graph; and during training, weighted joint optimization of text generation loss and reconstruction loss is adopted. The method gives consideration to global semantics and local details, suppresses false changes and improves the accuracy of change description.

Description

Technical Field

[0001] This invention relates to the field of computer vision and intelligent analysis technology of remote sensing images, specifically to a method for describing changes in remote sensing images based on multi-scale spatiotemporal alignment and saliency-driven approaches. Background Technology

[0002] Remote sensing imagery is a crucial data source in scenarios such as local security, urban management, environmental monitoring, and emergency response. Compared to natural images, remote sensing images possess higher spectral resolution and richer ground feature information, enabling precise identification and situational assessment. Against this backdrop, the Remote Sensing Image Change Captioning (RSICC) task, building upon change identification, further generates natural language text to describe the changes. Compared to traditional change detection methods that only output changed areas, this approach offers greater readability and decision support value.

[0003] However, the RSICC task faces greater challenges: it not only needs to locate changes, but also needs to understand the evolution of land cover attributes, quantities and spatial relationships, and achieve cross-modal mapping from visual features to linguistic symbols, ultimately generating grammatically coherent and logically consistent descriptive text.

[0004] Existing methods typically employ an overall paradigm of "image encoder - differential encoder - text decoder," where the encoder extracts features from dual-temporal images, the differential encoder models change information, and the decoder generates descriptive text.

[0005] However, in complex remote sensing scenarios, existing methods are still easily affected by background noise and spurious changes, and are insufficient in perceiving key local changes. At the same time, using high-dimensional feature sequences directly for generation can lead to problems such as redundant computation and unstable semantic expression.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] This invention provides a remote sensing image change description method based on multi-scale spatiotemporal alignment and saliency-driven approach, a computer-readable storage medium, and a computer program product, which can effectively overcome the defects existing in the prior art.

[0008] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0009] According to a first aspect of the present invention, a method for describing changes in remote sensing images based on multi-scale spatiotemporal alignment and saliency-driven approaches is provided, the method comprising: Multi-level feature fusion is performed on the dual-temporal remote sensing image pair to be described to obtain image fusion features; The image fusion features are input into the saliency prediction head to generate a saliency feature map of the image fusion features; and based on the saliency feature map, the image fusion features are decomposed into local image features and global image features; wherein, the saliency feature map is used to represent the saliency distribution of the image fusion features in spatial location; Local and global features of the image are tokenized and encoded to obtain a joint token sequence; then, the joint token sequence is reduced in dimensionality using a learnable token compressor to obtain a compact token representation. The compact token representation is decoded to generate a change description of the two-temporal remote sensing image pair to be described.

[0010] In some exemplary embodiments, the dual-temporal remote sensing image pair to be described includes: a first temporal remote sensing image and a second temporal remote sensing image; The process involves multi-level feature fusion of the dual-temporal remote sensing images to be described, resulting in image fusion features, including: Multi-scale encoders are used to encode the first and second time-phase remote sensing images at multiple scales, respectively, and the first multi-level features of the first time-phase remote sensing image and the second multi-level features of the second time-phase remote sensing image are extracted respectively. Based on the deformable cross-attention module, the first and second multi-level features at the same level are aligned across time, and the hierarchical difference representation at each level scale is calculated based on the alignment result. Based on the feature encoder, cross-scale semantic fusion is performed on the hierarchical difference features to obtain image fusion features.

[0011] In some exemplary embodiments, the step of decomposing the image fusion features into local image features and global image features based on the saliency feature map includes: The saliency values ​​at each spatial location in the saliency feature map are sorted to generate a sorting result; Based on the sorting results, spatial locations are selected according to a preset ratio to obtain the target area; Set the spatial location corresponding to the target area to the selected state and set the remaining spatial locations outside the target area to the masked state to construct a local mask; Set the spatial location corresponding to the target area to the masked state and set the remaining spatial locations outside the target area to the selected state to construct a global mask; By using local and global masks to perform weighted decomposition of image fusion features, local and global image features are obtained.

[0012] In some exemplary embodiments, the step of tokenizing and encoding local and global image features to obtain a joint token sequence includes: Spatial flattening is performed on local and global image features respectively to obtain corresponding local and global sequence features. Linear projections are performed on the local sequence features and the global sequence features respectively to obtain the corresponding local embedding sequence and global embedding sequence; Add positional encoding to the local embedding sequence and the global embedding sequence respectively to obtain the corresponding local token sequence and global token sequence; The global token sequence is concatenated with the local token sequence to obtain the combined token sequence.

[0013] In some exemplary embodiments, decoding the compact token representation to generate a change description of the two-temporal remote sensing image pair to be described includes: Attention modeling of the compact token representation is performed using the master decoder in the dual-branch decoder to obtain the context representation; The output words are predicted word by word based on context representation in an autoregressive manner, and the words are combined in the order of generation to form a description of the changes in the two-temporal remote sensing image pair to be described.

[0014] In some exemplary embodiments, the method further includes: The training sample set is preprocessed to obtain the preprocessed training sample set; the training sample set includes: multiple sets of sample dual-temporal remote sensing image pairs and change description annotations corresponding to each set of sample dual-temporal remote sensing image pairs; Using the feature extraction and alignment module to be trained, multi-level feature fusion is performed on the preprocessed sample dual-temporal remote sensing image pairs to obtain the sample image fusion features. The sample image fusion features are input into the saliency prediction head to be trained to generate a sample saliency feature map; and based on the sample saliency feature map, the sample image fusion features are decomposed into sample image local features and sample image global features; wherein, the sample saliency feature map is used to represent the saliency distribution of the sample image fusion features in spatial location; The local and global features of the sample image are tokenized and encoded to obtain the joint token sequence of the sample; and the joint token of the sample is reduced in dimensionality based on the learnable token compressor to be trained to obtain the compact token representation of the sample. Attention modeling of the sample compact token representation is performed using the master decoder in the dual-branch decoder to be trained, generating a predicted change description; and spatial mapping decoding of the sample compact token representation is performed using the auxiliary decoder in the dual-branch decoder to be trained, to obtain a reconstructed change difference map. An initial remote sensing image change description model is constructed based on the feature extraction and alignment module to be trained, the saliency prediction head, the learnable token compressor, and the dual-branch decoder. The initial remote sensing image change description model is trained based on the predicted change description and the reconstructed change difference map to obtain the trained target remote sensing image change description model; wherein, the target remote sensing image change description model is used to generate change descriptions for the two-temporal remote sensing image pairs to be described.

[0015] In some exemplary embodiments, the step of training the initial remote sensing image change description model based on the predicted change description and the reconstructed change difference map to obtain the trained target remote sensing image change description model includes: Calculate text generation loss based on predicted change descriptions and corresponding change description annotations; The auxiliary task loss is calculated based on the reconstructed change difference map and the labeled change difference map; wherein, the labeled change difference map is generated based on the change description labels; The text generation loss and the auxiliary task loss are weighted and fused to obtain the comprehensive loss; Backpropagation is performed on the initial remote sensing image change description model based on the comprehensive loss, and the model parameters are updated using an optimizer to obtain the trained target remote sensing image change description model.

[0016] In some exemplary embodiments, the step of performing image preprocessing on the training sample set to obtain a preprocessed training sample set includes: Adjust the sample dual-temporal remote sensing image pairs in the training sample set to the preset image size to obtain the sample dual-temporal remote sensing image pairs with adjusted size. Image enhancement processing is performed on the resized dual-temporal remote sensing image pairs to obtain the preprocessed training sample set.

[0017] According to a second aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the storage medium is located to execute the above-described method for describing remote sensing image changes based on multi-scale spatiotemporal alignment and saliency driving.

[0018] According to a third aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the above-described method for describing changes in remote sensing images based on multi-scale spatiotemporal alignment and saliency-driven approaches.

[0019] According to a fourth aspect of the present invention, an electronic device is provided, comprising: Processor; and Memory for storing the executable instructions of the processor; The processor is configured to implement the aforementioned remote sensing image change description method based on multi-scale spatiotemporal alignment and saliency-driven methods when executing the executable instructions.

[0020] The embodiments of this invention provide a remote sensing image change description method based on multi-scale spatiotemporal alignment and saliency-driven approach. First, two temporal images are input into a shared encoder to extract multi-scale features. Deformable cross-attention is introduced at each scale to achieve cross-temporal alignment, compensate for registration residuals and local displacements, and calculate the aligned difference representation. Subsequently, cross-scale semantic integration is performed through hierarchical fusion and a Transformer encoder to obtain unified fused features. To ensure the model focuses on real change areas, a saliency-driven branch is set to generate a saliency map, and a local / global mask is constructed based on this map, decomposing the fused features into local and global features. These two feature paths are further tokenized, aggregated, and compressed into a compact token representation. Finally, the compact token is input into a dual-branch decoder: the main decoder autoregressively generates change description text, and the auxiliary decoder reconstructs the change difference map. During the training phase, a weighted joint optimization of text generation loss and reconstruction loss is used to improve the accuracy and robustness of change localization and text description.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0023] Figure 1 This schematically illustrates a flowchart of a remote sensing image change description method based on multi-scale spatiotemporal alignment and saliency-driven exemplary embodiments of the present invention. Figure 2 This schematic diagram illustrates the model training process of a remote sensing image change description method based on multi-scale spatiotemporal alignment and saliency-driven exemplary embodiment of the present invention. Figure 3A This schematic diagram illustrates the inference results of Samples 1 and 2 of the remote sensing image change description method based on multi-scale spatiotemporal alignment and saliency-driven exemplary embodiment of the present invention. Figure 3B This schematic diagram illustrates the inference results of samples 3 and 4 of the remote sensing image change description method based on multi-scale spatiotemporal alignment and saliency-driven exemplary embodiment of the present invention. Figure 3CThis schematic diagram illustrates the inference results of samples 5 and 6 of the remote sensing image change description method based on multi-scale spatiotemporal alignment and saliency-driven exemplary embodiment of the present invention. Figure 4 The diagram illustrates the composition of an electronic device according to an exemplary embodiment of the present invention. Detailed Implementation

[0024] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0025] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0026] Compared with traditional remote sensing change detection tasks, remote sensing image change description (RSICC) tasks face more severe challenges, mainly in the following three aspects: (1) Higher level of semantic abstraction: Change detection only needs to focus on pixel-level difference localization, while change description needs to cross the pixel space to further understand the attribute changes, quantity increases and decreases and spatial relationship changes of ground objects. It aims to generate natural language that can accurately express the "semantic evolution" process of ground objects, and realizes a qualitative leap from "perceiving change" to "cognizing change".

[0027] (2) Cross-modal mapping is difficult: This task requires the construction of a mapping mechanism from high-dimensional visual features to discrete language symbols. It not only needs to solve the "semantic gap" between vision and language, but also needs to deal with the complex spatiotemporal dependencies in dual-phase images. The model is required to have a strong ability to align and fuse multimodal features.

[0028] (3) High requirements for the logical consistency of content generation: The output natural language text needs to comprehensively express the category, dynamic behavior and surrounding context of the changing object. The model is required to generate grammatically coherent, logically consistent and highly readable descriptive text on the basis of ensuring accurate extraction of visual features, so as to meet the actual needs of intelligent monitoring and decision support.

[0029] These three layers of characteristics make remote sensing change description tasks extremely complex in terms of model architecture design and feature learning. Therefore, developing reliable change description methods is of significant research importance. Remote sensing change description relies on the development of remote sensing change detection and image description technologies, and has thus formed a relatively mature learning paradigm in a short period of time. It mainly consists of an image encoder, a differential coding strategy, and a text decoder, as detailed below: (1) Image description techniques based on image encoders. In early work, encoders were typically composed of convolutional neural networks (CNNs), which were pre-trained on large image classification datasets to learn general image feature representations. For example, a pre-trained VGG network model was used as a feature extractor, and a Siamese network structure was used to process bi-temporal images. The Siamese network consists of two CNN branches with identical structures and shared weights, which are input as "before the change" and "after the change" images, respectively. Based on the above model, local convolutions can be introduced to enhance the changed image patches, and then a pre-trained change detection P2V-CD model is used to process the fused features, helping the model extract object-level change information.

[0030] (2) Image description techniques based on differential coding strategies. Early differential coding strategies are the most intuitive and simplest differential coding methods. The core idea is to perform differential operations directly in the original image space or shallow feature space before inputting the image into the encoder for feature extraction. Early methods include calculating the pixel-wise difference between the "changed" image and the "unchanged" image, and then using this difference map as the input to the encoder, and stitching the difference map with the original image before inputting it together. Based on this, attempts were made to add projection and position coding to the shallow feature map, and then guide the differential module to extract the changed region through cross-attention. Although this method can reduce the requirements for registration, it cannot distinguish different types of changes and is difficult to handle complex, non-rigid changes. The most widely used differential coding method in the current RSICC model is the late differential coding strategy. This strategy first allows the twin encoder to independently extract the high-level semantic features of the "unchanged" and "changed" images, and then perform fusion and differential calculation in the feature space. For example, some simple fusion methods include simple element-wise subtraction, multiplication, taking the absolute difference, and stitching.

[0031] (3) Image description techniques based on text decoders. In the early stages of remote sensing change description research, Recurrent Neural Networks (RNNs) and their improved versions, such as Long Short-Term Memory Networks and Gated Recurrent Units, were the standard choice for building decoders. RNNs, through their internal recurrent structure, can process sequence data of arbitrary length and, when generating each word, utilize hidden states to remember information from previously generated words, thereby capturing temporal dependencies in the sequence. For example, using an RNN as a decoder, the visual feature vector output by the difference extractor is first used as the initial hidden state, and then words in the descriptive sentence are generated one by one. Although RNN-based models have achieved some success in sequence modeling, they also have some inherent drawbacks. When processing long sequences, RNNs are prone to gradient vanishing or gradient exploding problems, making it difficult to capture long-distance dependencies, which may lead to inconsistencies or logical confusion in the generated sentences.

[0032] However, existing Transformer decoding strategies still have limitations when processing remote sensing imagery: on the one hand, the standard Transformer architecture tends to focus on global contextual information, and when processing large-scale remote sensing imagery, it often overlooks small but crucial local change areas, resulting in insufficiently refined descriptions; on the other hand, existing decoders lack explicit filtering mechanisms for "pseudo-changes" in visual features, easily introducing erroneous semantic descriptions caused by noise during the decoding stage. Furthermore, directly inputting high-dimensional visual tokens into the decoder leads to computational redundancy and lacks effective filtering and compression of key change information.

[0033] To address the shortcomings and deficiencies of existing technologies, this example implementation provides a remote sensing image change description method based on multi-scale spatiotemporal alignment and saliency-driven approach. (Reference) Figure 1 As shown, it can specifically include: Step S10: Perform multi-level feature fusion on the dual-temporal remote sensing image pair to be described to obtain image fusion features; Step S12: Input the image fusion features into the saliency prediction head to generate a saliency feature map of the image fusion features; and based on the saliency feature map, decompose the image fusion features into local image features and global image features; wherein, the saliency feature map is used to represent the saliency distribution of the image fusion features in spatial location; Step S14: Token serialization encoding is performed on local and global features of the image to obtain a joint token sequence; and the joint token sequence is dimensionality reduced based on a learnable token compressor to obtain a compact token representation. Step S16: Decode the compact token representation to generate a change description of the dual-temporal remote sensing image pair to be described.

[0034] Based on steps S10 to S16 above, the two temporal images are first input into a shared encoder to extract multi-scale features. Deformable cross-attention is introduced at each scale to achieve cross-temporal alignment, compensate for registration residuals and local displacements, and calculate the aligned difference representation. Subsequently, hierarchical fusion and a Transformer encoder are used for cross-scale semantic integration to obtain unified fused features. To ensure the model focuses on real-world change areas, a saliency-driven branch is set to generate a saliency map, and a local / global mask is constructed based on this map, decomposing the fused features into local and global features. These two feature paths are further tokenized, aggregated, and compressed into a compact token representation. Finally, the compact token is input into a dual-branch decoder: the main decoder generates change description text via autoregression, and the auxiliary decoder reconstructs the change difference map. During the training phase, a weighted joint optimization of text generation loss and reconstruction loss is used to improve the accuracy and robustness of change localization and text description.

[0035] The following will describe in more detail each step of a remote sensing image change description method based on multi-scale spatiotemporal alignment and saliency-driven approach in this exemplary embodiment, with reference to the accompanying drawings and embodiments.

[0036] For example, in step S10, the dual-temporal remote sensing image pair to be described includes: a first temporal remote sensing image and a second temporal remote sensing image; The process involves multi-level feature fusion of the dual-temporal remote sensing images to be described, resulting in image fusion features, including: Step S101: Use a multi-scale encoder to encode the first time-phase remote sensing image and the second time-phase remote sensing image at multiple scales respectively, and extract the first multi-level features of the first time-phase remote sensing image and the second multi-level features of the second time-phase remote sensing image respectively. In step S101, the aforementioned first-temporal remote sensing image and second-temporal remote sensing image are used to characterize the remote sensing observation results of the same geographic area at different imaging times. For example, the first-temporal remote sensing image is an image of a certain urban area acquired in June 2023, and the second-temporal remote sensing image is an image of the same area acquired in June 2024; by comparing them, it can be reflected whether there are changes in land features such as new buildings, road expansions, changes in farmland, or changes in the extent of water bodies in the area between the two temporal phases.

[0037] Step S102: Based on the deformable cross attention module, the first multi-level features and the second multi-level features at the same level are aligned across time, and the hierarchical difference representation at each level scale is calculated based on the alignment result. Step S103: Based on the feature encoder, perform cross-scale semantic fusion on the hierarchical difference features to obtain image fusion features.

[0038] In step S103, the feature encoder is a Transformer encoder. After the hierarchical difference features are uniformly adjusted to a mesoscale, they are input into the Transformer encoder to perform cross-scale semantic integration and obtain image fusion features.

[0039] Specifically, the dual-temporal remote sensing image pairs to be described Input to a multiscale encoder to extract features at different levels. .in, These represent the first and second temporal remote sensing images in the dual-temporal remote sensing image pair to be described; The first and second time-phase remote sensing images respectively represent the images at the [missing information - likely a time period or time frame]. Feature representations extracted at each hierarchical scale; Corresponding to different hierarchical scales.

[0040] Furthermore, a multi-scale encoder using the Swin-Tiny network as its backbone is employed, and the output feature maps of the first three stages of the Swin-Tiny network are selected as feature representations at different scales. ~ .in, The original image has been downsampled by 1 / 4, resulting in a resolution of 64. 64, with 96 channels; and They passed through the original Figure 1 With downsampling at 8x and 1 / 16x, the resolution is gradually halved while the number of channels doubles.

[0041] Each scale The hierarchical features below are spatiotemporally aligned using a Deformable Cross-Attention (DCA) module to obtain hierarchical difference representations at each level scale. The specific formula is as follows: (1) in, The first and second time-phase remote sensing images are respectively located at the first time phase. Feature representations extracted at each hierarchical scale; The first The height and width of the layer feature map; For the corresponding number Attention weights for each sampling point; The center coordinates of the reference point, The features are obtained by sampling at the offset positions on the second phase feature map; The first time learned from the network The spatial offset of each sampling point is used to align features.

[0042] Subsequently, the hierarchical difference representations at each scale are integrated into image fusion features in a unified semantic space via a hierarchical fusion module. In other words, the hierarchical differential representations at each scale are uniformly adjusted to a mesoscale and then input into the Transformer encoder to perform cross-scale semantic integration, thereby obtaining image fusion features. .

[0043] For example, in step S12, the decomposition of image fusion features into local image features and global image features based on the saliency feature map includes: Step S121: Sort the significance values ​​of each spatial location in the significance feature map and generate the sorting results; Step S122: Based on the sorting results, select spatial locations according to a preset ratio to obtain the target area; Step S123: Set the spatial location corresponding to the target area to the gating state and set the remaining spatial locations outside the target area to the masking state to construct a local mask; Step S124: Set the spatial location corresponding to the target area to the masked state and set the remaining spatial locations outside the target area to the selected state to construct a global mask; Step S125: Use local and global masks to perform weighted decomposition of image fusion features to obtain local and global image features.

[0044] Specifically, high-precision semantic alignment is achieved by adaptively correcting for temporal shifts and illumination differences through multi-head attention. To ensure the model focuses on truly changing regions, a saliency-driven branch is designed. A saliency map is generated using a saliency prediction head. And select the most significant one based on the threshold. Using the region as the target region, construct a local mask. The remaining areas form a global mask. Image fusion features are thus decomposed into local features. With global features .

[0045] in, The parameters representing the saliency prediction head can be set to a 1×1 convolution / linear mapping to compress the C-dimensional features at each location into a single score (logit). That is, for each spatial location... Transform its feature vector into a "significance score"; The Sigmoid function is used to map scores to... , thus obtaining the saliency plot ,therefore The larger the value, the more "significant / more likely to be a real area of ​​change / more worthy of description" the location is.

[0046] For example, for a real change (new construction): t1: an empty lot; t2: a building appears. Image fusion features in the building region will contain stable semantic differences indicating the "structure from nothing to something". The saliency prediction head will make the building region... Higher (closer to 1), lower in open space / road background areas. When selecting the Top-K% salient regions subsequently, the local mask will mainly cover the building location, guiding the model to generate a description that is more likely to be "new building / appearing building".

[0047] For pseudo-variations (shadow / lighting changes): the ground features remained unchanged in t1 and t2, but t2 showed a large area of ​​shadow due to the different sun altitude, resulting in noticeable pixel differences. Because... Alignment and semantic integration have been performed, and the shaded areas lack evidence of "semantic alteration of land features." The saliency prediction head tends to assign lower values ​​to these areas. This results in the shading being excluded or having its weight reduced in the Top-K% selection, thus reducing erroneous descriptions caused by "treating shading as change".

[0048] For example, in step S14, the token serialization encoding of local and global image features to obtain a joint token sequence includes: Step S141: Spatial flattening is performed on the local features and global features of the image respectively to obtain the corresponding local sequence features and global sequence features; Step S142: Perform linear projection on the local sequence features and global sequence features respectively to obtain the corresponding local embedding sequence and global embedding sequence; Step S143: Add position encoding to the local embedding sequence and the global embedding sequence respectively to obtain the corresponding local token sequence and global token sequence; Step S144: Concatenate the global token sequence with the local token sequence to obtain the combined token sequence.

[0049] Specifically, for global image features and local features of the image First, a flattening operation is performed in the spatial dimension, transforming it from... Transform into The sequence format is used to obtain the corresponding local and global sequence features; a linear projection layer is then used to map the feature dimensions of the local and global sequence features to a unified hidden layer dimension. This process yields the corresponding local and global embedding sequences. Then, learnable positional codes are added to both sequences to preserve spatial location information, resulting in the corresponding local token sequences. With global token sequence Finally, the processed global token sequence With local token sequence Concatenate the sequences to encode them into a combined token sequence. .

[0050] For example, in step S16, decoding the compact token representation to generate a change description of the two-temporal remote sensing image pair to be described includes: Step S161: Use the master decoder in the dual-branch decoder to perform attention modeling on the compact token representation to obtain the context representation; Step S162: Based on the context representation, predict the output words word by word in an autoregressive manner, and combine the words in the order of generation to form a change description of the two-temporal remote sensing image pair to be described.

[0051] Specifically, to reduce computational burden while preserving key information, a learnable token compressor is introduced to perform adaptive dimensionality reduction on the joint token sequence, resulting in a compact token representation. The token compression module adaptively compresses 320 tokens into 64 equal-dimensional tokens. The specific formula is as follows:

[0052] Among them, Pool( ) represents the adaptive average pooling operation, and MLP is a linear mapping layer.

[0053] Representing compact tokens As input to the main decoder (Transformer Decoder), the main decoder uses previously generated historical lexical units as conditions during the transformation description generation process, and through... Cross-attention is used to acquire visual contextual information, and then output words are predicted word by word in an autoregressive manner to generate a sequence of descriptive text. Among them, the t-th word element satisfies , For the target word to be predicted at the current time t, It represents all historical word sequences generated before time t.

[0054] For example, the method further includes: Step S20: Perform image preprocessing on the training sample set to obtain the preprocessed training sample set; wherein, the training sample set includes: multiple sets of sample dual-temporal remote sensing image pairs and change description annotations corresponding to each set of sample dual-temporal remote sensing image pairs; Step S21: Using the feature extraction and alignment module to be trained, multi-level feature fusion is performed on the preprocessed sample dual-temporal remote sensing image pairs to obtain sample image fusion features. Step S22: Input the sample image fusion features into the saliency prediction head to be trained to generate a sample saliency feature map; and based on the sample saliency feature map, decompose the sample image fusion features into sample image local features and sample image global features; wherein, the sample saliency feature map is used to represent the saliency distribution of the sample image fusion features in spatial location; Step S23: Token serialization encoding is performed on the local features and global features of the sample image to obtain the sample joint token sequence; and the sample joint token is reduced in dimensionality based on the learnable token compressor to be trained to obtain the sample compact token representation. Step S24: Use the master decoder in the dual-branch decoder to perform attention modeling on the sample compact token representation to generate a predicted change description; and use the auxiliary decoder in the dual-branch decoder to perform spatial mapping decoding on the sample compact token representation to obtain a reconstructed change difference map. In step S24, the main decoder and auxiliary decoder in the dual-branch decoder are two functionally independent but mutually reinforcing parallel branches in terms of optimization. The main decoder is a standard Transformer decoder structure, responsible for processing semantic information and decoding the compressed token into natural language text. The auxiliary decoder is a lightweight convolutional decoding network (composed of upsampling layers and convolutional layers), responsible for reconstructing a two-dimensional spatial variation map from the compressed token.

[0055] Both share the same compressed sample compact token representation. This design is intended to enforce a compact token representation for samples. It contains both the semantic information needed to generate text and the spatial structural information needed for location changes.

[0056] Therefore, the aforementioned dual-branch encoder employs a twin network architecture, where the two branches fully share weights. On one hand, sharing weights ensures that the two temporal images are mapped to the same feature space, making subsequent difference calculations mathematically meaningful. On the other hand, the twin network reduces the number of encoder parameters.

[0057] Step S25: Based on the feature extraction and alignment module to be trained, the saliency prediction head, the learnable token compressor, and the dual-branch decoder, an initial remote sensing image change description model is constructed. Step S26: The initial remote sensing image change description model is trained based on the predicted change description and the reconstructed change difference map to obtain the trained target remote sensing image change description model; wherein, the target remote sensing image change description model is used to generate change descriptions of the two-temporal remote sensing image pairs to be described.

[0058] For details, please refer to Figure 2 As shown. Figure 2 The diagram illustrates the model training process of this invention. First, the input consists of two registered bi-temporal remote sensing images, for example, both RGB images with a size of 256*256. After standardization and resizing, the two temporal images are input into the feature extraction and alignment module. The feature extraction and alignment module includes a multi-scale encoder to be trained and a deformable cross-attention mechanism. Specifically, the multi-scale encoder to be trained, using a Swin-Tiny network as its backbone, extracts feature maps L1-L3 at three levels. At each scale, the bi-temporal features are aligned using deformable cross-attention, and the sample-level difference representation is calculated. Furthermore, high-precision semantic alignment is achieved by adaptively correcting for temporal shifts and illumination differences through multi-head attention. The L1-L3 three-layer sample hierarchical differential representations are fused, uniformly adjusted to a mesoscale, and then input into the Transformer encoder to be trained. Cross-scale semantic integration is then performed to obtain fused features. .

[0059] Then, the sample images are fused with features. Input the saliency prediction head to be trained, generate sample saliency feature maps; and based on the sample saliency feature maps... Fuse features from sample images Decomposed into local features of the sample image and global features of sample images ; local features of sample images and global features of sample images The tokenization module converts the tokens into a sequence representation, resulting in the corresponding sample local token sequence. With sample global token sequence ;Will and The sequences are concatenated along the sequence dimension and encoded into a sample joint token sequence. .

[0060] Next, the sample joint token sequence is adaptively compressed using the Token Compression module (a learnable token compressor to be trained). Adaptive dimensionality reduction is performed to obtain a compact representation of the samples. For example, 320 tokens can be compressed into 64 equal-dimensional tokens.

[0061] Finally, the samples are represented in a compact manner. The input is fed into a dual-branch decoder to be trained. On one hand, it is fed into the main decoder to perform autoregressive language modeling, generating predicted change description text word by word; on the other hand, it is fed into the auxiliary decoder to reconstruct the difference map, providing spatial-level supervision signals. Based on the predicted change description and the reconstructed change difference map, the initial remote sensing image change description model is trained to obtain the trained target remote sensing image change description model.

[0062] For example, in step S26, the training of the initial remote sensing image change description model based on the predicted change description and the reconstructed change difference map to obtain the trained target remote sensing image change description model includes: Step S261: Calculate the text generation loss based on the predicted change description and the corresponding change description annotation; Step S262: Calculate the auxiliary task loss based on the reconstructed change difference map and the labeled change difference map; wherein, the labeled change difference map is generated based on the change description labels; Step S263: The text generation loss and the auxiliary task loss are weighted and fused to obtain the comprehensive loss; Step S264: Perform backpropagation on the initial remote sensing image change description model based on the comprehensive loss, and use the optimizer to update the model parameters to obtain the trained target remote sensing image change description model.

[0063] Specifically, the compressed token serves as input to the main decoder (Transformer Decoder) to generate a sequence of text describing the changes. It is then passed to the auxiliary decoder to reconstruct the variation difference map. This achieves joint supervision at both the semantic and spatial levels. The ultimate training objective is a joint loss for both tasks. (3) For the target word to be predicted at the current time t, This represents all historical word sequences generated before time t; Generate loss for the text; The hyperparameters used to balance the text generation loss and image reconstruction loss (can be set to 1.0 in experiments); This is the true value of the binary change mask, where a pixel value of 1 represents a changed region and 0 represents an unchanged region. In the change detection task, D is obtained by subtracting the two-phase images using the annotations provided by the dataset. The L2 norm (mean squared error) is used to calculate pixel-level reconstruction error. To mitigate losses in auxiliary tasks.

[0064] During the training phase, a random sampling strategy is employed. For each input image pair, at each iteration, one text is randomly selected from the corresponding five manually labeled ground truth texts as the target text Y for this training iteration to calculate the cross-entropy loss. This strategy is equivalent to data augmentation, which can prevent the model from overfitting to a specific sentence structure.

[0065] Furthermore, backpropagation is performed on the initial remote sensing image change description model based on the comprehensive loss to obtain the gradient of the model parameters; and the model parameters are updated using an optimizer; the above model training process is repeated until the preset training termination condition is met, and the trained target remote sensing image change description model is obtained.

[0066] This joint training strategy enables the model to simultaneously optimize semantic accuracy and spatial positioning capabilities, thereby generating more accurate and semantically coherent change description texts in complex contexts, achieving an organic integration of intelligent understanding of remote sensing images and language expression.

[0067] For example, in step S20, the image preprocessing of the training sample set to obtain the preprocessed training sample set includes: Step S201: Adjust the sample dual-temporal remote sensing image pairs in the training sample set to the preset image size to obtain the sample dual-temporal remote sensing image pairs after size adjustment. Step S202: Perform image enhancement processing on the resized sample dual-temporal remote sensing image pairs to obtain the preprocessed training sample set.

[0068] Specifically, the dataset used for model training is a publicly available dataset specifically designed for describing changes in remote sensing images (the Levir-CC dataset). The Levir-CC dataset is an extension of the LEVIR CCD dataset, containing 10,077 pairs of bi-temporal remote sensing images, each 256×256 pixels in size and with a spatial resolution of 0.5 meters per pixel. Each bi-temporal image pair consists of two remote sensing images of the same region and field of view (or coverage area) acquired at two different time points. These images are used as a single input to analyze the changes that occurred before and after the images. For each image pair, the dataset provides five different descriptions written by five independent annotators, generating a total of 50,385 text annotations covering various change types, including buildings, roads, sports fields, and rivers. To reduce annotation bias, image pairs containing only irrelevant changes such as lighting and shadow displacement were explicitly labeled as unchanged samples in the dataset, resulting in an approximately balanced distribution of 5,038 changed image pairs and 5,039 unchanged image pairs. At the text level, the average length of the descriptions for unchanged samples was approximately 5 words, while the descriptions for changed samples were more detailed, averaging approximately 11 words. The model was trained using a supervised method, and the dataset was divided into training, test, and validation sets in a 7:2:1 ratio.

[0069] Furthermore, the input data consists of a mixture of changed and unchanged image pairs from the training set. During training, the model randomly samples these two types of samples, aiming to enable the network to accurately describe changes while also outputting empty-set descriptions such as "The scene is the same as before" when there are no changes. Subsequently, the input data is preprocessed. All input images are uniformly resized to 256×256 pixels, and Gaussian noise is introduced during training, along with random adjustments to brightness, contrast, and saturation.

[0070] For example, according to the method provided in the embodiments of the present invention, a training and testing process for a remote sensing image change description model based on the public dataset LEVIR-CC is provided, and the effectiveness and superiority of the method in the change description generation task are verified through comparative experiments.

[0071] All experiments in this invention were conducted on a server equipped with an Intel(R) Xeon(R) Silver 4210R CPU and a 24GB NVIDIA GeForce RTX 4090 GPU. The system environment was Ubuntu 22.04.2. The algorithm was implemented in Python, relying on the PyTorch library, specifically Python version 3.9.19 and PyTorch version 1.13.1. The training and testing data used in the experiments were from the publicly available LEVIR-CC dataset.

[0072] To objectively evaluate the model performance, the following evaluation metrics are used in this embodiment: 1) BLEU-N(B-1,B-2,B-3,B-4): Used to evaluate the accuracy of N-gram overlap between the generated text and the reference text. The higher the score, the higher the accuracy of local phrases.

[0073] 2) METEOR: An evaluation metric based on matching single characters with synonyms, which takes into account both accuracy and recall, and can better reflect the consistency with human judgment.

[0074] 3) ROUGE-L: A statistical metric based on the longest common subsequence, which mainly focuses on the recall ability of the generated text to the sentence structure in the reference text.

[0075] 4) CIDEr-D: A metric specifically designed for image description tasks. It calculates consistency through weighted TF-IDF and can effectively measure the coverage of key information in semantic content of generated descriptions.

[0076] Meanwhile, the following mainstream comparison algorithms were selected for experimental verification: 1) DUDA: A recurrent neural network model based on dual attention mechanism, it was an early benchmark method.

[0077] 2) MCCFormers-S / D: Based on a variant of the Transformer architecture, this study explores feature extraction strategies for single-branch and dual-branch architectures.

[0078] 3) Chg2Cap: Introduces an additional change detection auxiliary task to guide the generation of the description network.

[0079] 4) SEN / SFEN: This method focuses on using Siamese networks to enhance the expression of feature differences.

[0080] 5) RSICCformer: A relatively advanced full Transformer architecture that uses self-attention mechanism to handle long-distance dependencies. 6) PSNet: Progressive Scale Aware Network, which improves descriptive performance through multi-scale feature fusion.

[0081] The experiment is as follows: Following the steps outlined in the specific implementation, the results of this network model are shown in the table below.

[0082] Table 1. AUC comparison results of this invention on all datasets.

[0083] Table 1 presents the quantitative comparison results of the proposed method with other methods on all datasets, with bold text representing the best results. As can be seen from Table 1, our invention (Ours) achieves significant improvements across all metrics. In particular, the CIDEr-D metric, reflecting semantic consistency, reaches 138.36, an improvement of approximately 1.75 compared to the second-best Chg2Cap and 5.49 compared to PSNet. This indicates that the descriptive text generated by our invention is semantically closest to human annotation and can more accurately capture key change information. Furthermore, the high scores in BLEU-4 and METEOR also demonstrate the advantages of the generated sentences in terms of grammatical structure and lexical accuracy.

[0084] Further, refer to Figures 3A-3B As shown. Figures 3A-3B The image shows a comparison of the inference results of this invention and the comparison algorithm on the LEVIR-CC dataset. In the image, words that were predicted incorrectly are marked in red, while words that were accurately and in detail are marked in green.

[0085] Figure 3A The image shows the change descriptions for two samples (1 and 2). Each sample contains a pair of bi-temporal remote sensing images (left and right corresponding to before and after the change, respectively) to observe scene changes (such as vegetation reduction, new roads / buildings, etc.).

[0086] The second half (textual description comparison): provides line-by-line descriptions of the changes generated for the same image pair using different methods, including GT (ground truth annotation), RSICCFormer, PromptCC, PSNet, Chg2Cap, and Ours (this invention).

[0087] Color coding meaning: In the image, red marks segments in the generation description that are inaccurate or inconsistent with the actual changes, while green marks key information that is more consistent with the actual changes.

[0088] For Sample 1: GT expressed "some trees were removed and new buildings appeared", however, the comparison method described the location / scene of the building as "in the woods" or the location was not accurately described; the present invention describes "a small house appeared on bare land and some trees were removed", which is closer to the key points of GT.

[0089] Regarding Sample 2: GT expresses "two new longitudinal roads and rows of villas replacing vegetation", while the comparison method only says "vegetation removal + one road / intersection", which is incomplete; this invention emphasizes "two longitudinal roads + rows of buildings replacing vegetation", which is more consistent with GT.

[0090] Figure 3B The results show the changes in two samples (3 and 4).

[0091] For Sample 3: GT is "The scene is the same as before (no change)". PromptCC, PSNet, and Ours all correctly output "no change"; while RSICCFormer and Chg2Cap misjudged it as "Some houses have been built along the road" (marked in red).

[0092] For Sample 4: GT is "A new building appears next to the original building". Ours describes it as "A new (built) house appears near the original house", and emphasizes "near the original house" in green, which is more consistent with GT; the other comparison methods either have inaccurate location descriptions (such as "A building appears at the edge of the scene"), or directly miss detection by saying "The scene is the same", or describe the change as "A house is built on the right side of the road", etc.

[0093] Figure 3C The image shows the descriptions of the changes in two samples (5 and 6). For Sample 5: GT describes it as: part of the woodland (vegetation) becomes a road, and houses appear on both sides of the road. While some comparative methods can say "roads / houses appear," others misdescribe the scene as desert / bareland, which is semantically inaccurate. Ours describes it as "the road and houses on both sides of the road replace part of the vegetation," which is closer to the essential change of "vegetation being replaced by construction land."

[0094] For Sample 6: GT describes it as: A row of detached houses has been added next to the grassland / pasture, distributed along the road. The comparison method only says "a road has appeared" or generalizes it to "many villas / houses", lacking the key information of "a row of houses appearing along the road". Ours describes it as "detached houses appearing along the road in the lower right area", which is a more specific description of the location and type of the object.

[0095] Therefore, it can be seen that, Figure 3A and Figure 3CAs shown, in complex scenarios such as Sample 2 and Sample 5, the comparison method is prone to false detections of changed objects. For example, RSICCFormer misreports roads as "residential area" and has some missed detections. In contrast, this invention can accurately locate changes in "vertical roads" and "vegetation". This is due to the saliency-driven mechanism's ability to focus on key areas and the spatiotemporal alignment module's effective suppression of pseudo-changes, making the model generate descriptions that are more consistent with the real visual scene and have clearer semantic logic when facing subtle or complex changes.

[0096] The beneficial effects of this invention are as follows: (1) Suppressing pseudo-changes and improving robustness: By accurately modeling cross-temporal spatial differences through multi-scale spatiotemporal alignment (deformable attention module), the background noise and pseudo-change response caused by registration error, illumination difference, etc. can be effectively reduced, thereby improving the stability and robustness of the change region discrimination.

[0097] (2) Focus on key change areas while taking into account the global context: The saliency-driven mechanism guides the model to focus on areas of significant change while retaining global context information, so that the generated description can cover key local details without losing the semantic consistency of the scene.

[0098] (3) Reduce computational burden and maintain key information: Structured feature compression and dynamic adaptation are achieved through learnable token compression (e.g., 320 tokens are adaptively compressed into 64 equal-dimensional tokens), which reduces the amount of decoding computation while maintaining the key information of change.

[0099] (4) Joint supervision improves accuracy and interpretability: The dual-branch decoders work together. The main decoder generates a natural language change description, and the auxiliary decoder reconstructs the change difference map to provide spatial supervision, promote spatial feature learning, and improve the accuracy, interpretability and overall robustness of the generated text.

[0100] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0101] It should be noted that although several modules or units of the device for performing actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0102] Figure 4 A schematic diagram of an electronic device suitable for implementing embodiments of the present invention is shown.

[0103] It should be noted that, Figure 4 The electronic device 1000 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0104] like Figure 4 As shown, the electronic device 1000 includes a Central Processing Unit (CPU) 1001, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1002 or programs loaded from storage section 1008 into Random Access Memory (RAM) 1003. The RAM 1003 also stores various programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An Input / Output (I / O) interface 1005 is also connected to the bus 1004. Furthermore, the electronic device 1000 also includes an FPGA device and a System-on-a-Chip (SoC) device.

[0105] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1010 as needed so that computer programs read from them can be installed into storage section 1008 as needed.

[0106] In particular, according to embodiments of the present invention, the processes described below with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs various functions defined in the system of this application.

[0107] Specifically, the aforementioned electronic devices can be airborne intelligent electronic devices.

[0108] It should be noted that the storage medium shown in the embodiments of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, wherein computer-readable program code is carried. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0110] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0111] It should be noted that, as another aspect, this application also provides a storage medium, which may be included in an electronic device or may exist independently without being assembled into the electronic device. The aforementioned storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to perform the methods described in the following embodiments. For example, the electronic device may perform... Figure 1 The steps of the method shown.

[0112] In one embodiment, this application provides a computer program product including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0113] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0114] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0115] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for describing changes in remote sensing images based on multi-scale spatiotemporal alignment and saliency-driven approaches, characterized in that, The method includes: Multi-level feature fusion is performed on the dual-temporal remote sensing image pair to be described to obtain image fusion features; The image fusion features are input into the saliency prediction head to generate a saliency feature map of the image fusion features; and based on the saliency feature map, the image fusion features are decomposed into local image features and global image features; wherein, the saliency feature map is used to represent the saliency distribution of the image fusion features in spatial location; Local and global features of the image are tokenized and encoded to obtain a joint token sequence; then, the joint token sequence is reduced in dimensionality using a learnable token compressor to obtain a compact token representation. The compact token representation is decoded to generate a change description of the two-temporal remote sensing image pair to be described.

2. The method according to claim 1, characterized in that, The dual-temporal remote sensing image pair to be described includes: a first-temporal remote sensing image and a second-temporal remote sensing image; The process involves multi-level feature fusion of the dual-temporal remote sensing images to be described, resulting in image fusion features, including: Multi-scale encoders are used to encode the first and second time-phase remote sensing images at multiple scales, respectively, and the first multi-level features of the first time-phase remote sensing image and the second multi-level features of the second time-phase remote sensing image are extracted respectively. Based on the deformable cross-attention module, the first and second multi-level features at the same level are aligned across time, and the hierarchical difference representation at each level scale is calculated based on the alignment result. Based on the feature encoder, cross-scale semantic fusion is performed on the hierarchical difference features to obtain image fusion features.

3. The method according to claim 1, characterized in that, The method of decomposing image fusion features into local image features and global image features based on salient feature maps includes: The saliency values ​​at each spatial location in the saliency feature map are sorted to generate a sorting result; Based on the sorting results, spatial locations are selected according to a preset ratio to obtain the target area; Set the spatial location corresponding to the target area to the selected state and set the remaining spatial locations outside the target area to the masked state to construct a local mask; Set the spatial location corresponding to the target area to the masked state and set the remaining spatial locations outside the target area to the selected state to construct a global mask; By using local and global masks to perform weighted decomposition of image fusion features, local and global image features are obtained.

4. The method according to claim 1, characterized in that, The step of tokenizing and encoding local and global image features to obtain a joint token sequence includes: Spatial flattening is performed on local and global image features respectively to obtain corresponding local and global sequence features. Linear projections are performed on the local sequence features and the global sequence features respectively to obtain the corresponding local embedding sequence and global embedding sequence; Add positional encoding to the local embedding sequence and the global embedding sequence respectively to obtain the corresponding local token sequence and global token sequence; The global token sequence is concatenated with the local token sequence to obtain the combined token sequence.

5. The method according to claim 1, characterized in that, Decoding the compact token representation to generate a change description of the two-temporal remote sensing image pair to be described includes: Attention modeling of the compact token representation is performed using the master decoder in the dual-branch decoder to obtain the context representation; The output words are predicted word by word based on context representation in an autoregressive manner, and the words are combined in the order of generation to form a description of the changes in the two-temporal remote sensing image pair to be described.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The training sample set is preprocessed to obtain the preprocessed training sample set; the training sample set includes: multiple sets of sample dual-temporal remote sensing image pairs and corresponding change description labels for each set of sample dual-temporal remote sensing image pairs; Using the feature extraction and alignment module to be trained, multi-level feature fusion is performed on the preprocessed sample dual-temporal remote sensing image pairs to obtain the sample image fusion features. The sample image fusion features are input into the saliency prediction head to be trained to generate a sample saliency feature map; and based on the sample saliency feature map, the sample image fusion features are decomposed into sample image local features and sample image global features; wherein, the sample saliency feature map is used to represent the saliency distribution of the sample image fusion features in spatial location; The local and global features of the sample image are encoded using token serialization to obtain the joint token sequence of the sample; and the joint token of the sample is reduced in dimensionality based on the learnable token compressor to be trained to obtain the compact token representation of the sample. Attention modeling of the sample compact token representation is performed using the master decoder in the dual-branch decoder to be trained, generating a predicted change description; and spatial mapping decoding of the sample compact token representation is performed using the auxiliary decoder in the dual-branch decoder to be trained, to obtain a reconstructed change difference map. An initial remote sensing image change description model is constructed based on the feature extraction and alignment module to be trained, the saliency prediction head, the learnable token compressor, and the dual-branch decoder. The initial remote sensing image change description model is trained based on the predicted change description and the reconstructed change difference map to obtain the trained target remote sensing image change description model; wherein, the target remote sensing image change description model is used to generate change descriptions of the two-temporal remote sensing image pairs to be described.

7. The method according to claim 6, characterized in that, The process of training the initial remote sensing image change description model based on predicted change description and reconstructed change difference map yields a trained target remote sensing image change description model, including: Calculate text generation loss based on predicted change descriptions and corresponding change description annotations; The auxiliary task loss is calculated based on the reconstructed change difference map and the labeled change difference map; wherein, the labeled change difference map is generated based on the change description labels; The text generation loss and the auxiliary task loss are weighted and fused to obtain the comprehensive loss; Backpropagation is performed on the initial remote sensing image change description model based on the comprehensive loss, and the model parameters are updated using an optimizer to obtain the trained target remote sensing image change description model.

8. The method according to claim 6, characterized in that, The step of preprocessing the training sample set to obtain the preprocessed training sample set includes: Adjust the sample dual-temporal remote sensing image pairs in the training sample set to the preset image size to obtain the sample dual-temporal remote sensing image pairs with adjusted size. Image enhancement processing is performed on the resized dual-temporal remote sensing image pairs to obtain the preprocessed training sample set.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 8.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images