A state space model driven remote sensing image change caption generation method

Abstract

The present application relates to the technical field of remote sensing image processing and artificial intelligence, in particular to a state space model driven remote sensing image change caption generation method, the method comprises the following steps: obtaining image data and word mapping table of remote sensing image intelligent interpretation; extracting double time phase features through a double time sharing branch feature extractor; constructing a joint input sequence of a decoder according to the difference enhanced features, multi-scale visual features and text sequences; taking the constructed joint input sequence as input, calculating word probability distribution by autoregressive calculation based on a pre-trained language model decoder, completing prediction and cyclic iteration according to the word mapping table, and obtaining complete image change description sentences; the present application solves the problem of losing high frequency details in long sequence modeling of state space model through frequency domain modulation, and effectively suppresses background noise interference by using difference perception aggregation, thereby improving the accuracy of remote sensing image difference description generation.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and artificial intelligence technology, specifically to a method for generating captions for changes in remote sensing images driven by a state-space model. Background Technology

[0002] With the rapid advancement of remote sensing satellite sensor technology, humanity's ability to acquire high-resolution, multi-temporal remote sensing data has been greatly enhanced. The explosive growth of Earth observation data has made it possible to automatically monitor and analyze changes on the Earth's surface. Remote sensing image change detection, as one of the core tasks in this field, aims to identify differences between images of the same geographic area acquired at different times.

[0003] Remote sensing image change detection tasks primarily focus on pixel-level localization. The resulting binary masks can only identify "where changes occur," lacking high-level semantic representations of the category, attributes, and spatial relationships of the changed objects, thus failing to effectively explain "what kind of changes have occurred." To bridge this semantic gap, remote sensing image change interpretation has emerged as a cross-modal task. It aims to comprehensively analyze dual-temporal remote sensing images to generate a fluent and accurate natural language text that comprehensively describes the category, location, attributes, and dynamic processes of changes in ground features, achieving a deep semantic understanding of surface dynamics. This has significant application value in fields such as urban planning and disaster assessment.

[0004] Existing remote sensing image change interpretation technologies can be mainly divided into the following two categories: The first category consists of early methods based on CNN-RNN. These methods typically use convolutional neural networks (CNNs) to extract image features and recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) to generate descriptions. However, due to the limitations of the RNN structure, these methods are prone to the vanishing gradient problem when processing long sequences, making it difficult to effectively capture the complex long-range spatial dependencies in remote sensing images, resulting in descriptions that often lack global consistency.

[0005] The second category is based on Transformer and its variants. These methods introduce self-attention mechanisms to capture long-range dependencies in images, significantly improving the global interaction capabilities of features. However, the standard Transformer architecture typically has a computational complexity that is quadratic with the sequence length, leading to significant computational overhead and memory consumption when processing high-resolution remote sensing imagery. Furthermore, traditional global attention mechanisms tend to treat all regions in the image equally, lacking explicit guidance for "changing regions." When faced with complex land cover (such as seasonal variations, illumination differences, and other background noise), they are prone to producing spurious change descriptions due to overemphasis on irrelevant background information. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention primarily solves the technical problems of easily losing high-frequency structural details and being severely affected by background noise in current remote sensing image change interpretation techniques. This invention provides a state-space model-driven method for generating change captions for remote sensing images. This method employs a deep neural network architecture based on a state-space model and achieves efficient and accurate difference description generation by introducing a frequency domain feature modulation mechanism and a multi-scale linear interactive decoding strategy.

[0007] The first objective of this invention is to provide a method for generating captions for changes in remotely sensed images driven by a state-space model, comprising: Two remote sensing images of the same area at different times are acquired and preprocessed to obtain image data and word mapping table for intelligent interpretation of remote sensing images; The image data is processed by a dual-temporal shared-branch feature extractor to extract dual-temporal features; wherein the dual-temporal shared-branch feature extractor is based on frequency domain priors and is composed of a state-space model; A differential guidance aggregation unit based on soft hybrid attention experts is constructed. The differential guidance aggregation unit uses a soft routing mechanism to aggregate differential features into several semantic slots and inputs them into a set of parallel attention experts. It interacts with temporal features through a cross-attention mechanism with shared key-value projection, and then reorganizes them to generate incremental change features and injects them into the dual temporal features. By constructing a difference-guided aggregation unit, difference features are extracted from the bi-temporal features and transformed into incremental change features, which are then injected into the original features to obtain difference-enhanced features. The decoder's joint input sequence is constructed based on the differentially enhanced features, multi-scale visual features, and text sequences. Based on a pre-trained language model decoder, the constructed joint input sequence is used as input. The word probability distribution is calculated by autoregression, and prediction is completed according to the word mapping table and iteratively to obtain a complete description of image changes. The constructed network model is then trained in a supervised manner to obtain the appropriate network parameters.

[0008] In one embodiment, when constructing the joint input sequence of the decoder, a semantic prototype library is retrieved using differential features and combined with instance meta tokens generated by the meta-network to obtain dynamic hybrid prompts, which are then concatenated with multi-scale visual features and text sequences to obtain the joint input sequence.

[0009] In one embodiment, dynamic hybrid hints are obtained by retrieving a semantic prototype library using differential features and combining it with instance meta-tokens generated by a meta-network, including: Global pooling is performed on the enhanced features to obtain the global difference vector; Use global difference vectors to retrieve a pre-defined semantic prototype library and generate general semantic prompts; The global difference vector is mapped to instance meta tokens using a meta-network; The general semantic hints are fused with the instance meta tokens to obtain a dynamic hybrid hint, which is then concatenated to the front end of the input sequence.

[0010] In one embodiment, the dual-time shared branch feature extractor processes the input features using a frequency-enhanced state-space feature extraction unit at each stage during feature extraction, and performs a frequency prior extraction step: the input features are decomposed into orthogonal frequency sub-bands by discrete wavelet transform, and high-frequency coefficient components in the horizontal, vertical and diagonal directions are extracted, and the high-frequency coefficient components in the three directions are concatenated in the channel dimension to form a high-frequency structure prior.

[0011] In one embodiment, the construction of the difference-guided aggregation unit includes: Calculate the absolute difference between the two temporal features and generate differential prior features through convolution; The routing score is calculated using a learnable routing matrix, and the differential prior features are weighted and aggregated into multiple semantic slots; The semantic slots are input into a group of parallel attention experts. The semantic slots are used as query terms, and the features of the current time phase are the key and value terms shared by all experts. The output features of each expert are extracted through a cross-attention mechanism. Based on the routing score, the combined weights are calculated, and the output features of all attention experts are weighted and mapped back to the original pixel space to obtain the incremental change features. The incremental change features are then injected into the dual-temporal features in the form of residuals. During model training, a routing orthogonality loss constraint is applied to the routing matrix, forcing the column vectors of the routing matrix to be mutually orthogonal.

[0012] In one embodiment, the supervised training process includes: The Argmax function is used to select the index corresponding to the element with the highest probability value at each time step of the probability vector sequence. The index is then mapped to a natural language word according to the word mapping table. This process is repeated until the end symbol is generated, resulting in a complete difference description statement. Construct a total loss function, which includes text generation loss and route orthogonalization loss.

[0013] In one embodiment, the pre-trained language model decoder is a Transformer architecture decoder; the sequence input to the decoder includes fixed text instructions, dynamic mixed prompts, multi-scale visual feature sequences, and text embedding sequences generated at the current time step; the decoder processes the sequence through a masked self-attention module, so that the prediction of the current position depends only on historical information.

[0014] In one embodiment, the preprocessing and word mapping table construction process includes: Adjust the two remote sensing images to the same size; Perform word frequency statistics on the image description text in the training data and filter out low-frequency words that appear less than a preset threshold; add indexes to the remaining remote sensing-related words and construct a restricted word mapping table; Set the dimension of the projection matrix of the output layer of the generative model to be the same as the size of the word mapping table.

[0015] A second objective of this invention is to provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a state-space model-driven method for generating captions for changes in remotely sensed images.

[0016] The third objective of this invention is to provide a state-space model-driven remote sensing image change caption generation system, employing a state-space model-driven remote sensing image change caption generation method, comprising: The data acquisition module is used to acquire two remote sensing images of the same area at different times and preprocess them to obtain image data and word mapping tables for intelligent interpretation of remote sensing images; The feature extraction module is used to extract dual-temporal features from image data through a dual-temporal shared-branch feature extractor; wherein the dual-temporal shared-branch feature extractor is based on frequency domain priors and is composed of a state-space model; The feature enhancement module is used to construct a difference-guided aggregation unit based on soft hybrid attention experts. The difference-guided aggregation unit uses a soft routing mechanism to aggregate difference features into several semantic slots and inputs them into a set of parallel attention experts. It interacts with temporal features through a cross-attention mechanism with shared key-value projection, and then reorganizes them to generate incremental change features and injects them into the dual-temporal features. The constructed difference-guided aggregation unit mines difference features from the extracted dual-temporal features and transforms them into incremental change features, which are then injected into the original features to obtain the difference-enhanced features. The sequence construction module is used to construct the joint input sequence of the decoder based on the differentially enhanced features, multi-scale visual features, and text sequences; The text generation module is used to calculate the word probability distribution based on the pre-trained language model decoder, taking the constructed joint input sequence as input, and performing autoregressive calculations. It then performs predictions based on the word mapping table and iterates cyclically to obtain complete image change descriptions. The module also includes supervised training of the built network model to obtain suitable network parameters.

[0017] The present invention has at least the following beneficial effects: This invention provides a state-space model-driven method for generating change captions for remote sensing images. This method is implemented through a single end-to-end deep learning framework, integrating the entire process of data acquisition, feature extraction, difference analysis, and text generation. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the implementation of the remote sensing image change caption generation method provided in this embodiment of the invention. Figure 2 This is a schematic diagram of the overall network architecture of the remote sensing image change caption generation method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the frequency-enhanced state-space feature extraction unit provided in an embodiment of the present invention; Figure 4 This is a simulation example of the remote sensing image change caption generation method provided in the embodiments of the present invention. Detailed Implementation

[0019] In order to illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description is provided in conjunction with the embodiments.

[0020] The purpose of this invention is to provide a state-space model-driven method for generating captions for changes in remote sensing images, mainly addressing the following problems: 1) To address the problem of high-frequency detail loss, a frequency-gated state-space feature extraction strategy is proposed. By introducing wavelet transform to extract high-frequency structural priors and injecting them into the state-space model, the local detail loss caused by block processing is explicitly compensated, achieving robust fusion of the space-frequency domain.

[0021] 2) To address the issue of background noise interference, a difference-aware aggregation mechanism was introduced. This mechanism uses feature differences to guide the model to focus on regions of change, effectively filtering redundant background information and enhancing the ability to capture targets with subtle changes.

[0022] 3) To address the challenges of cross-modal semantic mapping and the lack of descriptive detail, an instance-aware hybrid prompting and joint input sequence were constructed. Semantic prototypes are retrieved using differential features and combined with instance meta-tokens to generate dynamic prompts, which serve as guiding prefixes for the decoder. This strategy introduces explicit semantic priors and instance attributes, effectively bridging the modal gap and ensuring that the generated differential descriptive semantics are accurate and rich in detail.

[0023] This invention first acquires and preprocesses two remote sensing images of the same area taken at different times. The images are then input into a state-space model with a frequency domain prior to extract multi-scale features. Discrete wavelet transform is used to extract high-frequency details, which are then injected into the visual state-space module. A difference-guided aggregation unit is constructed, utilizing a soft hybrid expert mechanism and cross-attention interaction with dual-temporal features to generate difference-enhancing features. A large-scale model for generating joint sequence input text with semantic prompts is constructed. During generation, the output space is limited to a pre-defined fixed vocabulary to ensure that the generated descriptive words originate only from a restricted dictionary. Finally, a statement describing image changes is output. This invention solves the problem of lost high-frequency details in long-sequence modeling by using frequency domain modulation and effectively suppresses background noise interference by utilizing difference-aware aggregation, thus improving the accuracy of remote sensing image difference description generation.

[0024] To achieve the above objectives, see Figure 1 and Figure 2 As shown, a method for generating captions for changes in remote sensing images driven by a state-space model includes: S1. Acquire two remote sensing images of the same area at different times and preprocess them to obtain image data and word mapping table for intelligent interpretation of remote sensing images; The preprocessing and word mapping table construction process includes: Adjust the two remote sensing images to the same size; Perform word frequency statistics on the image description text in the training data and filter out low-frequency words that appear less than a preset threshold; add indexes to the remaining remote sensing-related words and construct a restricted word mapping table; Set the dimension of the projection matrix of the output layer of the generative model to be the same as the size of the word mapping table.

[0025] For example, acquiring two remote sensing images of the same area acquired at different times. and The remote sensing images are preprocessed to obtain image data and a word mapping table for intelligent interpretation of remote sensing images; specifically: The image size was adjusted to 3*256*256. The frequency of words in the image description was statistically analyzed, and low-frequency words with fewer than 5 occurrences were filtered out. The remaining words were indexed to form a word mapping table.

[0026] S2. Extract dual-temporal features from the image data using a dual-temporal shared-branch feature extractor; wherein the dual-temporal shared-branch feature extractor is based on frequency domain priors and is constructed from a state-space model; The dual-time shared branch feature extractor processes the input features at each stage using a frequency-enhanced state-space feature extraction unit, and performs a frequency prior extraction step: the input features are decomposed into orthogonal frequency sub-bands by discrete wavelet transform, and high-frequency coefficient components in the horizontal, vertical and diagonal directions are extracted. The high-frequency coefficient components in the three directions are then concatenated in the channel dimension to form a high-frequency structure prior.

[0027] For example, the obtained image data is input into a dual-temporal shared-branch feature extractor composed of a state-space model with fused frequency domain priors to extract dual-temporal features; specifically: This feature extractor employs a three-stage hierarchical Siamese architecture, utilizing a frequency-enhanced state-space feature extraction unit to process the input features at each stage. Let the... Phase 1 The input features of the time phase are If it is the first stage, The initial image features are extracted by a backbone network, which can be a convolutional neural network such as ResNet-101, used to map the original image into a high-dimensional feature space. See [link to relevant documentation]. Figure 3 As shown, the specific implementation process of this unit is as follows: First, a frequency prior extraction branch is constructed for the input features. After convolution processing, the discrete wavelet transform (DWT) is used to decompose it into orthogonal frequency subbands, and high-frequency coefficient components in the horizontal, vertical, and diagonal directions are extracted. The calculation formula is as follows:

[0028] in, These represent the high-frequency coefficient components in the horizontal, vertical, and diagonal directions, respectively. Indicates the convolution operation; This represents the discrete wavelet transform.

[0029] Subsequently, the high-frequency components are spliced ​​and aggregated along the channel dimension, and then input into a space-frequency feature integration unit containing a nonlinear activation function for feature projection to generate a frequency-gated signal containing high-frequency structural priors. The calculation formula is as follows:

[0030] in, Indicates a splicing operation; This represents the SiLU nonlinear activation function.

[0031] Meanwhile, to address the issue of reduced frequency subband spatial resolution caused by discrete wavelet transform, a step size of 2 is used... Convolutional layers on original input features By performing downsampling projection, spatial alignment features consistent with the spatial dimension of the frequency-gated signal are obtained. The calculation formula is as follows:

[0032] Based on this, a frequency information injection mechanism is established, which utilizes the generated frequency-gated signal through element-by-element multiplication. Spatial alignment features Modulation is performed to obtain frequency conditional characteristics. The calculation formula is as follows:

[0033] in, This represents element-wise multiplication; this step utilizes frequency domain priors to suppress background noise and significantly enhance high-frequency details.

[0034] Finally, the modulated features The input is fed into the visual state space module, where the global context is modeled using a two-dimensional selective scan mechanism (SS2D), and the final multi-scale features are output. The calculation formula is as follows:

[0035] in, This indicates a two-dimensional selective scanning operation.

[0036] Specifically, the SS2D mechanism first processes the input features Unfolding along four directions (top left, bottom right, top right, bottom left) into a one-dimensional sequence, denoted as... (in ).

[0037] For the sequence in each direction, parallel scanning processing is performed using the discretized state-space equations. The calculation process is as follows: First, the zero-order preserve (ZOH) method is used to convert the continuous state-space parameters. Discretize into :

[0038]

[0039] in, For time scale parameters; The state evolution matrix; For the input projection matrix; The identity matrix is ​​used; subsequently, a recursive scan is performed based on the discretization parameters to update the hidden state. And output features :

[0040]

[0041] in, This is the input for the current moment; This is the hidden state from the previous moment; This is for outputting the projection matrix.

[0042] Finally, the scan output sequences in four directions are... The feature maps are then converted to two-dimensional feature maps, summed and fused, and passed through a linear projection layer to output the final feature maps. : .

[0043] It should be noted that the multi-scale features mentioned above are the features output by the three stages of the model encoder. Here, "feature" can be considered a temporary symbol, mainly used to explain the multi-scale feature calculation process. Numerically, multi-scale feature = ... Linear ( Fout ).

[0044] S3. Construct a differential guidance aggregation unit based on soft hybrid attention experts; wherein, the differential guidance aggregation unit uses a soft routing mechanism to aggregate differential features into several semantic slots and inputs them into a set of parallel attention experts, interacts with temporal features through a cross-attention mechanism with shared key-value projection, and then reorganizes to generate incremental change features and injects them into the dual temporal features; By constructing a difference-guided aggregation unit, difference features are extracted from the bi-temporal features and transformed into incremental change features, which are then injected into the original features to obtain difference-enhanced features. The construction of difference-guided aggregation units includes: Calculate the absolute difference between the two temporal features and generate differential prior features through convolution; The routing score is calculated using a learnable routing matrix, and the differential prior features are weighted and aggregated into multiple semantic slots; The semantic slots are input into a group of parallel attention experts. The semantic slots are used as query terms, and the features of the current time phase are the key and value terms shared by all experts. The output features of each expert are extracted through a cross-attention mechanism. Based on the routing score, the combined weights are calculated, and the output features of all attention experts are weighted and mapped back to the original pixel space to obtain the incremental change features. The incremental change features are then injected into the dual-temporal features in the form of residuals. During model training, a routing orthogonality loss constraint is applied to the routing matrix, forcing the column vectors of the routing matrix to be mutually orthogonal.

[0045] For example, a differential guidance aggregation unit based on soft hybrid attention experts is constructed. A soft routing mechanism is used to aggregate differential features into several semantic slots, which are then input into a set of parallel attention experts. These experts interact with temporal features through a cross-attention mechanism using shared key-value projection. Finally, the features are recombined to generate incremental change features and injected into the dual-temporal features. Specifically: First, regarding the features output by the feature extractor in S2 Calculate the absolute difference between the two, and then use a... Convolutional layers and nonlinear activation functions (SiLU) extract differential prior features containing information about varying spatial locations. To accommodate the input requirements of hybrid expert models, Flattened into a one-dimensional labeled sequence in spatial dimension ,in To mark the total number.

[0046] Subsequently, a soft routing mechanism with orthogonal constraints is introduced to perform semantic aggregation of differential features. A learnable routing matrix is ​​defined. ,in The total number of expert slots is set (e.g., 16). The column vectors of this matrix are constrained to be mutually orthogonal during training to ensure that different slots capture non-redundant semantic components (such as edge variations, texture variations, color variations, etc.) in the differential features. For the input sequence... conduct Normalization is performed to stabilize the gradient, and the routing score matrix is ​​computed. On the token dimension Perform a softmax operation to obtain the weight distribution matrix. The input labels are weighted and aggregated using this weight. semantic slots The calculation formula is: .

[0047] Next, build A parallel single-head attention expert Semantic slots are processed. To reduce computational complexity and the number of parameters, a shared key-value projection strategy is adopted: using the temporal feature sequence to be enhanced... As a global context, key terms are generated through a shared projection matrix. Sum of values ,Right now ,in Shared by all experts. Each expert Only a separate query projection matrix needs to be maintained. , will the Slot characteristics Mapped to a specific query vector Subsequently, each expert calculates the relevance of the differential semantics they are responsible for to the global temporal context and extracts features: , in, C is the expert's independent output projection matrix; C is... The matrix dimension value.

[0048] Finally, a soft composition mechanism is used to complete feature recombination and injection. The aforementioned routing score matrix is ​​then analyzed from an expert perspective. Perform a softmax operation to obtain the combined weight matrix. . use All experts' output The weighted mapping back to the original pixel space yields the incremental change features. Introduce a learnable scalar initialized to 0. The generated incremental features are injected back into the original temporal features as residuals to generate the final enhanced features. In this way, the model significantly enhances its semantic perception of changing regions while preserving the original temporal information.

[0049] S4. Construct the joint input sequence of the decoder based on the enhanced features, multi-scale visual features, and text sequence; When constructing the joint input sequence of the decoder, the semantic prototype library is retrieved using differential features and combined with instance meta tokens generated by the meta-network to obtain dynamic hybrid cues. These cues are then concatenated with multi-scale visual features and text sequences to obtain the joint input sequence.

[0050] Dynamic hybrid hints are obtained by retrieving a semantic prototype library using differential features and combining it with instance meta-tokens generated by a meta-network, including: Global pooling is performed on the enhanced features to obtain the global difference vector; Use global difference vectors to retrieve a pre-defined semantic prototype library and generate general semantic prompts; The global difference vector is mapped to instance meta tokens using a meta-network; The general semantic hints are fused with the instance meta tokens to obtain a dynamic hybrid hint, which is then concatenated to the front end of the input sequence.

[0051] For example, an instance-aware hybrid cue and joint input sequence are constructed. A semantic prototype library is retrieved using differential features, and instance meta-tokens generated by a meta-network are combined to obtain a dynamic hybrid cue. This cue is then concatenated with multi-scale visual features and a text sequence as the joint input to the decoder; specifically: A dynamic hybrid cue containing general semantic cues and instance meta-tokens is constructed, and the decoder input sequence is assembled. Specifically, firstly, based on the differential features output from the third stage of the encoder in step S3... Construct a global difference-aware vector. For Perform global average pooling to compress it into a one-dimensional vector. This vector aggregates global semantic information about changing regions in the image.

[0052] Subsequently, this global vector is used to retrieve a pre-defined library of learnable and variable prototypes. Let the prototype library be... ,in Indicates the number of potential change patterns. This represents the embedding dimension of the decoder. The prototype library is established using a semantic-guided strategy during the initialization phase. This involves pre-defining a set of semantic phrases containing various descriptions of land cover state transitions (such as "road to water" and "forest to building"). Each phrase is input into the embedding layer of a pre-trained language model to extract word vector sequences, and their mean is calculated to obtain a fixed-length semantic transition prototype vector, which is then used to initialize the matrix. In reasoning, Mapped to a query vector through a linear projection layer. ,calculate With prototype library The cosine similarity of each prototype vector in the dataset, and through... Function normalization yields the attention weight vector The calculation formula is as follows: .

[0053] Using weights For prototype library Perform weighted summation to generate a general semantic cue vector representing the semantic category of change in the current image. .

[0054] To address the shortcomings of the general prototype in details such as specific textures and intensity of changes, a lightweight meta-Net (Meta-Net) is further introduced to generate instance meta tokens. A nonlinear mapping network composed of multilayer perceptrons (MLPs) is constructed. , global difference vector Directly mapped to instance token The formula is The token carries continuous value attribute information specific to the current image. Next, general semantic hints will be provided. With instance meta token Adaptive fusion is performed to obtain the final dynamic soft suggestion. :

[0055] in, This is the balance coefficient; this dynamic soft hint It will be used as a context prefix for subsequent text generation.

[0056] Finally, the complete joint sequence of the input decoder is constructed. Pre-temporal image features extracted by the encoder in step S2 are selected. Features of post-temporal images (i.e., dual-temporal features), and the differential enhancement features generated in step S3. The linear projection layer is used to uniformly map the above features to the decoder dimension. Furthermore, by flattening and cascading the data in the spatial dimension, a visual feature sequence containing complete dual-temporal information is obtained. Fixed text instructions (e.g., "Describe the change"), dynamic soft prompts Visual feature sequences and the text embedding sequence generated at the current time step Concatenate them sequentially to form a three-part joint input sequence. : .

[0057]

[0058] This is a matrix obtained by uniquely encoding the generated set of words.

[0059] It should be noted that the fixed text instruction is a prompt word that is manually determined and input into the large model; the visual feature sequence is a set of temporal and differential features output by the encoder in three stages; and the text embedding sequence is obtained by inputting the generated text into the embedding layer.

[0060] S5. Based on the pre-trained language model decoder, the constructed joint input sequence is used as input, the word probability distribution is calculated by autoregression, the prediction is completed according to the word mapping table and iteratively to obtain a complete image change description statement; among them, the built network model is trained in a supervised manner to obtain the adapted network parameters.

[0061] The pre-trained language model decoder is a Transformer architecture decoder; the input sequence to the decoder includes fixed text instructions, dynamic mixed prompts, multi-scale visual feature sequences, and text embedding sequences generated at the current time step; the decoder processes the sequence through a masked self-attention module, so that the prediction at the current position depends only on historical information.

[0062] The supervised training process includes: The Argmax function is used to select the index corresponding to the element with the highest probability value at each time step of the probability vector sequence. The index is then mapped to a natural language word according to the word mapping table. This process is repeated until the end symbol is generated, resulting in a complete difference description statement. Construct a total loss function, which includes text generation loss and route orthogonalization loss.

[0063] For example, text is generated based on a pre-trained language model decoder. Hybrid prompts are used as prefixes to guide the pre-trained Transformer architecture decoder in autoregressive word probability distribution calculations. Predictions are then made based on a word mapping table, and the process iterates until a complete description of image changes is obtained. Specifically: A pre-trained decoder based on the Transformer architecture and a causal language model with a GPT-2 structure are used for progressive text generation. Specifically, the joint sequence constructed in step S4 is... The input is fed into the decoder. The decoder internally contains multiple stacked masked self-attention modules, which use a causal masking mechanism to ensure that the prediction at the current position depends only on historical information in the sequence. This is due to dynamic soft cues. Located at the beginning of the sequence, it uses an attention mechanism as prior knowledge to guide the feature updates of all subsequent layers, directing the model to focus on the visual feature sequence. The key regions related to change are identified, thereby enabling cross-modal conversion from visual change semantics to natural language description.

[0064] At the output of the decoder, extract the current time step. The corresponding hidden state vector .Will The input is fed into a linear classification head, which maps it to a predefined vocabulary space with a dimension equal to the vocabulary size. Subsequently, using The function calculates the probability distribution vector of the next word in the vocabulary. :

[0065] in, and These are the weight matrix and bias term of the classification head, respectively.

[0066] Finally, the cyclic prediction process is executed. Based on the probability distribution vector... The word corresponding to the index with the highest probability value is selected as the prediction result for the current time step. The predicted words Convert to an embedding vector and append it to the generated text sequence. The end of the input is used as the input for the next time step. The decoding, probability calculation, and word selection process described above is repeated until the model outputs an end symbol (such as...). <eos>(or the generated sequence length reaches a preset threshold, thus obtaining a complete descriptive statement about the changes in the remote sensing image.)

[0067] In this invention, a supervised training process is performed on the end-to-end remote sensing image change caption generation network driven by the established state-space model to obtain network parameters suitable for the model. Specifically: Image pairs The input is fed into the network model, where it undergoes encoder feature extraction and decoder sequence generation at each time step. Output the probability distribution vector of the current predicted word on the predefined vocabulary. .

[0068] Text generation loss The cross-entropy loss function is used to measure the predicted sequence generated by the model. The difference between the actual description and the loss value. The calculation formula is as follows:

[0069] in, Indicates the length of the descriptive statement sequence. Indicates the size of the vocabulary; Indicates at time step At that time, the first in the real label One-hot encoded values ​​of each word; This indicates that the model predicts at time step Output the first The probability of each word.

[0070] Routing Orthogonalization Loss To address the issue in soft hybrid expert models where different slots tend to focus on the same feature patterns, leading to reduced expert utilization, the routing matrix in step S3... An orthogonalization constraint is introduced. The correlation between the column vectors of the routing matrix is ​​calculated, and off-diagonal elements are penalized, as shown in the following formula:

[0071] in, For routing matrix, It is the identity matrix. This represents the Frobenius norm. This loss term forces the 16 slots to be mapped onto mutually orthogonal basis vectors in the feature space, ensuring that each expert is responsible for capturing the different semantic components of the differential features, thus maximizing the diversity and utilization of experts.

[0072] Total loss function The final training objective is composed of the weighted sum of the above two parts:

[0073] in, This is a balance coefficient used to adjust the strength of orthogonal constraints. It is based on the calculated total loss value. The Adam optimization algorithm is used to calculate the gradient of the network parameters and perform backpropagation to update the weight parameters in the model.

[0074] See Figure 4 As shown in the simulation example of the remote sensing image change caption generation method, after obtaining remote sensing images acquired by satellite, radiometric calibration, correction, dual temporal registration, cropping and normalization are performed on them, and then input into the model constructed in this invention to obtain a natural language description of the image changes.

[0075] The model training method used in this invention includes: (1) Experimental conditions The hardware platform used in this invention experiment was: Nvidia GeForce RTX 3090 and Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz.

[0076] The software platform used in this invention experiment is: Ubuntu 22.04.1, Python 3.10, and PyTorch 2.1.0.

[0077] The remote sensing imagery used in this invention's experiments was the Levir-CC dataset, which collected 10,077 pairs of dual-time-series remote sensing images (cropped to 256×256 pixels, resolution 0.5 m / pixel) covering 20 regions in Texas, USA, spanning 5–15 years. These images were accurately registered without any viewpoint variations. For each pair of images, five annotation sentences from different annotators were used to describe the differences, totaling 50,385 sentences.

[0078] (2) Evaluation indicators To objectively and comprehensively verify the effectiveness of the method proposed in this invention, this embodiment employs four common natural language processing evaluation metrics for quantitative testing, specifically including: BLEU-n (n=1,2,3,4): Used to evaluate the matching accuracy of n-gram phrases in the generated text; Meteor: Introduces a synonym matching mechanism to evaluate the semantic consistency between the generated text and the reference text; ROUGE-L: Based on the longest common subsequence, evaluate the recall and structural integrity of generated descriptions to reference information; CIDEr-D: Based on TF-IDF weights, it is specifically designed to evaluate the salience and consensus of key information in generated descriptions. The higher the score of the above indicators, the more accurate, fluent and human-language-compliant the description of changes in the generated remote sensing image.

[0079] (3) Comparison of algorithms and results analysis To verify the effectiveness of this invention, four methods for generating change captions from remote sensing images—RSICCformer, PSNet, PromptCC, and SEN—were selected. These methods were trained and tested on the Levir-CC dataset to obtain the final change interpretation results for the test set.

[0080] Table 1 Objective Evaluation Results

[0081] The objective evaluations of different methods are shown in Table 1. Bold text indicates the best performance for each indicator, and underlined text indicates the second-best performance for each indicator. This invention achieves the best results in Meteor, ROUGE-L, and CIDEr-D metrics. This indicates that the descriptive text generated by this invention has a more complete grammatical structure and can use richer and more accurate vocabulary to express the same changes in land features, thus exhibiting stronger robustness.

[0082] The visual state space model fused with frequency domain priors designed in this invention is compared with the conventional visual state space model (Vision Mamba), demonstrating that by introducing high-frequency components in the frequency domain as prior knowledge, this invention can more effectively adapt to remote sensing images rich in fine-grained textures. To make the experiment more convincing, this experiment constructed a basic version of the state space model with the frequency domain prior branches removed and only the spatial scanning mechanism retained as a benchmark for ablation testing. The experimental results are shown in Table 2.

[0083] Table 2. Performance of the Visual State Space Model Integrating Frequency Domain Priors

[0084] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a method for generating captions for changes in remote sensing images driven by a state-space model.

[0085] This invention provides a state-space model-driven remote sensing image change caption generation system and a state-space model-driven remote sensing image change caption generation method, including: The data acquisition module is used to acquire two remote sensing images of the same area at different times and preprocess them to obtain image data and word mapping tables for intelligent interpretation of remote sensing images; The feature extraction module is used to extract dual-temporal features from image data through a dual-temporal shared-branch feature extractor; wherein the dual-temporal shared-branch feature extractor is based on frequency domain priors and is composed of a state-space model; The feature enhancement module is used to construct a difference-guided aggregation unit based on soft hybrid attention experts. The difference-guided aggregation unit uses a soft routing mechanism to aggregate difference features into several semantic slots and inputs them into a set of parallel attention experts. It interacts with temporal features through a cross-attention mechanism with shared key-value projection, and then reorganizes them to generate incremental change features and injects them into the dual-temporal features. The constructed difference-guided aggregation unit mines difference features from the extracted dual-temporal features and transforms them into incremental change features, which are then injected into the original features to obtain the difference-enhanced features. The sequence construction module is used to construct the joint input sequence of the decoder based on the differentially enhanced features, multi-scale visual features, and text sequences; The text generation module is used to calculate the word probability distribution based on the pre-trained language model decoder, taking the constructed joint input sequence as input, and performing autoregressive calculations. It then performs predictions based on the word mapping table and iterates cyclically to obtain complete image change descriptions. The module also includes supervised training of the built network model to obtain suitable network parameters.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.< / eos>

Claims

1. A method for generating captions for changes in remotely sensed images driven by a state-space model, characterized in that, include: Two remote sensing images of the same area at different times are acquired and preprocessed to obtain image data and word mapping table for intelligent interpretation of remote sensing images; The image data is processed by a dual-temporal shared-branch feature extractor to extract dual-temporal features; wherein the dual-temporal shared-branch feature extractor is based on frequency domain priors and is composed of a state-space model; A differential guidance aggregation unit based on soft hybrid attention experts is constructed. The differential guidance aggregation unit uses a soft routing mechanism to aggregate differential features into several semantic slots and inputs them into a set of parallel attention experts. It interacts with temporal features through a cross-attention mechanism with shared key-value projection, and then reorganizes them to generate incremental change features and injects them into the dual temporal features. By constructing a difference-guided aggregation unit, difference features are extracted from the bi-temporal features and transformed into incremental change features, which are then injected into the original features to obtain difference-enhanced features. The decoder's joint input sequence is constructed based on the differentially enhanced features, multi-scale visual features, and text sequences. Based on a pre-trained language model decoder, the constructed joint input sequence is used as input, and the word probability distribution is calculated by autoregression. The prediction is completed according to the word mapping table and iteratively to obtain a complete description of image changes. The constructed network model is trained in a supervised manner to obtain the appropriate network parameters.

2. The method for generating captions for changes in remotely sensed images driven by a state-space model according to claim 1, characterized in that, When constructing the joint input sequence of the decoder, the semantic prototype library is retrieved using differential features and combined with instance meta tokens generated by the meta-network to obtain dynamic hybrid cues. These cues are then concatenated with multi-scale visual features and text sequences to obtain the joint input sequence.

3. The method for generating subtitles for changes in remotely sensed images driven by a state-space model according to claim 2, characterized in that, Dynamic hybrid hints are obtained by retrieving a semantic prototype library using differential features and combining it with instance meta-tokens generated by a meta-network, including: Global pooling is performed on the enhanced features to obtain the global difference vector; Use global difference vectors to retrieve a pre-defined semantic prototype library and generate general semantic prompts; The global difference vector is mapped to instance meta tokens using a meta-network; The general semantic hints are fused with the instance meta tokens to obtain a dynamic hybrid hint, which is then concatenated to the front end of the input sequence.

4. The method for generating captions for changes in remotely sensed images driven by a state-space model according to claim 1, characterized in that, The dual-time shared branch feature extractor processes the input features at each stage using a frequency-enhanced state-space feature extraction unit, and performs a frequency prior extraction step: the input features are decomposed into orthogonal frequency sub-bands by discrete wavelet transform, and high-frequency coefficient components in the horizontal, vertical and diagonal directions are extracted. The high-frequency coefficient components in the three directions are then concatenated in the channel dimension to form a high-frequency structure prior.

5. The method for generating subtitles for changes in remotely sensed images driven by a state-space model according to claim 1, characterized in that, The construction of difference-guided aggregation units includes: Calculate the absolute difference between the two temporal features and generate differential prior features through convolution; The routing score is calculated using a learnable routing matrix, and the differential prior features are weighted and aggregated into multiple semantic slots; The semantic slots are input into a group of parallel attention experts. The semantic slots are used as query terms, and the features of the current time phase are the key and value terms shared by all experts. The output features of each expert are extracted through a cross-attention mechanism. Based on the routing score, the combined weights are calculated, and the output features of all attention experts are weighted and mapped back to the original pixel space to obtain the incremental change features. The incremental change features are then injected into the dual-temporal features in the form of residuals. During model training, a routing orthogonality loss constraint is applied to the routing matrix, forcing the column vectors of the routing matrix to be mutually orthogonal.

6. The method for generating change captions for remote sensing images driven by a state-space model according to claim 5, characterized in that, The supervised training process includes: The Argmax function is used to select the index corresponding to the element with the highest probability value at each time step of the probability vector sequence. The index is then mapped to a natural language word according to the word mapping table. This process is repeated until the end symbol is generated, resulting in a complete difference description statement. Construct a total loss function, which includes text generation loss and route orthogonalization loss.

7. The method for generating captions for changes in remotely sensed images driven by a state-space model according to claim 1, characterized in that, The pre-trained language model decoder is a Transformer architecture decoder; the sequence of input decoders includes fixed text instructions, dynamic mixed prompts, multi-scale visual feature sequences, and text embedding sequences generated at the current time step; The decoder processes the sequence through a masked self-attention module, making the prediction of the current position depend only on historical information.

8. The method for generating captions for changes in remotely sensed images driven by a state-space model according to claim 1, characterized in that, The preprocessing and word mapping table construction process includes: Adjust the two remote sensing images to the same size; Perform word frequency statistics on the image description text in the training data and filter out low-frequency words that appear less than a preset threshold; add indexes to the remaining remote sensing-related words and construct a restricted word mapping table; Set the dimension of the projection matrix of the output layer of the generative model to be the same as the size of the word mapping table.

9. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the state-space model-driven remote sensing image change caption generation method according to any one of claims 1 to 8.

10. A state-space model-driven remote sensing image change caption generation system, employing the state-space model-driven remote sensing image change caption generation method as described in any one of claims 1 to 8, characterized in that, include: The data acquisition module is used to acquire two remote sensing images of the same area at different times and preprocess them to obtain image data and word mapping tables for intelligent interpretation of remote sensing images; The feature extraction module is used to extract dual-temporal features from image data through a dual-temporal shared-branch feature extractor; wherein the dual-temporal shared-branch feature extractor is based on frequency domain priors and is composed of a state-space model; The feature enhancement module is used to construct a difference-guided aggregation unit based on soft hybrid attention experts. The difference-guided aggregation unit uses a soft routing mechanism to aggregate difference features into several semantic slots and inputs them into a set of parallel attention experts. It interacts with temporal features through a cross-attention mechanism with shared key-value projection, and then reorganizes them to generate incremental change features and injects them into the dual-temporal features. The constructed difference-guided aggregation unit mines difference features from the extracted dual-temporal features and transforms them into incremental change features, which are then injected into the original features to obtain the difference-enhanced features. The sequence construction module is used to construct the joint input sequence of the decoder based on the differentially enhanced features, multi-scale visual features, and text sequences; The text generation module is used to calculate the probability distribution of words based on the pre-trained language model decoder, taking the constructed joint input sequence as input, and performing autoregressive calculation of word probability distribution. It then performs prediction based on the word mapping table and iterates cyclically to obtain a complete description of image changes. The module also includes supervised training of the built network model to obtain suitable network parameters.

Images