Task-adaptive remote sensing image change description method based on prototype guidance

Abstract

This invention discloses a prototype-guided task-adaptive remote sensing image change description method. Given a pair of dual-temporal remote sensing images, a pre-trained CLIP visual encoder is used as a shared backbone to extract hierarchical feature representations from layers 6, 12, 18, and 24 of the Transformer. A prototype-guided change-aware interaction module performs structured semantic modeling of the dual-temporal features. A task-adaptive multi-head gating module generates detection-guided features and description-guided features at the attention head granularity. The detection decoder decodes the detection features into a change probability map. The description decoder encodes the detection features into compact detection tokens, which are then concatenated with the description features and fed into a language model to generate a change description. Compared with existing technologies, this approach, through K-means prototype clustering + RBF spatial recovery + prototype modulation cross-temporal attention, enables the model to explicitly perceive the type of change and guide temporal correspondence. Within a single unified feature space, head-level sigmoid gating generates differentiated representations for detection and description respectively, eliminating task conflicts and allowing them to mutually enhance each other. Encoding the finest-grained feature map of FPN as detection tokens and concatenating it with description features allows the language model to explicitly know the spatial distribution of the change region, effectively eliminating position illusion errors and achieving the dual advantages of accuracy and efficiency.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image change description technology, and more specifically, relates to a prototype-guided task-adaptive remote sensing image change description method. Background Technology

[0002] Remote Sensing Image Change Captioning (RSICC) is a cross-modal task that integrates computer vision and natural language processing. It aims to generate natural language descriptions of semantic changes between two-temporal remote sensing images, overcoming the limitations of traditional change detection which only outputs binary masks. It provides interpretable information for land use auditing, urban monitoring, and emergency response.

[0003] Current mainstream methods follow an encoder-decoder paradigm: RSICCformer introduces a dual-branch Transformer to interact with bi-temporal features; Chg2Cap uses an attention difference module to highlight regions of change; PromptCC employs a cue-based decoupling paradigm; RSCaMa uses a state-space model to capture long-range temporal dependencies; and Semantic-CC and KCFI introduce large language models to generate richer semantic descriptions. Furthermore, ChangeChat, CDChat, and BTCChat utilize multimodal large language models (MLLM) for change understanding. Other joint detection-description methods, such as Semantic-CC, KCFI, and ChangeMinds, attempt to improve description quality by sharing a backbone network and using dual task heads, leveraging spatial supervision signals from change detection.

[0004] However, existing technologies have the following two fundamental limitations:

[0005] (1) Implicit feature difference lacks structured semantic modeling of changes: Existing methods all rely on implicit feature difference operations that directly subtract features from each other, treating all spatial locations the same and failing to explicitly model change types, such as the structured semantic priors corresponding to building construction, demolition, renovation, and solar facility installation. This leads to inaccurate temporal correspondences in complex scenarios, a lack of semantic guidance for cross-temporal attention, and difficulty in generating accurate descriptions of changes;

[0006] (2) Conflict in Feature Requirements Between Detection and Description Tasks: Change detection requires fine spatial localization accuracy, while change description requires high semantic abstraction capabilities. The feature representation requirements of the two are inherently conflicting. Existing joint methods simply adopt shared feature design, failing to address this granularity mismatch at the representation level: shared features expose the detection branch to semantic ambiguity interference, while simultaneously subjecting the description branch to unnecessary spatial precision constraints, resulting in neither task fully realizing its potential. Furthermore, existing methods fail to effectively inject the spatial priors generated by the detection branch into the description generation process, meaning spatial localization information fails to truly serve language generation. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a prototype-guided task-adaptive remote sensing image change description method. The method takes change description as the main task and change detection as the spatial positioning prior. It explicitly models changes through a learnable change type prototype library, and semantically guides cross-temporal attention to focus on semantically relevant regions to generate accurate change descriptions. At the same time, it learns task-differentiated gating weights at the attention head granularity, thus satisfying the different requirements of spatial accuracy (detection) and semantic abstraction (description) in a unified feature space.

[0008] To achieve the above-mentioned objectives, the present invention provides a prototype-guided task-adaptive remote sensing image change description method, characterized by comprising the following steps:

[0009] (1) Obtaining dual-temporal features ;

[0010] (2) Prototype-guided change-aware interaction module for dual-temporal features Perform structured semantic modeling

[0011] 2.1) Prototype Library Initialization

[0012] Offline initialization of the training set dimension is prototype library ,in, For the number of prototype clusters, For the number of space tokens, For feature dimensions:

[0013] First, for the training set... The training samples are dual-temporal remote sensing images. The mid-level features extracted using the 12th layer of the CLIP visual encoder Calculate the two-phase difference characteristics :

[0014]

[0015] Then, for the changing samples, mask-guided dual-temporal difference features are used. Average pooling of the changed regions yields a compact representation, while for unchanged samples, the bi-temporal difference features are directly applied. Average pooling yields compact representation ;

[0016] Then, a compact representation of all training samples. Perform K-means clustering to achieve natural coverage. One prototype cluster, i.e., one unchanging mode and Different types of variations, and based on compact representation Weights are assigned based on the distance to each prototype cluster, with closer clusters receiving larger weights. Weights Radial basis function interpolation is used to compactly represent the aggregated vector. Re-unfold into a dense spatial map ;

[0017] Then, by weight By aggregating spatial features, the initial prototype is obtained. :

[0018]

[0019] in, This represents the number of training samples;

[0020] Finally, we obtained the prototype library. :

[0021]

[0022] Prototype Library As a learnable parameter, it can be jointly optimized with downstream tasks;

[0023] 2.2) Transtemporal Phase Interaction of Prototype Modulation

[0024] First, the dual-temporal features are fused. and its difference Build change query :

[0025]

[0026] in, This represents a multilayer perceptron;

[0027] Then, calculate the query. ,key as well as :

[0028]

[0029] in, , , For learnable parameter matrix, express Flatten ;

[0030] Modulation features are obtained through cross-attention. :

[0031]

[0032] in, The dimension of the key;

[0033] Then, calculate the query. ,key ,value :

[0034]

[0035]

[0036]

[0037] in, This indicates pixel-by-pixel multiplication;

[0038] Finally, the output shows the bi-temporal features of change perception. :

[0039]

[0040] (3) The task-adaptive multi-head gating module generates detection guidance features at the attention head granularity. and description of guiding features

[0041] 3.1) Dual-temporal characteristics of change perception Decompose them into For each attention head, a sigmoid gate is independently applied for both detection and description, resulting in bi-temporal gated features for detection. , And description of dual-temporal gating features , :

[0042]

[0043]

[0044]

[0045]

[0046] in, Indicates sigmoid gating. Indicates global draw pooling. , , , For a learnable parameter matrix, , , , For bias parameters, Attention head number;

[0047] 3.2) Detection of dual-temporal gating features , And description of dual-temporal gating features , Weighted summation yields the detection guidance features. and description of guiding features :

[0048]

[0049]

[0050] in, , These are weight parameters;

[0051] (4) Change detection

[0052] A detection decoder based on a feature pyramid network guides detection towards features. Decoded into a probability map of change Complete change detection;

[0053] (5) Spatial localization prior injection description generation of the detector decoder

[0054] After adaptive pooling, the finest-grained feature map of the detection decoder is encoded into a compact detection token through two layers of MLP. Describing directional features Detection token after linear projection and the detection token after linear projection The splicing yields the change description features :

[0055]

[0056] in, The learnable parameter matrix;

[0057] Describing the characteristics of change The data is fed into a large language model, and a change description is generated by autoregression based on task prompts.

[0058] The objective of this invention is achieved as follows.

[0059] This invention proposes a Prototype-Guided Task-Adaptive Network (PTNet), a unified framework for joint change detection and description, based on a prototype-guided task-adaptive remote sensing image change description method. This invention focuses on change description as the primary task, with change detection providing spatial localization priors. Through three core technical mechanisms, it specifically overcomes two major shortcomings of existing technologies: given a pair of bi-temporal remote sensing images, a pre-trained CLIP visual encoder is used as a shared backbone to extract hierarchical feature representations from layers 6, 12, 18, and 24 of the Transformer; a Prototype-Guided Change-Aware Interaction Module (PG-CAI) ​​performs structured semantic modeling of the bi-temporal features; a Task-Adaptive Multi-Head Gating Module (TAMG) generates detection-guided features and description-guided features at the attention head granularity; a Detection Decoder (FPN) decodes the detection features into a change probability map; and a description decoder (based on Qwen2-1.5B-Instruct, LoRA fine-tuned) encodes the detection features into compact detection tokens, which are then concatenated with the description features and fed into a language model to generate a change description.

[0060] Compared with existing technologies, the core differences and technical effects of this invention are as follows:

[0061] (1) Prototype-guided structured semantic modeling: Unlike existing methods that only perform implicit feature differencing, this patent uses K-means prototype clustering + RBF spatial recovery + prototype modulation cross-temporal attention to enable the model to explicitly perceive the type of change and guide temporal correspondence. Ablation experiments show that BLEU-4 is improved by 1.85 / 1.77 on two datasets, and CIDEr-D is improved by 4.93 / 5.88.

[0062] (2) Task feature decoupling at the attention head level: Unlike existing joint methods that share features or use simple bi-branch designs, TAMG generates differentiated representations for detection and description respectively within a single unified feature space through head-level sigmoid gating, eliminating task conflicts and enabling mutual enhancement between the two. Ablation experiments show that BLEU-4 further improves performance by 1.25 / 1.33, and the detection branch IoU is improved by up to 1.32 compared to dedicated detectors.

[0063] (3) Detection token injection description generation: Unlike existing methods that lack effective spatial prior transfer between detection and description, this invention encodes the finest-grained feature map of FPN into a detection token and concatenates it with the description feature, so that the language model can explicitly know the spatial distribution of the change region, effectively eliminating the illusion of location error, and further improving BLEU-4 by 0.91 / 0.93.

[0064] (4) Overall effect: With 165.71M parameters (significantly less than the 300M+ of the comparison method), the present invention surpasses the existing best method in all 7 indicators on two benchmark datasets, achieving the dual advantages of accuracy and efficiency. Attached Figure Description

[0065] Figure 1 This is the overall architecture diagram of the prototype-guided task-adaptive remote sensing image change description method of the present invention;

[0066] Figure 2 This is a flowchart of a specific implementation of the prototype-guided task-adaptive remote sensing image change description method of the present invention;

[0067] Figure 3 yes Figure 1 The diagram shown illustrates the principle of prototype-guided change-aware interaction, where (a) represents a learnable prototype library. (b) is a schematic diagram of the construction process and a schematic diagram of the prototype-guided change-aware interaction module structure.

[0068] Figure 4 This is a comparative graph showing the experimental results of the remote sensing image change description method of this invention and existing methods on the datasets WHU-CDC and UCCD. Detailed Implementation

[0069] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0070] Figure 1 This is an overall architecture diagram of the prototype-guided task-adaptive remote sensing image change description method of the present invention.

[0071] In this embodiment, the present invention proposes a Prototype-GuidedTask-Adaptive Network (PTNet), a unified framework for joint change detection and description. The method of the present invention takes change description as the main task and change detection as the spatial localization prior, and overcomes the two major shortcomings of the prior art through three core technical mechanisms.

[0072] Specifically, such as Figure 1 As shown, the overall architecture of PTNet is as follows: Given a pair of bi-temporal remote sensing images, a pre-trained CLIP visual encoder (ViT-L / 14, fine-tuned with LoRA) is used as a shared backbone to extract hierarchical feature representations from layers 6, 12, 18, and 24 of the Transformer. The Prototype-Guided Change-Aware Interaction (PG-CAI) ​​module performs structured semantic modeling of the bi-temporal features; the Task Adaptive Multi-Head Gating (TAMG) module generates detection-oriented features and description-oriented features at the attention head granularity; the Detection Decoder (FPN) decodes the detection features into a change probability map; the Description Decoder (based on Qwen2-1.5B-Instruct, fine-tuned with LoRA) encodes the detection features into compact detection tokens, concatenates them with the description features, and feeds them into the language model to generate a change description; finally, a visual-language semantic alignment loss based on InfoNCE enforces cross-modal consistency.

[0073] Figure 2 This is a flowchart of a specific implementation of the prototype-guided task-adaptive remote sensing image change description method of the present invention.

[0074] In this embodiment, as Figure 2 As shown, the prototype-guided task-adaptive remote sensing image change description method of the present invention includes the following steps:

[0075] Step S1: Obtain dual-temporal features

[0076] like Figure 1 As shown, in this embodiment, for a given pair of dual-temporal remote sensing images... A pre-trained CLIP visual encoder is used as a shared backbone to extract hierarchical feature representations from layers 6, 12, 18, and 24 of the Transformer. As a dual-temporal feature of the four layers, among which... These represent the 6th, 12th, 18th, and 24th Transformer layers, respectively. .

[0077] Step S2: The Prototype-Guided Change-Aware Interaction (PG-CAI) ​​module processes bi-temporal features. Perform structured semantic modeling

[0078] To address the deficiency of implicit feature difference in lacking structured semantic modeling of changes, PG-CAI explicitly models semantic changes through a learnable prototype library of change types, guiding cross-temporal attention to focus on semantically relevant regions.

[0079] Step S2.1: Prototype Library Initialization

[0080] In this embodiment, as Figure 3 As shown in (a), the dimension is initialized offline using the training set. prototype library ,in, For the number of prototype clusters, For the number of space tokens, For feature dimensions:

[0081] First, for the training set... The training samples are dual-temporal remote sensing images. The mid-level features extracted using the 12th layer of the CLIP visual encoder Calculate the two-phase difference characteristics :

[0082]

[0083] Then, for the changing samples, mask-guided dual-temporal difference features are used. Average pooling of the changed regions yields a compact representation, while for unchanged samples, the bi-temporal difference features are directly applied. Average pooling yields compact representation .

[0084] Then, a compact representation of all training samples. Perform K-means clustering to achieve natural coverage. One prototype cluster, i.e., one unchanging mode and Different types of variations, and based on compact representation Weights are assigned based on the distance to each prototype cluster, with closer clusters receiving larger weights. Weights .

[0085] To recover the spatial structure discarded by the pooling operation, radial basis functions (RBF) are used to compactly represent the aggregated vectors. Re-unfold into a dense spatial map .

[0086] Then, by weight By aggregating spatial features, the initial prototype is obtained. :

[0087]

[0088] in, This represents the number of training samples.

[0089] Finally, we obtained the prototype library. :

[0090]

[0091] Prototype Library It can be used as a learnable parameter for joint optimization with downstream tasks.

[0092] Step S2.2: Transtemporal Interaction of Prototype Modulation

[0093] First, the dual-temporal features are fused. and its difference Build change query :

[0094]

[0095] in, This represents a multilayer perceptron;

[0096] Then, calculate the query. ,key as well as :

[0097]

[0098] in, , , For learnable parameter matrix, express Flatten ;

[0099] In this embodiment, as Figure 3 As shown in (b), modulation features are obtained through cross-attention. :

[0100]

[0101] in, The dimension of the bond. Modulation features. The semantic priors of the change type to which each position belongs are encoded.

[0102] Then, calculate the query. ,key ,value :

[0103]

[0104]

[0105]

[0106] in, This indicates pixel-by-pixel multiplication.

[0107] Finally, in this embodiment, as Figure 3 As shown in (b), modulation characteristics Key-value projection of bidirectional transtemporal attention is modulated by element-wise multiplication: In the direction of For querying, based on modulation features Modulated As a key value; The system performs symmetrical processing and preserves the original features through residual connections, outputting change-aware bi-temporal features. :

[0108]

[0109] Step S3: The Task-Adaptive Multi-head Gating (TAMG) module generates detection guidance features at the attention head granularity. and description of guiding features

[0110] To address the conflict between the feature requirements of detection and description tasks, TAMG learns task-specific gating weights at the attention head granularity, thus simultaneously satisfying the different needs of spatial accuracy (detection) and semantic abstraction (description) within a unified feature space.

[0111] Step S3.1: Obtain the detection features of dual-temporal gating and describe the features of dual-temporal gating.

[0112] like Figure 1 As shown, for the dual-temporal characteristics of change perception Decompose them into For each attention head, a sigmoid gate is independently applied for both detection and description, resulting in bi-temporal gated features for detection. , And description of dual-temporal gating features , :

[0113]

[0114]

[0115]

[0116]

[0117] in, Indicates sigmoid gating. Indicates global draw pooling. , , , For a learnable parameter matrix, , , , For bias parameters, This is the attention header number.

[0118] Step S3.2: Obtain detection guidance features and description of guiding features

[0119] For detecting dual-temporal gating features , And description of dual-temporal gating features , Weighted summation yields the detection guidance features. and description of guiding features :

[0120]

[0121]

[0122] in, , These are the weight parameters.

[0123] Detection guidance features and description of guiding features Both share underlying change-aware features, but at the head level, they achieve task-specific representation decoupling through independent gating, which keeps the detection branch spatially sensitive and the description branch semantically abstract.

[0124] Step S4: Change Detection

[0125] A detection decoder based on a feature pyramid network guides detection towards features. Decoded into a probability map of change Complete the change detection.

[0126] In this embodiment, binary cross-entropy loss is used for supervised training for change detection.

[0127] Step S5: Spatial localization prior injection description generation for the detector decoder

[0128] To inject the spatial localization prior of the detection branch into the description generation, the finest-grained feature map of the detection decoder (preserving the richest spatial details in the changing regions) is adaptively pooled and then encoded into a compact detection token through two layers of MLP. Describing directional features Detection token after linear projection and the detection token after linear projection The splicing yields the change description features :

[0129]

[0130] in, This is the learnable parameter matrix.

[0131] Change Description Features The concatenation strategy is used to construct the large language model, enabling it to simultaneously focus on semantic variation features (derived from description-oriented features). ) and explicit spatial location information (from the detection token) It encodes the shape, extent, and spatial distribution of the changing region, thereby generating a change description that is both semantically accurate and spatially faithful.

[0132] Describing the characteristics of change The data is fed into a large language model, and a change description is generated by autoregression based on task prompts.

[0133] In this embodiment, the change describes the feature. It is fed into Qwen2-1.5B-Instruct (fine-tuned via LoRA), and a change description is generated by autoregression based on task prompts.

[0134] Visual-linguistic semantic alignment

[0135] In this embodiment, to align the visual representation of the description branch with the CLIP semantic space, InfoNCE contrastive loss is applied. Text embeddings are obtained using the frozen CLIP text encoder. The last hidden state of the language model is then average-pooled and linearly mapped to the same dimension to obtain the visual embeddings. InfoNCE contrastive loss is used to bring the paired visual-text representations closer together between positive and negative samples within a batch. The significant improvement in the METEOR metric validates that this objective function enables the model to learn synonym-aware semantic representations.

[0136] Figure 4 This is a comparative graph of the experimental results of the remote sensing image change description of the present invention and existing methods on the datasets WHU-CDC and UCCD. For each example, we provide the dual-temporal image pair, ground truth mask (cd-GT), prediction mask (cd-infer), and change description of each method. Error descriptions are highlighted in red.

[0137] from Figure 4As can be seen, existing methods often produce illusions or inaccurate spatial descriptions, such as misidentifying the location of changes or confusing the type of change. Our invention, however, generates more precise and spatially accurate descriptions of changes. This qualitative advantage stems from the spatial prior injected into the detection branch via a mask encoder. Clearly, our invention is more robust even in challenging conditions, such as partially occluded structures and densely built-up areas, whereas existing methods often suffer from location errors or illusory object descriptions.

[0138] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A prototype-guided task-adaptive remote sensing image change description method, characterized in that, Includes the following steps: (1) Obtaining dual-temporal features ; (2) Prototype-guided change-aware interaction module for dual-temporal features Perform structured semantic modeling; 2.1) Initialize the prototype library; Offline initialization of the training set dimension is prototype library ,in, For the number of prototype clusters, For the number of space tokens, For feature dimensions: First, for the training set... The training samples are dual-temporal remote sensing images. The mid-level features extracted using the 12th layer of the CLIP visual encoder Calculate the two-phase difference characteristics : ； Then, for the changing samples, mask-guided dual-temporal difference features are used. Average pooling of the changed regions yields a compact representation, while for unchanged samples, the bi-temporal difference features are directly applied. Average pooling yields compact representation ; Then, a compact representation of all training samples. Perform K-means clustering to achieve natural coverage. One prototype cluster, i.e., one unchanging mode and Different types of variations, and based on compact representation Weights are assigned based on the distance to each prototype cluster, with closer clusters receiving larger weights. Weights Radial basis function interpolation is used to compactly represent the aggregated vector. Re-unfold into a dense spatial map ; Then, by weight By aggregating spatial features, the initial prototype is obtained. : ； in, This represents the number of training samples; Finally, we obtained the prototype library. : ； Prototype Library As a learnable parameter, it can be jointly optimized with downstream tasks; 2.2) Trans-temporal interaction of prototype modulation; First, the dual-temporal features are fused. and its difference Build change query : ； in, This represents a multilayer perceptron; Then, calculate the query. ,key as well as : ； in, , , For learnable parameter matrix, express Flatten ; Modulation features are obtained through cross-attention. : ； in, The dimension of the key; Then, calculate the query. ,key ,value : ； ； ； in, This indicates pixel-by-pixel multiplication; Finally, the output shows the bi-temporal features of change perception. : ； (3) The task-adaptive multi-head gating module generates detection guidance features at the attention head granularity. and description of guiding features ; 3.1) Dual-temporal characteristics of change perception Decompose them into For each attention head, a sigmoid gate is independently applied for both detection and description, resulting in bi-temporal gated features for detection. , And description of dual-temporal gating features , : ； ； ； ； in, Indicates sigmoid gating. Indicates global draw pooling. , , , For a learnable parameter matrix, , , , For bias parameters, Attention head number; 3.2) Detection of dual-temporal gating features , And description of dual-temporal gating features , Weighted summation yields the detection guidance features. and description of guiding features : ； ； in, , These are weight parameters; (4) Change detection; A detection decoder based on a feature pyramid network guides detection towards features. Decoded into a probability map of change Complete change detection; (5) Detect the spatial localization prior injection description generation of the decoder; After adaptive pooling, the finest-grained feature map of the detection decoder is encoded into a compact detection token through two layers of MLP. Describing directional features Detection token after linear projection and the detection token after linear projection The splicing yields the change description features : ； in, The learnable parameter matrix; Describing the characteristics of change The data is fed into a large language model, and a change description is generated by autoregression based on task prompts.

2. The prototype-guided task-adaptive remote sensing image change description method according to claim 1, characterized in that, Step (1) involves obtaining dual-temporal features. for: For a given pair of two-temporal remote sensing images A pre-trained CLIP visual encoder is used as a shared backbone to extract hierarchical feature representations from layers 6, 12, 18, and 24 of the Transformer. As a dual-temporal feature of the four layers, among which... These represent the 6th, 12th, 18th, and 24th Transformer layers, respectively. .

Images