A change detection and semantic description method for dual-temporal remote sensing images

Abstract

The present application is to solve the technical problem that the existing remote sensing image change detection method lacks deep mutual promotion of semantic features and visual features, and the fusion effect is not good, and proposes a dual-time-phase remote sensing image change detection and semantic description method, including the following steps: acquiring dual-time-phase remote sensing images; feature extraction is performed through a double-branch feature extraction network to obtain high-level semantic features and multi-scale visual features; the features are integrated and bidirectionally modulated to obtain optimized cross-modal features; the optimized cross-modal features are respectively subjected to semantic enhancement fusion and change detection decoding to generate natural language description of the change content of the dual-time-phase remote sensing images; a double-task joint loss function is constructed to determine an optimization model, the to-be-detected dual-time-phase remote sensing images are input into the trained optimization model, and a pixel-level change region mask corresponding to the dual-time-phase remote sensing images and the natural language description of the change content are output to complete the change detection and semantic description of the dual-time-phase remote sensing images.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image change detection technology, specifically to a method for change detection and semantic description of dual-temporal remote sensing images. Background Technology

[0002] Remote sensing image change detection is one of the core research directions of time-series remote sensing analysis. It uses computer vision technology to process remote sensing images of the same geographic area at different time series to accurately obtain two key types of information: "change location" and "change attributes". It has important practical application value in many fields such as land planning, disaster assessment, and ecological monitoring.

[0003] Existing remote sensing image change detection methods treat change detection and image description as independent tasks. Because they lack a collaborative relationship, the spatial localization of changed areas and semantic understanding information become independent, requiring separate processing of change masks and textual semantics. This makes it difficult to meet the practical application requirements for integrated change detection and semantic description, increasing operational costs and affecting the ease of application. Secondly, in the cross-modal feature fusion stage, existing remote sensing image change detection methods often employ "one-way modulation" or "simple stitching," lacking deep mutual reinforcement between semantic and visual features. Due to the spatial ambiguity of semantic features and the lack of semantic correlation in visual features, the semantic expression of changed areas becomes disconnected from spatial localization, resulting in poor fusion performance.

[0004] Secondly, existing technologies struggle to fully capture cross-temporal change information when processing long sequence semantic features, and their methods for mining spatial dependencies are relatively limited, failing to comprehensively depict the spatial associations and temporal evolution patterns of changing regions in remote sensing images from different time series. This further impacts the accuracy of change detection and the completeness of semantic description. Summary of the Invention

[0005] The purpose of this invention is to address the technical problems of existing remote sensing image change detection methods, such as lack of deep interaction between semantic and visual features, poor fusion effect, low change detection accuracy, and low semantic description completeness. This invention proposes a change detection and semantic description method for dual-temporal remote sensing images.

[0006] To achieve the above objectives, the technical solution provided by this invention is as follows:

[0007] A method for change detection and semantic description of dual-temporal remote sensing images, characterized by:

[0008] Step 1: Acquire dual-temporal remote sensing images; the dual-temporal remote sensing images include a first temporal image T1 and a second temporal image T2 at different time points in the same geographical area;

[0009] A change detection and semantic description model for dual-temporal remote sensing images is constructed. The model includes a dual-branch feature extraction network, a bidirectional modulation module, a semantic enhancement fusion module, a visual decoder, and a text generation decoder.

[0010] Step 2: The first time phase image T1 and the second time phase image T2 are extracted by a dual-branch feature extraction network. The dual-branch feature extraction network includes a parallel Clip-ViT32 network and a ResNet50 network, which respectively obtain the high-level semantic features and multi-scale visual features of the first time phase image T1 and the second time phase image T2.

[0011] Step 3: The acquired high-level semantic features and multi-scale visual features are integrated and bidirectionally modulated through the bidirectional modulation module to obtain optimized cross-modal features;

[0012] Step 4: The optimized cross-modal features are subjected to semantic enhancement fusion and change detection decoding through the semantic enhancement fusion module and the change detection decoding module, respectively, to obtain the semantic enhancement fusion features and the pixel-level change region mask;

[0013] Step 5: Input the semantic enhancement fusion features into the text generation decoder for decoding to generate a natural language description of the changes in the dual-temporal remote sensing images;

[0014] Step 6: Based on the pixel-level change region mask and the natural language description of the change content in the dual-temporal remote sensing images, construct a dual-task joint loss function to determine whether the pixel-level change detection accuracy and text description quality simultaneously meet the preset requirements. If they do not meet the requirements simultaneously, update the model parameters based on the dual-task joint loss function and return to Step 2 for re-iteration. If they meet the requirements simultaneously, complete the model training and determine the optimized model.

[0015] The pixel-level change detection accuracy is the pixel error between the pixel-level change region mask and the actual change mask; the text description quality is the word-by-word prediction error between the natural language description of the change content of the dual-temporal remote sensing image and the actual labeled text.

[0016] Step 7: Input the dual-temporal remote sensing image to be tested into the trained optimization model, and output the pixel-level change region mask and natural language description of the change content corresponding to the dual-temporal remote sensing image, thus completing the change detection and semantic description of the dual-temporal remote sensing image.

[0017] Furthermore, in step 2, the Clip-ViT32 network includes a CLIP semantic encoder, which uses a ViT-32 visual transformer as an image encoder to map images and text to the same semantic space.

[0018] The ResNet50 network includes an input layer, an initial convolutional layer, a pooling layer, and a ResNet50 visual encoder consisting of multi-level stacked residual blocks. It is used to extract deep features of the first temporal image T1 and the second temporal image T2 layer by layer and output multi-scale visual features including image feature vectors.

[0019] Step 2 is as follows:

[0020] Step 2.1: The CLIP semantic encoder in the dual-branch feature extraction network divides the first phase image T1 and the second phase image T2 into image block sequences of a set size. After linear projection and position encoding, semantic features are extracted and high-level semantic features are output.

[0021] Step 2.2: The ResNet50 visual encoder in the dual-branch feature extraction network performs multi-scale visual feature extraction on the first phase image T1 and the second phase image T2 respectively, and outputs multi-scale visual features, which include visual features at four different scales.

[0022] Further, step 3 includes:

[0023] Step 3.1: Calculate the differences in visual features at four different scales. Generate multi-scale, cross-temporal phase difference feature maps;

[0024] in, The visual features of the first phase image T1 at the i-th scale are... Let T2 be the visual feature of the second temporal image T2 at the i-th scale, i=1,2,3,4;

[0025] Step 3.2: Convert the multi-scale cross-temporal difference feature map into a spatial variation mask using the sigmoid activation function;

[0026] Step 3.3: Multiply the spatial variation mask element-wise with the high-level semantic features to modulate the spatial response intensity of the high-level semantic features, and obtain the initially modulated semantic features. Use the initially modulated semantic features to perform feedback enhancement on the multi-scale visual features to achieve bidirectional modulation, and obtain the enhanced multi-scale visual features. Then, perform cross-modal fusion between the enhanced multi-scale visual features and the initially modulated semantic features to obtain the optimized cross-modal features.

[0027] Furthermore, step 4, the step of obtaining the semantically enhanced fusion features, includes:

[0028] Step a.1: Using the optimized cross-modal features as input, add 2D positional encoding to the high-level semantic features of the first temporal image T1 and the second temporal image T2 respectively to generate enhanced dual-temporal semantic features. , ;

[0029] Step a.2: Calculate the cosine similarity matrix between two semantic features to generate a semantic change mask.

[0030] in, This is the function for calculating cosine similarity.

[0031] Step a.3, design a differentiated spatial perspective rearrangement strategy, and... , The features were rearranged from four different perspectives to obtain four sets of feature sequences with different spatial orders.

[0032] Step a.4: Input the four sets of feature sequences with different spatial orders into the Mamba model, extract the feature association information from different spatial perspectives, and then concatenate and fuse the feature association information to obtain semantically enhanced fusion features.

[0033] Furthermore, step 4, the step of obtaining the pixel-level variation region mask, includes:

[0034] Step b.1: Using the optimized cross-modal features as input, the visual features at different levels are fused from the bottom up through a multi-scale cross-temporal feature fusion change detection decoder.

[0035] Step b.2: During the fusion process, the visual features are enhanced. Channel attention weights are obtained by adaptively weighting the channel dimensions of the visual features using a channel attention branch, and spatial attention weights are obtained by responding to the spatial dimensions using a spatial attention branch. Finally, the enhanced visual feature F is obtained. enhance :

[0036] ;

[0037] in, For channel attention weights, is the spatial attention weight; F is the tensor describing the features in the network;

[0038] Step b.2: The enhanced visual features are mapped to a single-channel output through a convolutional layer to obtain a pixel-level change region mask.

[0039] Furthermore, in step 5, the text generation decoder includes multiple decoding layers and a built-in text embedding feature semantic module. Each decoding layer includes a masked self-attention layer, a cross-attention layer, a feedforward network, and a linear projection layer. The masked self-attention layer and the cross-attention layer are information extraction layers used to capture context. The feedforward network layer is a linear transformation layer used for information enhancement. The linear projection layer is used to map the hidden state to the vocabulary.

[0040] Step 5 specifically includes:

[0041] Step 5.1: Flatten the semantic enhancement fusion features into sequence features in the spatial dimension, input them into the text generation decoder, and align them with the built-in text embedding features semantically through the linear projection layer of the text generation decoder.

[0042] Step 5.2: The text generation decoder captures the contextual dependencies within the text sequence through its masked self-attention layer, and the aligned semantic enhancement fusion features are fused with the text embedding features built into the decoder through the cross-attention mechanism in the cross-attention layer to obtain cross-modal latent state features.

[0043] Step 5.3: Select text sequences from cross-modal latent state features using a preset sampling strategy to generate a natural language description of the changes in the dual-temporal remote sensing images.

[0044] Furthermore, in step 6, the joint loss function for the two tasks is:

[0045] loss_total = loss_cap + λ × loss_mask

[0046] Where loss_cap is the text generation loss; loss_mask is the change detection loss; and λ is the weighting coefficient of the change detection loss.

[0047] Furthermore, in step a.1, the 2D position encoding is formed by concatenating the position vectors in the horizontal direction w and the vertical direction h, wherein the position vectors in both the w and h directions have a dimension of 384, and the total encoding dimension is consistent with the semantic feature dimension.

[0048] Furthermore, in step a.3, the differentiated spatial perspective rearrangement strategy includes the original spatial order, the horizontally flipped spatial order, the vertically flipped spatial order, and the diagonally flipped spatial order.

[0049] Compared with the prior art, the present invention has the following beneficial technical effects:

[0050] This invention provides a method for change detection and semantic description of dual-temporal remote sensing images. By integrating a training architecture that combines bi-branch feature extraction, cross-modal bidirectional modulation, and multi-view Mamba fusion, it achieves joint training of change detection and semantic description of dual-temporal remote sensing images. This eliminates the need for additional cross-task correlation processing and directly completes feature complementarity and collaborative optimization during training. This effectively ensures the accuracy of pixel-level change region localization and the logical consistency of natural language description, significantly reducing the cost of manual interpretation of change information in remote sensing images. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating an embodiment of the method for change detection and semantic description of dual-temporal remote sensing images according to the present invention.

[0052] Figure 2 This is a diagram showing the effect of an embodiment of the method for change detection and semantic description of dual-temporal remote sensing images of the present invention on the WHU-CC dataset. Detailed Implementation

[0053] To make the objectives, advantages, and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0054] like Figure 1 As shown, the present invention provides a method for change detection and semantic description of dual-temporal remote sensing images, comprising the following steps:

[0055] Step 1: Acquire dual-temporal remote sensing images, which include a first temporal image T1 and a second temporal image T2 of the same geographic area at different time points; the two images have been preprocessed by geometric registration and radiometric normalization, and the size is uniformly 256×256 pixels.

[0056] A change detection and semantic description model for dual-temporal remote sensing images is constructed. The model includes a dual-branch feature extraction network, a bidirectional modulation module, a semantic enhancement fusion module, a visual decoder, and a text generation decoder.

[0057] A dual-branch feature extraction network independently extracts features from the preceding and following temporal remote sensing images, outputting multi-scale visual features for both images. A bidirectional modulation module receives the two sets of temporal features output from the dual branches, enabling mutual guidance and constraint between the two temporal features. A semantic enhancement and fusion module fuses and enhances the dual-temporal change features, providing feature inputs to the two subsequent decoding branches. A visual decoder receives the change features output from the semantic enhancement and fusion module, upsamples them to restore the image spatial resolution, and outputs a pixel-level change prediction mask to complete the change detection of the dual-temporal remote sensing images. A text generation decoder performs cross-modal decoding based on the visual fusion features, generating a natural language description of the change content.

[0058] Step 2: The first time-phase image T1 and the second time-phase image T2 are extracted using a dual-branch feature extraction network to obtain the high-level semantic features and multi-scale visual features corresponding to the first time-phase image T1 and the second time-phase image T2 respectively; wherein, the dual-branch feature extraction network includes a parallel Clip-ViT32 network and a ResNet50 network.

[0059] The Clip-ViT32 network includes a trained CLIP semantic encoder, which uses the ViT-32 visual transformer as an image encoder to map images and text to the same semantic space. The ResNet50 network includes an input layer, an initial convolutional layer, a pooling layer, and multi-level stacked residual blocks, which are used to extract deep features of the first temporal image T1 and the second temporal image T2 layer by layer and output multi-scale visual features including image feature vectors.

[0060] The specific process for extracting multi-scale visual features is as follows: The CLIP semantic encoder in the Clip-ViT32 network adopts a pre-trained ViT-B / 32 model, which divides the first phase image T1 and the second phase image T2 into 7×7 image patch sequences. After linear projection and position encoding, the patches are input into the Transformer encoder for semantic feature extraction, and the output is a high-level semantic feature with dimensions [B, 49, 768], where B is the training batch size, 49 is the number of image patches, and 768 is the feature dimension.

[0061] The ResNet50 visual encoder in the ResNet50 network extracts multi-scale visual features from the first temporal image T1 and the second temporal image T2 through four stacked feature extraction blocks (stage1~stage4), outputting four visual feature maps at different scales: F1 (256 channels, spatial resolution H / 4×W / 4), F2 (512 channels, spatial resolution H / 8×W / 8), F3 (1024 channels, spatial resolution H / 16×W / 16), and F4 (2048 channels, spatial resolution H / 32×W / 32), where H and W are the height and width of the input remote sensing image, respectively.

[0062] Step 3: Integrate the acquired high-level semantic features and multi-scale visual features and perform bidirectional modulation to obtain optimized cross-modal features;

[0063] The specific process is as follows: Calculate the differences in visual features across different scales and temporal phases output by the ResNet50 visual encoder. (i=1,2,3,4), generate multi-scale cross-temporal phase difference feature maps;

[0064] in, The visual features of the first phase image T1 at the i-th scale are... Let T2 be the visual feature of the second temporal image T2 at the i-th scale, i=1,2,3,4;

[0065] The differential feature map is converted into a spatial variation mask M using the sigmoid activation function. i (i=1,2,3,4);

[0066] The spatial variation mask M i Element-wise multiplication with CLIP high-level semantic features modulates the spatial response intensity of the high-level semantic features, yielding preliminary modulated semantic features.

[0067] To achieve bidirectional modulation, the semantic features obtained from the initial modulation are further used to enhance the multi-scale visual features. Specifically, the semantic features obtained from the initial modulation are passed through a 1×1 convolutional layer to compress the number of channels to a dimension that matches the visual features at each scale. Then, upsampling or pooling operations are used to adjust the spatial size to match the dimensions of each feature. , Consistent, after generating the semantic modulation weight map, the original visual features are multiplied element by element with the semantic modulation weight map to obtain the enhanced multi-scale visual features;

[0068] The initially modulated semantic features and the enhanced visual features are fused across modally (concatenated along the channel dimension and then subjected to 3×3 convolution for dimensionality reduction) to obtain the optimized cross-modal feature F. embed .

[0069] Step 4: Perform semantic enhancement fusion and change detection decoding on the optimized cross-modal features to obtain semantic enhancement fusion features and pixel-level change region masks;

[0070] The specific process for obtaining semantically enhanced fusion features is as follows: 2D positional encoding is added to the CLIP high-level semantic features of the optimized first temporal image T1 and the second temporal image T2. The 2D positional encoding is composed of position vectors in the horizontal direction (w) and the vertical direction (h), where the dimensions of the position vectors in the w and h directions are both 384. The total encoding dimension is consistent with the semantic feature dimension (768), thus realizing the fusion of spatial information and semantic features to generate semantic features. and .

[0071] Calculate the cosine similarity matrix between the optimized semantic features of the first time-phase image T1 and the second time-phase image T2, and generate a semantic change mask based on this matrix.

[0072]

[0073] in M is the cosine similarity calculation function. The larger the value of M, the more significant the semantic change in the corresponding region. M is used to weight and enhance the semantic features, thereby strengthening the information about the changes.

[0074] Four differentiated spatial perspective rearrangement strategies were designed, including the original spatial order, horizontally flipped spatial order, vertically flipped spatial order, and diagonally flipped spatial order. The enhanced bi-temporal semantic features were rearranged according to the four perspectives to obtain four sets of feature sequences with different spatial orders.

[0075] The four rearranged feature sequences were input into the Mamba model. The Mamba model was used to efficiently capture long sequence dependencies and extract feature association information from different spatial perspectives. The output features of the four Mamba models were concatenated and fused. Channel dimension compression and information integration were performed through a 1×1 convolutional layer to finally obtain a semantically enhanced fusion feature with dimensions [B,768,7,7].

[0076] In the process of obtaining the pixel-level change region mask, the visual decoder contains 4-level decoding units and 1 dual-channel attention-guided change perception enhancement module. The process adopts a "bottom-up" multi-scale fusion strategy. The 8 feature maps generated by the ResNet50 visual encoder from the two remote sensing images are used as the input of the decoder. The two feature maps in the same stage are subtracted, convolved and normalized to obtain the supplementary feature map generated in this stage.

[0077] First, the channel dimension of the visual features is adaptively weighted through a channel attention branch to enhance the expression of effective feature channels and suppress redundant channels. Then, the spatial dimension of the feature map is responded to through a spatial attention branch to generate a spatial attention weight map, focusing on regions of cross-temporal change. Finally, the enhanced visual features are output. The calculation process is as follows:

[0078] Channel attention: , where σ is the sigmoid activation function and MLP is a multilayer perceptron;

[0079] Spatial attention: Where Conv() is a two-dimensional convolution operation and Concat() is a concatenation operation;

[0080] Enhanced visual features: ;

[0081] Decoding process: The deepest feature F4 (T1 and T2 phases) output by the ResNet50 visual encoder is used as the initial input. After being enhanced by the dual-channel attention-guided change perception enhancement module, it is upsampled by 2 times through transposed convolution and fused with the previous layer feature F3 (T1 and T2 phases) across time phases. The above "enhancement-upsampling-fusion" process is repeated to fuse F2 and F1 features in turn. Finally, the fused features are mapped to a single-channel output through a 1×1 convolutional layer to obtain a pixel-level change region mask with dimensions [B, 1, 256, 256]. The mask value range is [0, 1], which represents the probability that the corresponding pixel belongs to the change region.

[0082] Step 5: Input the semantic enhancement fusion features into the text generation decoder for decoding to generate a natural language description of the changes in the dual-temporal remote sensing images;

[0083] The specific process is as follows:

[0084] The semantically enhanced fusion feature [B,768,7,7] obtained in step 4 is flattened into a sequence feature (B,49,768) in the spatial dimension and input into the linear projection layer of the text generation decoder, which maps it to the same dimension as the built-in text embedding of the decoder (e.g., 512 dimensions), thereby achieving semantic alignment between visual features and text features.

[0085] The text generation decoder adopts a Transformer decoder structure, including a masked self-attention layer, a cross-attention layer, a feedforward network, a linear projection layer, and built-in text embedding feature semantics; the self-attention layer and the cross-attention layer are used for information extraction and capturing context; the feedforward network layer is used for information enhancement; the linear projection layer is the output layer of the text generation decoder, which is used to map the hidden state to the vocabulary;

[0086] The masked self-attention layer is used to capture the contextual dependencies within the generated text sequence; the cross-attention layer uses the aligned visual sequence features as keys and values, and the current hidden state of the text decoder as a query, to achieve deep fusion of visual and text features and output cross-modal hidden state features.

[0087] A pre-defined sampling strategy (such as cluster search or greedy search) is used to predict the output sequence word by word from cross-modal latent state features. At each time step, the probability distribution on the vocabulary is calculated through a linear layer and a softmax function. The word with the highest probability is selected as the optimal text sequence until an end marker is generated. Finally, a natural language description of the changes in the dual-temporal remote sensing images is obtained, such as "new building areas and reduced vegetation area".

[0088] Step 6: Based on the pixel-level change region mask and the natural language description of the change content in the dual-temporal remote sensing images, construct a dual-task joint loss function, specifically as follows:

[0089] loss_total = loss_cap + λ × loss_mask

[0090] Wherein, loss_cap is the text generation loss, which uses the cross-entropy loss function to calculate the word-by-word prediction error between the generated text sequence and the real text sequence; loss_mask is the change detection loss, which uses the Dice-BCE joint loss function; λ is the weight coefficient of the change detection loss, which is set to 1 in this model and can be dynamically adjusted according to the training accuracy of the dual task.

[0091] Based on the dual-task joint loss function, determine whether the pixel-level change detection accuracy and text description quality simultaneously meet the preset requirements. If they do not meet the requirements simultaneously, update the network parameters (network parameters of the dual-branch feature extraction network, bidirectional modulation module, semantic enhancement fusion module, change detection decoding module, and text generation decoder) based on the dual-task joint loss function. Return to step 2 and re-execute. If they meet the requirements simultaneously, complete the training and determine the optimized model. The condition for meeting the requirements can be that after each training cycle, if the calculated verification loss_mask is lower than the first preset threshold (e.g., 0.05) and the loss_cap is lower than the second preset threshold (e.g., 0.5), then the pixel-level change detection accuracy and text description quality are considered to meet the preset requirements simultaneously.

[0092] The pixel-level change detection accuracy is the pixel error between the pixel-level change region mask and the actual change mask; the text description quality is the word-by-word prediction error between the natural language description of the change content in the dual-temporal remote sensing image and the actual labeled text.

[0093] The performance evaluation metrics for the optimization model include:

[0094] Change detection task: Intersection over union (IoU) and F1 score are used as evaluation indicators.

[0095] Text generation task: BLEU-1, BLEU-2, BLEU-3, BLEU-4, METEOR, ROUGE-L, and CIDEr are used as evaluation metrics to comprehensively measure the accuracy, fluency, richness, and semantic consistency of the generated text.

[0096] During model training, the model with the best BLEU-4 score for the text generation task and the model with the best mIoU score for the change detection task are saved separately for subsequent inference.

[0097] like Figure 2As shown, the method of this invention achieves high performance on the WHU-CC dataset, with the outlines of building changes closely matching the ground truth map: During T1 to T2 (upper half), a small building is added to the upper left corner of the scene. This building is attached to an existing building. Both the ground truth mask and the model prediction results accurately locate the newly added building area in the upper left corner, with a high degree of consistency between shape and position, indicating that the model performs accurately at the visual segmentation level. The model generates a description of "a house is built in the bottom of the scene." Although there is a deviation in the location description (mistakenly describing "upper left corner" as "bottom"), it accurately identifies the core change event of "newly built house," which is highly consistent with the core semantics of the manual label, such as "a small building attached to an existing one appears." During T1 to T2 (lower half), there are two changes in the scene: first, a white-roofed house on the left has been expanded; second, a small building has been added nearby, and the original construction area has been transformed into houses and parking lots. The ground truth mask contains two independent white areas, and the model's predictions completely cover both changes, with good shape and boundary fit, demonstrating the model's ability to segment multi-object changes. The model-generated description, "a building appears in the bottom of the scene," accurately identifies the core change of "building appearing," which is consistent with the semantic direction of the human-generated labels such as "a house has been extended" and "a small house appears nearby," effectively capturing the main change events in the scene.

[0098] In addition, the performance comparison of the proposed method with other methods on the Massachusetts dataset is shown in Table 1:

[0099] Table 1

[0100] Comparative results on the Massachusetts dataset demonstrate that the proposed method significantly outperforms the CTMTN method across all evaluation metrics. In terms of segmentation performance, the proposed method achieves an IoU of 82.6% and an F1 score of 90.1%, representing improvements of 1.1 and 1.2 percentage points respectively compared to CTMTN. This indicates that the enhanced feature fusion mechanism can more accurately capture image content and improve segmentation accuracy.

[0101] In terms of text generation metrics, this invention demonstrates comprehensive and robust performance improvements: all BLEU series metrics (BLEU-1 to BLEU-4) show significant increases, with BLEU-4 showing the largest increase at 7.3 percentage points, indicating a significant enhancement in the semantic matching between the generated text and image content. Meanwhile, METEOR and ROUGE-L each improved by 3.9 percentage points, indicating improvements in both semantic consistency and linguistic fluency of the generated text. Particularly noteworthy is the CIDEr metric, reaching 161.2, an improvement of 11.8 compared to CTMTN, fully validating the effectiveness of this invention in visual-linguistic cross-modal alignment.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A method for change detection and semantic description of dual-temporal remote sensing images, characterized in that: Step 1: Acquire dual-temporal remote sensing images; the dual-temporal remote sensing images include a first temporal image T1 and a second temporal image T2 at different time points in the same geographical area; A change detection and semantic description model for dual-temporal remote sensing images is constructed. The model includes a dual-branch feature extraction network, a bidirectional modulation module, a semantic enhancement fusion module, a visual decoder, and a text generation decoder. Step 2: The first time phase image T1 and the second time phase image T2 are extracted by a dual-branch feature extraction network. The dual-branch feature extraction network includes a parallel Clip-ViT32 network and a ResNet50 network, which respectively obtain the high-level semantic features and multi-scale visual features of the first time phase image T1 and the second time phase image T2. Step 3: The acquired high-level semantic features and multi-scale visual features are integrated and bidirectionally modulated through the bidirectional modulation module to obtain optimized cross-modal features; Step 4: The optimized cross-modal features are subjected to semantic enhancement fusion and change detection decoding through the semantic enhancement fusion module and the change detection decoding module, respectively, to obtain the semantic enhancement fusion features and the pixel-level change region mask; Step 5: Input the semantic enhancement fusion features into the text generation decoder for decoding to generate a natural language description of the changes in the dual-temporal remote sensing images; Step 6: Based on the pixel-level change region mask and the natural language description of the change content in the dual-temporal remote sensing images, construct a dual-task joint loss function to determine whether the pixel-level change detection accuracy and text description quality simultaneously meet the preset requirements. If they do not meet the requirements simultaneously, update the model parameters based on the dual-task joint loss function and return to Step 2 for re-iteration. If they meet the requirements simultaneously, complete the model training and determine the optimized model. The pixel-level change detection accuracy is the pixel error between the pixel-level change region mask and the actual change mask; the text description quality is the word-by-word prediction error between the natural language description of the change content of the dual-temporal remote sensing image and the actual labeled text. Step 7: Input the dual-temporal remote sensing image to be tested into the trained optimization model, and output the pixel-level change region mask and natural language description of the change content corresponding to the dual-temporal remote sensing image, thus completing the change detection and semantic description of the dual-temporal remote sensing image.

2. The method for change detection and semantic description of dual-temporal remote sensing images according to claim 1, characterized in that, In step 2, the Clip-ViT32 network includes a CLIP semantic encoder, which uses a ViT-32 visual transformer as an image encoder to map images and text to the same semantic space. The ResNet50 network includes an input layer, an initial convolutional layer, a pooling layer, and a ResNet50 visual encoder consisting of multi-level stacked residual blocks. It is used to extract deep features of the first temporal image T1 and the second temporal image T2 layer by layer and output multi-scale visual features including image feature vectors. Step 2 is as follows: Step 2.1: The CLIP semantic encoder in the dual-branch feature extraction network divides the first phase image T1 and the second phase image T2 into image block sequences of a set size. After linear projection and position encoding, semantic features are extracted and high-level semantic features are output. Step 2.2: The ResNet50 visual encoder in the dual-branch feature extraction network performs multi-scale visual feature extraction on the first phase image T1 and the second phase image T2 respectively, and outputs multi-scale visual features, which include visual features at four different scales.

3. The method for change detection and semantic description of dual-temporal remote sensing images according to claim 2, characterized in that, Step 3 includes: Step 3.1: Calculate the differences in visual features at four different scales. Generate multi-scale, cross-temporal phase difference feature maps; in, The visual features of the first phase image T1 at the i-th scale are... Let T2 be the visual feature of the second temporal image T2 at the i-th scale, i=1,2,3,4; Step 3.2: Convert the multi-scale cross-temporal difference feature map into a spatial variation mask using the sigmoid activation function; Step 3.3: Multiply the spatial variation mask element-wise with the high-level semantic features to modulate the spatial response intensity of the high-level semantic features, and obtain the initially modulated semantic features. Use the initially modulated semantic features to perform feedback enhancement on the multi-scale visual features to achieve bidirectional modulation, and obtain the enhanced multi-scale visual features. Then, perform cross-modal fusion between the enhanced multi-scale visual features and the initially modulated semantic features to obtain the optimized cross-modal features.

4. The method for change detection and semantic description of dual-temporal remote sensing images according to claim 3, characterized in that, Step 4, the step of obtaining the semantically enhanced fusion features, includes: Step a.1: Using the optimized cross-modal features as input, add 2D positional encoding to the high-level semantic features of the first temporal image T1 and the second temporal image T2 respectively to generate enhanced dual-temporal semantic features. , ; Step a.2: Calculate the cosine similarity matrix between two semantic features to generate a semantic change mask. ; in, This is the function for calculating cosine similarity. Step a.3, design a differentiated spatial perspective rearrangement strategy, and... , The features were rearranged from four different perspectives to obtain four sets of feature sequences with different spatial orders. Step a.4: Input the four sets of feature sequences with different spatial orders into the Mamba model, extract the feature association information from different spatial perspectives, and then concatenate and fuse the feature association information to obtain semantically enhanced fusion features.

5. The method for change detection and semantic description of dual-temporal remote sensing images according to claim 4, characterized in that, Step 4, the step of obtaining the pixel-level change region mask, includes: Step b.1: Using the optimized cross-modal features as input, the visual features at different levels are fused from the bottom up through a multi-scale cross-temporal feature fusion change detection decoder. Step b.2: During the fusion process, the visual features are enhanced. Channel attention weights are obtained by adaptively weighting the channel dimensions of the visual features using a channel attention branch, and spatial attention weights are obtained by responding to the spatial dimensions using a spatial attention branch. Finally, the enhanced visual feature F is obtained. enhance : ； in, For channel attention weights, is the spatial attention weight; F is the tensor describing the features in the network; Step b.2: The enhanced visual features are mapped to a single-channel output through a convolutional layer to obtain a pixel-level change region mask.

6. The method for change detection and semantic description of dual-temporal remote sensing images according to claim 5, characterized in that, In step 5, the text generation decoder includes multiple decoding layers and a built-in text embedding feature semantic module. Each decoding layer includes a masked self-attention layer, a cross-attention layer, a feedforward network, and a linear projection layer. The masked self-attention layer and the cross-attention layer are information extraction layers used to capture context. The feedforward network layer is a linear transformation layer used for information enhancement; Linear projection layers are used to map hidden states to a vocabulary; Step 5 specifically includes: Step 5.1: Flatten the semantic enhancement fusion features into sequence features in the spatial dimension, input them into the text generation decoder, and align them with the built-in text embedding features semantically through the linear projection layer of the text generation decoder. Step 5.2: The text generation decoder captures the contextual dependencies within the text sequence through its masked self-attention layer, and the aligned semantic enhancement fusion features are fused with the text embedding features built into the decoder through the cross-attention mechanism in the cross-attention layer to obtain cross-modal latent state features. Step 5.3: Select text sequences from cross-modal latent state features using a preset sampling strategy to generate a natural language description of the changes in the dual-temporal remote sensing images.

7. The method for change detection and semantic description of dual-temporal remote sensing images according to claim 6, characterized in that: In step 6, the joint loss function for the two tasks is: loss_total = loss_cap + λ × loss_mask; Where loss_cap is the text generation loss; loss_mask is the change detection loss; and λ is the weighting coefficient of the change detection loss.

8. The method for change detection and semantic description of dual-temporal remote sensing images according to claim 7, characterized in that: In step a.1, the 2D position encoding is formed by concatenating the position vectors in the horizontal direction w and the vertical direction h, where the position vectors in both the w and h directions have a dimension of 384, and the total encoding dimension is consistent with the semantic feature dimension.

9. The training method for the dual-temporal remote sensing image change detection and semantic description model according to claim 8, characterized in that: In step a.3, the differentiated spatial perspective rearrangement strategy includes the original spatial order, horizontally flipped spatial order, vertically flipped spatial order, and diagonally flipped spatial order.

Images