Method and system for generating multi-temporal remote sensing image description with change semantic information enhancement

Abstract

The application discloses a kind of change semantic information enhanced multi-temporal remote sensing image description generation method and system, the application change semantic information enhanced multi-temporal remote sensing image description generation method includes change detection for the multi-temporal remote sensing image to be processed to obtain its change image;Change image is used to the multi-temporal remote sensing image to be processed for enhancement;The multi-temporal remote sensing image after enhancement is fused with change image to obtain visual change feature;Visual change feature is decoded using decoder to obtain the change description of multi-temporal remote sensing image.The application aims to use remote sensing change detection technology to locate the visual change area of multi-temporal image, better capture visual change feature by image enhancement and feature fusion incentive model to improve the ability of capturing visual change information, improve the quality of multi-temporal remote sensing image change description sentence, generate more accurate text description.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, specifically to a method and system for generating multi-temporal remote sensing image descriptions with enhanced semantic information about changes. Background Technology

[0002] With the continuous development of the remote sensing field, researchers have gradually shifted their focus from traditional remote sensing image description tasks to in-depth research on remote sensing image change description techniques. Multi-temporal remote sensing image change description aims to analyze visual information in remote sensing images spanning different time periods and generate detailed natural language descriptions of visual change characteristics. Because multi-temporal remote sensing image change descriptions can intuitively reflect surface changes in a specific scene over a period of time using natural language, fully leveraging the informational advantages of remote sensing images, this technology has significant application value in fields such as disaster monitoring, urban planning, and military decision-making.

[0003] In recent years, the rapid development of deep learning has promoted the advancement of remote sensing image interpretation. Currently, advanced remote sensing image change description techniques tend to adopt an encoder-decoder architecture. Specifically, this method utilizes convolutional neural networks to extract features from multi-temporal remote sensing images, then directly concatenates these features and inputs them into a recurrent neural network to generate change descriptions. To improve model performance by better capturing significant changes in images, encoder-decoder models based on attention mechanisms have emerged. However, existing methods rely on realistic descriptions during model training. This reliance leads to the attention mechanism capturing unreliable visual information, severely impacting the quality of the generated descriptions and making it difficult to produce accurate descriptions. Since remote sensing change detection technology has unique advantages in identifying and analyzing surface changes over time, how to combine this technology with encoder-decoder models to fully utilize significant visual change information and enhance change description capabilities has become a key technical problem that urgently needs to be solved. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method and system for generating multi-temporal remote sensing image descriptions with enhanced semantic information of change, in response to the above-mentioned problems in the prior art. This invention aims to use remote sensing change detection technology to locate visual change areas in multi-temporal images, and to better capture visual change features through image enhancement and feature fusion excitation models, thereby improving the ability to capture visual change information, improving the quality of multi-temporal remote sensing image change descriptions, and generating more accurate text descriptions.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for generating multi-temporal remote sensing image descriptions with enhanced semantic information of change includes the following steps: S101, perform change detection on the multi-temporal remote sensing image to be processed to obtain its changed image; S102, Enhance the multi-temporal remote sensing image to be processed using the changed image; S103, the enhanced multi-temporal remote sensing images and the change images are fused to obtain visual change features; S104, the visual change features are decoded using a decoder to obtain a change description of the multi-temporal remote sensing image.

[0006] Optionally, the multi-temporal remote sensing images to be processed include an earlier remote sensing image B and a later remote sensing image A.

[0007] Optionally, step S101 includes: generating a binary mask containing change information from the previous remote sensing image B and the later remote sensing image A in the multi-temporal remote sensing image to be processed using a pre-designed remote sensing change detection model; then performing an element-wise dot product of the binary mask with the previous remote sensing image and the later remote sensing image to obtain the change image B of the previous remote sensing image B. c Changes in image A from later remote sensing images c .

[0008] Optionally, the function expression for enhancing the multi-temporal remote sensing image to be processed using the changed image in step S102 is as follows: , In the above formula, Enhanced previous remote sensing images or enhanced post-remote sensing images , ReLU Represents the modified linear unit activation function. Conv1 and Conv2 Represents the convolution operation. This is a change image of the previous remote sensing image B. Or the changes in the later remote sensing image A. , I This refers to either the earlier remote sensing image B or the later remote sensing image A.

[0009] Optionally, step S103 includes: S201, change image of previous remote sensing image B Changes in the later remote sensing image A Enhanced pre-remote sensing images and enhanced post-remote sensing images Feature extractors were used to extract features to obtain the change image features of the previous remote sensing image B. Changes in image features of remote sensing image A in later stages Enhanced features of previous remote sensing images and enhanced features of post-remote sensing images ; S202, change image features of previous remote sensing image B Enhanced features of previous remote sensing images Connecting along the channel dimension yields combined features from previous remote sensing images. The changes in image features of the later remote sensing image A Enhanced features of post-remote sensing images Connecting along the channel dimension yields the combined features of the later remote sensing images. ; S203 combines features from previous remote sensing images Features of post-remote sensing image combination Visual change features are obtained through visual feature fusion. .

[0010] Optionally, step S203 includes: S301, combining features from previous remote sensing images Features of post-remote sensing image combination A Dimensionality reduction is achieved through parameter-sharing convolutional layers to reduce the computational cost of subsequent operations, resulting in the dimensionality-reduced combined features of the previous remote sensing images. Features of post-remote sensing image combination ; S302, Based on the following formula, combine the features of the previous remote sensing images after dimensionality reduction. Features of post-remote sensing image combination Location coding is performed to obtain the coding features of the previous remote sensing images. Post-remote sensing image coding features : , In the above formula, Represents the coding features of early remote sensing images Or features encoded in later remote sensing images , Linear Indicates a linear layer. This represents the combined features of earlier or later remote sensing images after dimensionality reduction. P Represents learnable and Two-dimensional position encoding with consistent dimensions ; S303, calculate the coding features of the previous remote sensing images according to the following formula. Post-remote sensing image coding features The difference between the two serves as a representation of spatial semantic change information. : , S304, representing spatial semantic change information Extracting multi-temporal semantic change features using linear layer residual networks : , In the above formula, ReLU Represents the modified linear unit activation function; S305, multi-temporal semantic change features and early remote sensing image coding features Input a Siamese crossover Transformer network to extract deep fusion features from previous remote sensing images. Multi-temporal semantic change features and post-remote sensing image coding features Input a Siamese crossover Transformer network to extract deep fusion features from post-remote sensing images : , In the above formula, Features for deep fusion of early remote sensing images Or features of deep fusion of remote sensing images in later stages , FFN Represents a feedforward network. CroAtt Represents cross-attention calculation, Represents the coding features of early remote sensing images Or features encoded in later remote sensing images ; S306, Deeply fuse the extracted features from the previous remote sensing images. Deep fusion features with later remote sensing images Multi-temporal deep fusion features are obtained by connecting along the channel dimension. Furthermore, the residual network was used to extract visual change features containing important geographic change information at different times. : , In the above formula, The standardization of the representative layer is represented by ×2, which indicates that the corresponding process contains two operations, and ×3 indicates that the corresponding process contains three operations.

[0011] Optionally, the decoder in step S104 is a Transformer-based decoder, which includes a masked multi-head attention network, a multi-head attention network, and a feedforward network. The Transformer-based decoder decodes visual change features to obtain a change description of the multi-temporal remote sensing image, including: S401 uses a multi-head attention network to process visual change features. The text features Q corresponding to the generated partial descriptive text are subjected to multi-head attention query, then added to the original text features Q, and finally output the corresponding fused features after layer normalization. The generation of the text features Q corresponding to the generated partial descriptive text includes: position encoding the generated partial descriptive text to obtain descriptive text encoding features, integrating text information using a masked multi-head attention network, then adding to the original descriptive text encoding features, and finally outputting the corresponding text features Q after layer normalization. S402, Text feature Q is used as the query vector, visual change features As key and value vectors, these two features are cross-fused through multi-head attention to obtain fused features. The text features Q are then added to the fused features and then processed through layer standardization. Finally, the predicted probability of the next word is obtained through a linear layer, thus obtaining the change description text of the multi-temporal remote sensing image.

[0012] Optionally, the loss function used by the Transformer-based decoder during the training phase has the following expression: , In the above formula, For loss function, To describe the length of the statement, For word size, This represents the position of the description statement in the model prediction. t The word is k The probability of; Representing words k Is it in position? t If the word k In position t Place, then =1, otherwise It is 0.

[0013] Furthermore, the present invention also provides a multi-temporal remote sensing image description generation system with enhanced semantic change information, comprising a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the multi-temporal remote sensing image description generation method with enhanced semantic change information.

[0014] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program / instructions programmed or configured to execute the multi-temporal remote sensing image description generation method enhanced with change semantic information via a processor.

[0015] Furthermore, the present invention also provides a computer program product, including a computer program / instructions programmed or configured to execute the multi-temporal remote sensing image description generation method enhanced with the change semantic information via a processor.

[0016] Compared with existing technologies, the present invention has the following main advantages: The multi-temporal remote sensing image description generation method with enhanced semantic information of change in the present invention includes: detecting changes in the multi-temporal remote sensing image to be processed to obtain its changed image; enhancing the multi-temporal remote sensing image to be processed using the changed image; fusing features between the enhanced multi-temporal remote sensing image and the changed image to obtain visual change features; and decoding the visual change features using a decoder to obtain the change description of the multi-temporal remote sensing image. The present invention aims to use remote sensing change detection technology to locate the visual change region of multi-temporal images, and to better capture visual change features through image enhancement and feature fusion stimulation models to improve the ability to capture visual change information, improve the quality of multi-temporal remote sensing image change description sentences, and generate more accurate text descriptions. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the overall framework structure of the method in an embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of the principle of the remote sensing detector P2V-CD in an embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram of the network structure for visual feature fusion in an embodiment of the present invention.

[0021] Figure 5 This is a schematic diagram of the first set of visualization results on the LEVIR-CC dataset according to an embodiment of the present invention.

[0022] Figure 6 This is a schematic diagram of the second set of visualization results on the LEVIR-CC dataset according to an embodiment of the present invention.

[0023] Figure 7 This is a schematic diagram of the third set of visualization results on the LEVIR-CC dataset in an embodiment of the present invention.

[0024] Figure 8 This is a schematic diagram of the fourth set of visualization results in the LEVIR-CC dataset according to an embodiment of the present invention. Detailed Implementation

[0025] like Figure 1As shown, the multi-temporal remote sensing image description generation method with enhanced semantic information of change in this embodiment includes the following steps: S101, perform change detection on the multi-temporal remote sensing image to be processed to obtain its changed image; S102, Enhance the multi-temporal remote sensing image to be processed using the changed image; S103, the enhanced multi-temporal remote sensing images and the change images are fused to obtain visual change features; S104, the visual change features are decoded using a decoder to obtain a change description of the multi-temporal remote sensing image.

[0026] like Figure 2 As shown, the multi-temporal remote sensing image description generation method with enhanced semantic information of change in this embodiment mainly consists of three parts: a. change-visual enhancement, b. multi-temporal feature fusion, and c. description generation. To accurately locate the visual change regions in the multi-temporal images, the multi-temporal remote sensing image description generation method with enhanced semantic information of change in this embodiment uses an existing P2V-CD as a remote sensing detector to detect changes in the multi-temporal remote sensing images to be processed and obtain their change images.

[0027] In this embodiment, the multi-temporal remote sensing images to be processed include an earlier remote sensing image B and a later remote sensing image A. For example... Figure 3 As shown, step S101 includes: generating a binary mask containing change information from the previous remote sensing image B and the later remote sensing image A in the multi-temporal remote sensing image to be processed using a pre-designed remote sensing change detection model; then performing an element-wise dot product of the binary mask with the previous remote sensing image and the later remote sensing image to obtain the change image B of the previous remote sensing image B. c Changes in image A from later remote sensing images c The remote sensing change detection model can be any known model, as needed. As an optional implementation, this embodiment uses the existing P2V-CD detection model to acquire change images from multi-temporal remote sensing images, such as... Figure 3 As shown.

[0028] Since the change image obtained using the P2V-CD detector is itself part of a multi-temporal image, directly fusing the features of the two would result in a large amount of information redundancy. The solution to this problem in this embodiment of the invention is to combine the change image B from the previous remote sensing image B... c Changes in image A from later remote sensing images c This is achieved by using two consecutive convolutional networks, each with a size of 1×1. Figure 2 The expression is represented as (convolution 1×1). ×2The first convolutional network compresses the number of channels in the changed image to 1, achieving information compression and mitigating the impact of information redundancy. The second convolutional network restores the number of channels in the changed image to match the multi-temporal image and adds the changed image back into the multi-temporal image as an element. This operation enhances meaningfully changed regions while preserving the overall structural information of the multi-temporal image. Specifically, the function expression for enhancing the multi-temporal remote sensing image to be processed using the changed image in step S102 of this embodiment is: , In the above formula, Enhanced previous remote sensing images or enhanced post-remote sensing images , ReLU Represents the modified linear unit activation function. Conv1 and Conv2 Represents the convolution operation. This is a change image of the previous remote sensing image B. Or the changes in the later remote sensing image A. , I This refers to either the earlier remote sensing image B or the later remote sensing image A.

[0029] To obtain discriminative representations of visual information changes and generate accurate descriptions of visual changes, the model needs to consider all the change information between a pair of images simultaneously, such as... Figure 2 As shown, step S103 in this embodiment includes: S201, change image of previous remote sensing image B Changes in the later remote sensing image A Enhanced pre-remote sensing images and enhanced post-remote sensing images Feature extractors were used to extract features to obtain the change image features of the previous remote sensing image B. Changes in image features of remote sensing image A in later stages Enhanced features of previous remote sensing images and enhanced features of post-remote sensing images The feature extractor can be any known feature extractor as needed. For example, as an optional implementation, ResNet101 is used as the feature extractor in this embodiment. Introducing ResNet101 as the feature extractor, features of the enhanced multi-temporal images and changing images are obtained, and the extracted features are concatenated according to the channel dimension. The mathematical expression of this process is as follows: , , , in, This demonstrates the use of the ResNet101 feature extractor to extract visual features from multi-temporal images. ; S202, change image features of previous remote sensing image B Enhanced features of previous remote sensing images Connecting along the channel dimension yields combined features from previous remote sensing images. The changes in image features of the later remote sensing image A Enhanced features of post-remote sensing images Connecting along the channel dimension yields the combined features of the later remote sensing images. ; S203 combines features from previous remote sensing images Features of post-remote sensing image combination Visual change features are obtained through visual feature fusion. .

[0030] The visual feature fusion method in step S203 can be selected according to the actual situation, for example, as an optional implementation method, such as Figure 4 As shown, step S203 in this embodiment includes: S301, combining features from previous remote sensing images Features of post-remote sensing image combination Dimensionality reduction is achieved through parameter-sharing convolutional layers to reduce the computational cost of subsequent operations, resulting in the dimensionality-reduced combined features of the previous remote sensing images. Features of post-remote sensing image combination ; S302, Based on the following formula, combine the features of the previous remote sensing images after dimensionality reduction. Features of post-remote sensing image combination Location coding is performed to obtain the coding features of the previous remote sensing images. Post-remote sensing image coding features : , In the above formula, Represents the coding features of early remote sensing images Or features encoded in later remote sensing images , Linear Indicates a linear layer. This represents the combined features of earlier or later remote sensing images after dimensionality reduction. P Represents learnable and Two-dimensional position encoding with consistent dimensions ; S303, calculate the coding features of the previous remote sensing images according to the following formula. Post-remote sensing image coding features The difference between the two serves as a representation of spatial semantic change information. : , S304, representing spatial semantic change information Extracting multi-temporal semantic change features using linear layer residual networks : , In the above formula, ReLU Represents the modified linear unit activation function; S305, In order to achieve deep fusion of features related to change, multi-temporal semantic change features are integrated. and early remote sensing image coding features Input a Siamese crossover Transformer network to extract deep fusion features from previous remote sensing images. Multi-temporal semantic change features and post-remote sensing image coding features Input a Siamese crossover Transformer network to extract deep fusion features from post-remote sensing images : , In the above formula, Features for deep fusion of early remote sensing images Or features of deep fusion of remote sensing images in later stages FFN Represents a feedforward network. CroAtt Represents cross-attention calculation, Represents the coding features of early remote sensing images Or features encoded in later remote sensing images ; S306, Deeply fuse the extracted features from the previous remote sensing images. Deep fusion features with later remote sensing images Multi-temporal deep fusion features are obtained by connecting along the channel dimension. Furthermore, residual networks are used to extract visual change features containing important geographic change information at different times to explore high-level multi-temporal image semantic representations. , In the above formula, For visual change features, LN represents layer normalization, ×2 indicates that the corresponding process contains two operations, and ×3 indicates that the corresponding process contains three operations.

[0031] like Figure 2As shown, the decoder in step S104 of this embodiment is a Transformer-based decoder. The Transformer-based decoder includes a masked multi-head attention network, a multi-head attention network, and a feedforward network. The Transformer-based decoder obtains a description of changes in multi-temporal remote sensing images from visual change features, including: S401 uses a multi-head attention network to process visual change features. The text features Q corresponding to the generated partial descriptive text are subjected to multi-head attention query, then added to the original text features Q, and finally output the corresponding fused features after layer normalization. The generation of the text features Q corresponding to the generated partial descriptive text includes: position encoding the generated partial descriptive text to obtain descriptive text encoding features, integrating text information using a masked multi-head attention network, then adding to the original descriptive text encoding features, and finally outputting the corresponding text features Q after layer normalization. S402, Text feature Q is used as the query vector, visual change features As key and value vectors, these two features are cross-fused through multi-head attention to obtain fused features. The text features Q are then added to the fused features, followed by layer normalization. Finally, a linear layer is used to obtain the predicted probability of the next word, thus yielding the change description text for multi-temporal remote sensing images. In this embodiment, the Transformer-based decoder trains the model during the training phase using the difference between the generated change description and the true description. The loss function used is cross-entropy loss. The expression for the loss function used by the Transformer-based decoder during the training phase is: , In the above formula, For loss function, To describe the length of the statement, For word size, This represents the position of the description statement in the model prediction. t The word is k The probability of; Representing words k Is it in position? t If the word k In position t Place, then =1, otherwise It is 0.

[0032] To verify the performance of the method in this embodiment, this embodiment uses the largest available dataset in the field of remote sensing change description, LEVIR-CC, which contains 10,077 multi-temporal remote sensing image pairs. In this embodiment, the number of image pairs in the training set, validation set, and test set are 6,815, 1,333, and 1,929, respectively, of which the test set contains 964 pairs of unchanged images.

[0033] We selected the highly competitive RSICCformer and Chg2Cap models for comparison, conducting three sets of experiments. The RSICCformer model employs a Transformer-based dual-branch encoder to capture discriminative features from multi-temporal remote sensing images. Subsequently, based on the generated text information and image features, a Transformer-based decoder predicts the next word of the descriptive sentence. The Chg2Cap model uses an encoder-decoder structure similar to the RSICCformer model, introducing stable multi-level self-attention modules to capture change-related features, thus making it more likely to generate more accurate descriptions.

[0034] In Experiments 1 and 2, this embodiment uses BLEU-n, METEOR, ROUGE-L, and CIDEr-D as evaluation metrics to quantitatively compare the scores of the three models on different datasets, while Experiment 3 compares the parameter scales of the three models. The results of the three sets of experiments are shown in Tables 1 to 3.

[0035] Table 1 Results of Experiment 1

[0036] In Experiment 1, the experimental results for the two methods other than the one in this embodiment were obtained from the data in the original paper. The highest-scoring results in Experiment 1 are shown in bold in Table 1. By quantitatively comparing the model in this embodiment with other methods on the LEVIR-CC dataset, Table 1 shows that the method in this embodiment has the best performance. For example, compared with the previous best model Chg2Cap, the method in this embodiment improves the scores of BLEU-1, METEOR, and CIDEr-D by 1.12%, 1.62%, and 0.85%, respectively.

[0037] Table 2 Results of Experiment 2

[0038] In Table 2, "No-change" represents the 964 pairs of unchanged images in the LEVIR-CC test dataset, and "Change" represents the 965 pairs of changed images in the LEVIR-CC dataset. The highest scores for each metric in Experiment 2 are shown in bold in the table. Experimental data show that the model of this embodiment is superior to the other two methods in both test images.

[0039] Table 3 Results of Experiment 3

[0040] The data in Table 3 shows that, compared to other models, the method in this embodiment also has advantages in terms of parameter scale.

[0041] Figures 5-8 The visualization results of the method in this embodiment on the LEVIR-CC dataset are shown, where (a) is the previous remote sensing image B, (b) is the later remote sensing image A, (c) is the masked image, and (d) is the changed image B of the previous remote sensing image B. c (e) shows the changes in the later remote sensing image A. c . Figure 5 and Figure 6 This embodiment provides accurate descriptions of both changed and unchanged image pairs. Figure 7 and Figure 8 This demonstrates the ability of this embodiment to capture objects with small area changes and objects with significant changes. Overall, although the description sentences generated by the model in this embodiment do not achieve absolute accuracy at the word level, the generated sentences can achieve semantic alignment with the reference sentences as a whole.

[0042] In summary, the multi-temporal remote sensing image description generation method with enhanced semantic information of change in this embodiment includes change-visual enhancement, multi-temporal feature fusion, and Transformer-based change description generation. Change detection refers to using an existing high-quality remote sensing change detection module P2V-CD as a detector, inputting multi-temporal remote sensing images, extracting object-level visual change information, and outputting change images. The change-visual enhancement module enhances the multi-temporal images using change images to enhance visual change features while preserving the latter's structural information. The multi-temporal feature fusion module extracts features from the enhanced multi-temporal images and change images, connects the features along the channel dimension, and then performs feature fusion to obtain discriminative visual change features. The Transformer-based description generator uses the fused visual change features and generated text information to progressively predict the next word of the change description sentence. This multi-temporal remote sensing image description generation method with enhanced semantic information of change in this embodiment can use remote sensing change detection technology to locate visual change regions in multi-temporal images. Through image enhancement and feature fusion, it incentivizes the model to better capture visual change features, thereby generating more accurate text descriptions.

[0043] Furthermore, this embodiment also provides a multi-temporal remote sensing image description generation system with enhanced semantic change information, including a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the multi-temporal remote sensing image description generation method with enhanced semantic change information.

[0044] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program / instructions that are programmed or configured to execute the multi-temporal remote sensing image description generation method enhanced with change semantic information via a processor.

[0045] Furthermore, this embodiment also provides a computer program product, including a computer program / instruction, which is programmed or configured to execute the multi-temporal remote sensing image description generation method enhanced with the change semantic information via a processor.

[0046] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, and computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0047] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating multi-temporal remote sensing image descriptions with enhanced semantic information of change, characterized in that, Includes the following steps: S101, perform change detection on the multi-temporal remote sensing image to be processed to obtain its changed image; S102, Enhance the multi-temporal remote sensing image to be processed using the changed image: ， In the above formula, Enhanced previous remote sensing images or enhanced post-remote sensing images , ReLU Represents the modified linear unit activation function. Conv1 and Conv2 Represents the convolution operation. This is a change image of the previous remote sensing image B. Or the changes in the later remote sensing image A. , I This refers to either the earlier remote sensing image B or the later remote sensing image A; S103, perform feature fusion of the enhanced multi-temporal remote sensing image and the change image to obtain visual change features, including: S201, fusing the change image of the previous remote sensing image B. Changes in the later remote sensing image A Enhanced pre-remote sensing images and enhanced post-remote sensing images Feature extractors were used to extract features to obtain the change image features of the previous remote sensing image B. Changes in image features of remote sensing image A in later stages Enhanced features of previous remote sensing images and enhanced features of post-remote sensing images S202, change image features of the previous remote sensing image B. Enhanced features of previous remote sensing images Connecting along the channel dimension yields combined features from previous remote sensing images. The changes in image features of the later remote sensing image A Enhanced features of post-remote sensing images Connecting along the channel dimension yields the combined features of the later remote sensing images. S203, combining features from previous remote sensing images Features of post-remote sensing image combination Visual change features are obtained through visual feature fusion. ; S104, the visual change features are decoded using a decoder to obtain a change description of the multi-temporal remote sensing image; the decoder is a Transformer-based decoder, which includes a masked multi-head attention network, a multi-head attention network, and a feedforward network. The step of decoding the visual change features using the decoder to obtain a change description of the multi-temporal remote sensing image includes: S401, using a multi-head attention network to decode the visual change features... The text features Q corresponding to the generated partial descriptive text are subjected to multi-head attention query, then added to the original text features Q, and finally processed through layer normalization to output the corresponding fused features; the generation of the text features Q corresponding to the generated partial descriptive text includes: position encoding the generated partial descriptive text to obtain descriptive text encoding features, integrating text information using a masked multi-head attention network, then adding to the original descriptive text encoding features, and finally processing through layer normalization to output the corresponding text features Q; S402, using the text features Q as a query vector, visual change features As key and value vectors, these two features are cross-fused through multi-head attention to obtain fused features. The text features Q are then added to the fused features and then processed through layer standardization. Finally, the predicted probability of the next word is obtained through a linear layer, thus obtaining the change description text of the multi-temporal remote sensing image.

2. The method for generating multi-temporal remote sensing image descriptions with enhanced semantic information of change according to claim 1, characterized in that, Step S101 includes: generating a binary mask containing change information from the previous remote sensing image B and the later remote sensing image A in the multi-temporal remote sensing image to be processed using a pre-designed remote sensing change detection model; then performing an element-wise dot product of the binary mask with the previous and later remote sensing images to obtain the change image B of the previous remote sensing image B. c Changes in image A from later remote sensing images c .

3. The method for generating multi-temporal remote sensing image descriptions with enhanced semantic information of change according to claim 1, characterized in that, Step S203 includes: S301, combining features from previous remote sensing images Features of post-remote sensing image combination Dimensionality reduction is achieved through parameter-sharing convolutional layers to reduce the computational cost of subsequent operations, resulting in the dimensionality-reduced combined features of the previous remote sensing images. Features of post-remote sensing image combination ; S302, Based on the following formula, combine the features of the previous remote sensing images after dimensionality reduction. Features of post-remote sensing image combination Location coding is performed to obtain the coding features of the previous remote sensing images. Post-remote sensing image coding features : ， In the above formula, Represents the coding features of early remote sensing images Or features encoded in later remote sensing images , Linear Indicates a linear layer. This represents the combined features of earlier or later remote sensing images after dimensionality reduction. P Represents learnable and Two-dimensional position encoding with consistent dimensions ; S303, calculate the coding features of the previous remote sensing images according to the following formula. Post-remote sensing image coding features The difference between the two serves as a representation of spatial semantic change information. : ， S304, representing spatial semantic change information Extracting multi-temporal semantic change features using linear layer residual networks : ， In the above formula, ReLU Represents the modified linear unit activation function; S305, multi-temporal semantic change features and early remote sensing image coding features Input a Siamese crossover Transformer network to extract deep fusion features from previous remote sensing images. Multi-temporal semantic change features and post-remote sensing image coding features Input a Siamese crossover Transformer network to extract deep fusion features from post-remote sensing images : ， In the above formula, Features for deep fusion of early remote sensing images Or features from deep fusion of remote sensing images in later stages. , FFN Represents a feedforward network. CroAtt Represents cross-attention calculation, Represents the coding features of early remote sensing images Or features encoded in later remote sensing images ; S306, Deeply fuse the extracted features from the previous remote sensing images. Deep fusion features with later remote sensing images Multi-temporal deep fusion features are obtained by connecting along the channel dimension. Furthermore, the residual network was used to extract visual change features containing important geographic change information at different times. : ， In the above formula, The standardization of the representative layer is represented by ×2, which indicates that the corresponding process contains two operations, and ×3 indicates that the corresponding process contains three operations.

4. The method for generating multi-temporal remote sensing image descriptions with enhanced semantic information of change according to claim 1, characterized in that, The expression for the loss function used by the Transformer-based decoder during the training phase is as follows: ， In the above formula, For loss function, To describe the length of the statement, For word size, This represents the position of the description statement in the model prediction. t The word is k The probability of; Representing words k Is it in position? t If the word k In position t Place, then =1, otherwise It is 0.

5. A multi-temporal remote sensing image description generation system with enhanced semantic information of change, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the multi-temporal remote sensing image description generation method with enhanced semantic information of change as described in any one of claims 1 to 4.

6. A computer-readable storage medium storing a computer program / instructions, characterized in that, The computer program / instructions are programmed or configured to execute, via a processor, the multi-temporal remote sensing image description generation method with enhanced semantic information of change as described in any one of claims 1 to 4.

7. A computer program product comprising a computer program / instructions, characterized in that, The computer program / instructions are programmed or configured to execute, via a processor, the multi-temporal remote sensing image description generation method with enhanced semantic information of change as described in any one of claims 1 to 4.

Images