Multi-temporal remote sensing image change general semantic description generation method

Abstract

The invention relates to a multi-temporal remote sensing image change generality semantic description generation method, and belongs to the technical field of surveying and mapping. The method comprises the following steps: 1, splicing a remote sensing image set in a time sequence manner, and extracting key frames in the remote sensing image set at equal time intervals; 2, carrying out single-temporal image spatial feature extraction on the key frame in an image segmentation mode by utilizing a Transform model; 3, performing multi-temporal image time feature fusion on the spatial features of the single-temporal image by using a Transform model; 4, performing feature decoding on the time features of the multi-temporal image by using a multi-layer Transform decoder so as to obtain a general descriptive word of the change of the multi-temporal remote sensing image; and 5, in the training stage, using the Mask matrix to suppress the attention of the to-be-generated multi-temporal remote sensing image change general word queue. Compared with the prior art, the method has the advantages that the autoregressive generation capability of the multi-layer Transform decoder is ensured, and then the multi-temporal remote sensing image change generality semantic analysis efficiency in the surveying and mapping field is improved.

Description

Technical Field

[0001] The invention relates to a method for generating a generalized semantic description of multi-temporal remote sensing image changes, and belongs to the technical field of surveying and mapping. Background Art

[0002] Semantic description of changes in remote sensing imagery is a key research area in computer vision and natural language processing. In practical applications, algorithms analyzing remote sensing imagery at different temporal phases can automatically extract the characteristics of surface changes at different times and describe these changes in land cover using natural language. This process is of great significance in fields such as environmental monitoring, resource management, and disaster assessment, helping to identify and analyze significant changes in the Earth's surface over time.

[0003] While current technologies can generate semantic descriptions of changes in multi-temporal remote sensing imagery to a certain extent, these descriptions focus on changes in two or more temporal images, lacking a comprehensive overview of multi-temporal change processes. A comprehensive semantic description of multi-temporal remote sensing imagery can not only comprehensively summarize surface change trends and reduce information redundancy, but also improve the efficiency of extracting change information and enhance the timeliness of information transmission, providing a reference for rapid decision-making and response in practical applications.

[0004] Due to the lack of an end-to-end generalized semantic description algorithm for multi-temporal remote sensing imagery, practical applications still require manual integration and summarization of the results of multi-temporal change semantic descriptions to form a generalized description. Therefore, how to automatically generate generalized semantic descriptions from multi-temporal remote sensing image change detection results to more efficiently serve practical engineering applications is an urgent problem in this field. Summary of the Invention

[0005] This paper aims to improve the efficiency of semantically summarizing changes in multi-temporal remote sensing images in surveying and mapping. It proposes a method for generating a semantically summarizing description of changes in multi-temporal remote sensing images. This method uses an end-to-end approach to automatically extract the change characteristics of multi-temporal remote sensing images and generate readable textual descriptions.

[0006] The workflow of the present invention is as follows: first, the image encoding model is initialized using the ViT-B / 16 parameters pre-trained by CLIP, and the text decoding model is initialized using the decoder parameters of the GIT model fine-tuned on the video-text description MSR-VTT dataset. Since CLIP uses image-text alignment learning and the decoder of GIT uses the Decoder-Only continuation mode, its hidden states are aligned to a certain extent with the space where the word embedding is located. Secondly, based on the initialization, the model is continued to be used in an end-to-end manner. Based on the massive video key frame-text description data, the pre-trained image encoding model and text decoding model are frozen, and the time series fusion model is trained to better align its hidden state with the space where the word embedding is located. Finally, the model is fine-tuned end-to-end using multi-temporal remote sensing image annotation data to adapt to the knowledge in the surveying and mapping field.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] The present invention provides a method for generating a generalized semantic description of multi-temporal remote sensing image changes, which is applied to a remote sensing mapping semantic analysis scenario and includes the following steps:

[0009] Step 1: Splice the multi-temporal remote sensing images acquired by the visible light sensor into a remote sensing image set in a time series manner, and extract the key frames in the remote sensing image set at equal time intervals;

[0010] Step 1.1: Construct a remote sensing image set, specifically by splicing multi-temporal remote sensing images in a time series manner;

[0011] Step 1.2: Extract key frames from the remote sensing image set at equal time intervals;

[0012] Step 2: Use the Transformer model to extract single-phase image spatial features from key frames in an image segmentation manner;

[0013] Step 2.1: Obtain the patch space sequence for image segmentation as shown in formula (1);

[0014]

[0015] Among them, stack represents the stacking operation of vectors, X t Represents the patch space sequence corresponding to the remote sensing image at time t, Represents the result of flattening the pixels of each patch into 1 dimension, N = HW / P 2 .

[0016] Step 2.2: Map the patch space sequence to the dimension consistent with the hidden layer of the Transformer model to obtain the embedded representation shown in formula (2);

[0017]

[0018] in, is a trainable parameter, D B represents the hidden layer dimension, are the embedding of each patch and the embedding of the stacked patches respectively.

[0019] Step 2.3: Use the method shown in formula (3) to encode the position of the patch space sequence of the embedded representation to obtain the hidden state sequence;

[0020]

[0021] Among them, x cls For additional learnable hidden states, Encode the position.

[0022] Step 2.4: Decompose the hidden state sequence into multiple heads in the hidden layer dimension using the method shown in formula (4) to obtain multiple sub-hidden state sequences, each of which is called a head.

[0023]

[0024] Among them, concat means splicing on the hidden state dimension, H B represents the total number of attention heads, d B =D B / H B Represents the hidden layer dimension of each head. represents the hth head obtained by step 2.4 in the lth layer network, l = 1, 2, ..., L, h = 1, 2, ..., H B , L represents the number of layers in the network.

[0025] Step 2.5: Calculate the query matrix, key matrix, and value matrix of each header using the method shown in formula (5):

[0026]

[0027] in, is the trainable parameter matrix.

[0028] Step 2.6: Use the self-attention mechanism on the hidden state sequence to obtain the single-head spatial self-attention weighted value as shown in formula (6);

[0029]

[0030] in Represents the self-attention weighted value of the h-th head of the l-th layer network.

[0031] Step 2.7: Use the method shown in formula (7) to perform multi-head fusion on the single-head spatial self-attention weighted values; further, splice the single-head spatial self-attention weighted values on the hidden layer dimension to restore the hidden layer dimension of the weighted value to the full hidden layer dimension, and perform a linear transformation without changing the dimension;

[0032]

[0033] in, is a trainable parameter;

[0034] Step 2.8: Use the method shown in formula (8) to perform residual connection and normalization on the input and the self-attention weighted value after multi-head spatial fusion, and then pass it through two layers of feedforward neural network to obtain the output of this layer network.

[0035]

[0036] Among them, LN represents the layer normalization operation.

[0037] Step 2.9: Treat steps 2.4 to 2.8 as a single layer of network calculation, and use the output of the previous layer as the input of the next layer. After multi-layer network calculation, the spatial features of the single-phase image are obtained;

[0038] Step 3: Use the Transformer model to fuse the spatial features of single-phase images with the temporal features of multi-phase images;

[0039] Step 3.1: Use the method shown in formula (8) to encode the spatial features of the single-phase image into time series positions, and then obtain the hidden state sequence;

[0040]

[0041] Among them D C represents the hidden layer dimensions, E1, E2, ..., E T represents the single-phase image spatial features of all T phases obtained in step 2, Encode the position.

[0042] Step 3.2: Decompose the hidden state sequence into multiple heads in the hidden layer dimension using the method shown in formula (9) to obtain multiple sub-hidden state sequences, each of which is called a head.

[0043]

[0044] Among them, concat means splicing on the hidden state dimension, H C represents the total number of attention heads, d C =D C / H C represents the hidden layer dimension of each head. e l(h) represents the hth head obtained by decomposing the lth layer network in step 3.2, l = 1, 2, ..., L C ,h=1,2,...,H C , L C Indicates the number of layers in the network.

[0045] Step 3.3: Calculate the query matrix, key matrix, and value matrix of each header using the method shown in formula (10):

[0046]

[0047] in, is the trainable parameter matrix.

[0048] Step 3.4: Use the self-attention mechanism on the sub-hidden state sequence to obtain the single-head temporal self-attention weighted value as shown in formula (11);

[0049]

[0050] in Represents the self-attention weighted value of the h-th head of the l-th layer network.

[0051] Step 3.5: Use the method shown in formula (12) to perform multi-head fusion on the single-head temporal self-attention weighted values; further, splice the single-head temporal self-attention weighted values on the hidden layer dimension to restore the hidden layer dimension of the weighted value to the full hidden layer dimension, and perform a linear transformation without changing the dimension;

[0052]

[0053] in, is a trainable parameter;

[0054] Step 3.6: Use the method shown in formula (13) to perform residual connection and normalization on the input and the time self-attention weighted value after multi-head fusion, and then pass through two layers of feedforward neural network to obtain the output of this layer network.

[0055] e l′ =LN(e l +N l ) (13)

[0056] Among them, LN represents the layer normalization operation.

[0057] Step 3.7: Consider steps 3.2 to 3.6 as a layer of network calculation, and use the output of the previous layer as the input of the next layer. After multi-layer network calculation, the fused multi-temporal image temporal features are obtained;

[0058] Step 4: Use a multi-layer Transformer decoder to decode the temporal features of the multi-temporal image, and then obtain a summary descriptive word of the multi-temporal remote sensing image changes;

[0059] Step 4.1: Use the method shown in formula (14) to perform position encoding of the multi-temporal image temporal features in text order to obtain the text feature hidden state sequence;

[0060]

[0061] Among them, S1, S2, ..., S T The fused multi-temporal image temporal features generated in step 3. T+1 ,...,S T+m is the word embedding corresponding to the generated word, m represents its length, and it is initially empty. Encode the position.

[0062] Step 4.2: Use the method shown in formula (15) to perform multi-head decomposition on the hidden layer dimension of the text feature hidden state sequence to obtain multiple sub-text hidden state sequences, each of which is called a head;

[0063]

[0064] Among them, concat means splicing on the hidden state dimension, H D represents the total number of attention heads, d D =D D / H D Indicates the hidden layer dimension of each head. s l(h) represents the hth head obtained by decomposing the lth layer network, l=1,2,...,L D ,h=1,2,...,H D , L D Indicates the number of layers in the network.

[0065] Step 4.3: Calculate the Query matrix, Key matrix, and Value matrix of each header using the method shown in formula (16):

[0066]

[0067] in, is the trainable parameter matrix.

[0068] Step 4.4: Use the self-attention mechanism on the sub-text hidden state sequence to obtain the single-head text self-attention weighted value as shown in formula (17);

[0069]

[0070] in Represents the single-head text self-attention weighted value of the h-th head of the l-th layer network.

[0071] Step 4.5: Use the method shown in formula (18) to perform multi-head fusion on the single-head text self-attention weighted values; further, splice the single-head text self-attention weighted values on the hidden layer dimension to restore the hidden layer dimension of the weighted value to the full hidden layer dimension size, and perform a linear transformation without changing the dimension;

[0072]

[0073] in, is a trainable parameter;

[0074] Step 4.6: Use the method shown in formula (19) to perform residual connection and normalization on the current layer input and the text self-attention weighted value after multi-head fusion, and then obtain the output of this layer network after passing through two layers of feedforward neural network.

[0075] s l′ =LN(s l +F l ) (19)

[0076] Among them, LN represents the layer normalization operation.

[0077] Step 4.7: Treat steps 4.2 to 4.6 as a single layer of network calculation, and use the output of the previous layer as the input of the next layer. After multi-layer network calculation, the probability distribution of the next word is obtained as shown in formula (20);

[0078] P(w t+m+1 |w1,w2,...,w t+m )=softmax(W out h t+m +b out ) (20)

[0079] in is a trainable parameter, d voc is the vocabulary size, w i represents the i-th word of the generated sentence, h t+mRepresents the hidden state at position t+m in the hidden state sequence. Based on the probability distribution of the next word, a selection strategy such as top-k or beam-search is used to select the next word, and the word embedding of the selected word is placed at the end of the input queue s.

[0080] Step 4.8: Iterate steps 4.1 to 4.7 in a loop until a sentence end symbol is generated or the maximum sentence length is reached, and then stop generating the generalized semantic description words for the multi-temporal remote sensing image changes;

[0081] Step 5: During the training phase, the Mask matrix in Equation (21) is used to suppress attention to the generated word queues summarizing the changes in multi-temporal remote sensing images, thereby ensuring the autoregressive generation capability of the multi-layer Transformer decoder and improving the efficiency of semantic parsing of the summarization of multi-temporal remote sensing image changes in the field of surveying and mapping.

[0082]

[0083] Where T represents the length of the sequence being processed, It is a triangular matrix: the value above the diagonal is negative infinity, and the value below the diagonal is 0;

[0084] Beneficial effects:

[0085] Compared with the prior art, the present invention has the following effects:

[0086] 1. Ability to automatically generate generalized semantic descriptions of changes in multi-temporal remote sensing images in an end-to-end manner;

[0087] 2. Ability to analyze the temporal changes of landform features, improve the efficiency of interpreting multi-temporal remote sensing images, and assist in the judgment of future changes;

[0088] 3. The invention results can provide technical support for urban planning, regional monitoring, insurance claims and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 A schematic diagram of the flow of the generation method of this embodiment;

[0090] Figure 2 Schematic diagram of the generation result of this embodiment. DETAILED DESCRIPTION

[0091] In order to better illustrate the purpose and advantages of the present invention, the following is a further description of the invention in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the scope of protection of the present invention.

[0092] Example

[0093] like Figure 1 As shown, the method for generating a generalized semantic description of multi-temporal remote sensing image changes in this embodiment is specifically implemented in the following steps:

[0094] Step 1: Splice the multi-temporal remote sensing images acquired by the visible light sensor into a remote sensing image set in a time series manner, and extract the key frames in the remote sensing image set at equal time intervals;

[0095] Step 1.1: Construct a remote sensing image set, specifically by splicing multi-temporal remote sensing images in a time series manner;

[0096] In the embodiment, the multi-temporal remote sensing image I t (t=1,2,…,n) are concatenated in the form of frames, and a fixed number of key frames A are extracted uniformly in time t The model input is a sequence of (t=1, 2, …, m) (m represents the total number of keyframes). When the number of input remote sensing images, n, is less than m, the keyframes will contain duplicate images. Each set of multi-temporal remote sensing images is concatenated into a single video file. The total video length is set to 15 seconds, with a frame rate of 30 fps and a total of 450 frames. The remote sensing images are replicated 450 / n times in chronological order to fill the video, with each replicated image occupying 450 / n consecutive frames.

[0097] Step 1.2: Extract key frames from the remote sensing image set at equal time intervals;

[0098] In the embodiment, k=6 is set as the total number of extractions, that is, one frame is extracted every 450 / k=75 frames.

[0099] Step 2: Use the Transformer model to extract single-phase image spatial features from key frames in an image segmentation manner;

[0100] Step 2.1: Obtain the patch space sequence for image segmentation as shown in formula (1);

[0101]

[0102] Among them, stack represents the stacking operation of vectors, X t Represents the patch space sequence corresponding to the remote sensing image at time t, Represents the result of flattening the pixels of each patch into 1 dimension, N = HW / P 2 .

[0103] In the embodiment, H=224, W=224, C=3, P=16, N=HW / P 2 =196.

[0104] Step 2.2: Map the patch space sequence to the dimension consistent with the hidden layer of the Transformer model to obtain the embedded representation shown in formula (2);

[0105]

[0106] in, is a trainable parameter, D B represents the hidden layer dimension, are the embedding of each patch and the embedding of the stacked patches respectively.

[0107] In the embodiment, D B =768.

[0108] Step 2.3: Use the method shown in formula (3) to encode the position of the patch space sequence of the embedded representation to obtain the hidden state sequence;

[0109]

[0110] Among them, x cls For additional learnable hidden states, Encode the position.

[0111] In the embodiment, Pos B Use fixed sine and cosine coding.

[0112] Step 2.4: Decompose the hidden state sequence into multiple heads in the hidden layer dimension using the method shown in formula (4) to obtain multiple sub-hidden state sequences, each of which is called a head.

[0113]

[0114] Among them, concat means splicing on the hidden state dimension, H B represents the total number of attention heads, d B =D B / H B Represents the hidden layer dimension of each head. represents the hth head obtained by step 2.4 in the lth layer network, l = 1, 2, ..., L, h = 1, 2, ..., H B , L represents the number of layers in the network.

[0115] In the embodiment, L=12, H B =12.

[0116] Step 2.5: Calculate the query matrix, key matrix, and value matrix of each header using the method shown in formula (5):

[0117]

[0118] in, is the trainable parameter matrix.

[0119] Step 2.6: Use the self-attention mechanism on the hidden state sequence to obtain the single-head spatial self-attention weighted value as shown in formula (6);

[0120]

[0121] in Represents the self-attention weighted value of the h-th head of the l-th layer network.

[0122] Step 2.7: Use the method shown in formula (7) to perform multi-head fusion on the single-head spatial self-attention weighted values; further, splice the single-head spatial self-attention weighted values on the hidden layer dimension to restore the hidden layer dimension of the weighted value to the full hidden layer dimension, and perform a linear transformation without changing the dimension;

[0123]

[0124] in, is a trainable parameter;

[0125] Step 2.8: Use the method shown in formula (8) to perform residual connection and normalization on the input and the self-attention weighted value after multi-head spatial fusion, and then pass it through two layers of feedforward neural network to obtain the output of this layer network.

[0126]

[0127] Among them, LN represents the layer normalization operation.

[0128] In the embodiment, the output of the current layer network is calculated in the following way

[0129]

[0130] in, is a trainable parameter, D ′ B =3072.

[0131] Step 2.9: Treat steps 2.4 to 2.8 as a single layer of network calculation, and use the output of the previous layer as the input of the next layer. After multi-layer network calculation, the spatial features of the single-phase image are obtained;

[0132] In the embodiment, the output obtained after 12-layer network calculation is Take out its first line: o t That is, the spatial features of the image at time phase t. In addition, all trainable parameters in steps 2.2 to 2.9 are initialized using the corresponding parameters of the ViT-B / 16 model pre-trained using the CLIP (Contrastive Language-Image Pretraining) framework.

[0133] Step 3: Use the Transformer model to fuse the spatial features of single-phase images with the temporal features of multi-phase images;

[0134] Step 3.1: Use the method shown in formula (8) to encode the spatial features of the single-phase image into time series positions, and then obtain the hidden state sequence;

[0135]

[0136] Among them D C represents the hidden layer dimensions, E1, E2, ..., E T represents the single-phase image spatial features of all T phases obtained in step 2, Encode the position.

[0137] In the embodiment, to avoid additional dimension conversion operations, D C Adopt and D B Equal dimensions, i.e. D C =D B =768; It is a fixed sine and cosine position code; T=k=6.

[0138] Step 3.2: Decompose the hidden state sequence into multiple heads in the hidden layer dimension using the method shown in formula (9) to obtain multiple sub-hidden state sequences, each of which is called a head.

[0139]

[0140] Among them, concat means splicing on the hidden state dimension, H C represents the total number of attention heads, d C =D C / H C represents the hidden layer dimension of each head. e l(h) represents the hth head obtained by decomposing the lth layer network in step 3.2, l = 1, 2, ..., L C ,h=1,2,...,H C , L C Indicates the number of layers in the network.

[0141] In the embodiment, H C =12,dC =768 / 12=64, L C =6.

[0142] Step 3.3: Calculate the query matrix, key matrix, and value matrix of each header using the method shown in formula (10):

[0143]

[0144] in, is the trainable parameter matrix.

[0145] Step 3.4: Use the self-attention mechanism on the sub-hidden state sequence to obtain the single-head temporal self-attention weighted value as shown in formula (11);

[0146]

[0147] in Represents the self-attention weighted value of the h-th head of the l-th layer network.

[0148] Step 3.5: Use the method shown in formula (12) to perform multi-head fusion on the single-head temporal self-attention weighted values; further, splice the single-head temporal self-attention weighted values on the hidden layer dimension to restore the hidden layer dimension of the weighted value to the full hidden layer dimension, and perform a linear transformation without changing the dimension;

[0149]

[0150] in, is a trainable parameter;

[0151] Step 3.6: Use the method shown in formula (13) to perform residual connection and normalization on the input and the time self-attention weighted value after multi-head fusion, and then pass through two layers of feedforward neural network to obtain the output of this layer network.

[0152] e l′ =LN(e l +N l ) (13)

[0153] Among them, LN represents the layer normalization operation.

[0154] In the embodiment, the output e of the current layer network is calculated as follows: l+1 .

[0155]

[0156] in, is a trainable parameter, D ′ C=3072.

[0157] Step 3.7: Consider steps 3.2 to 3.6 as a layer of network calculation, and use the output of the previous layer as the input of the next layer. After multi-layer network calculation, the fused multi-temporal image temporal features are obtained;

[0158] In the embodiment, after 6 layers of network calculation, the output of the last layer of network is obtained is the fused multi-temporal image temporal feature. In addition, the parameters in steps 3.3 to 3.7 are initialized using the model parameters pre-trained on the video semantic description dataset MSR-VTT.

[0159] Step 4: Use a multi-layer Transformer decoder to decode the temporal features of the multi-temporal image, and then obtain a summary descriptive word of the multi-temporal remote sensing image changes;

[0160] Step 4.1: Use the method shown in formula (14) to perform position encoding of the multi-temporal image temporal features in text order to obtain the text feature hidden state sequence;

[0161]

[0162] Among them, S1, S2, ..., S T The fused multi-temporal image temporal features generated in step 3. T+1 ,...,S T+m is the word embedding corresponding to the generated word, m represents its length, and it is initially empty. Encode the position.

[0163] In the embodiment, D D =768, It is a fixed sine and cosine position encoding.

[0164] Step 4.2: Use the method shown in formula (15) to perform multi-head decomposition on the hidden layer dimension of the text feature hidden state sequence to obtain multiple sub-text hidden state sequences, each of which is called a head;

[0165]

[0166] Among them, concat means splicing on the hidden state dimension, H D represents the total number of attention heads, d D =D D / H D Indicates the hidden layer dimension of each head. s l(h) represents the hth head obtained by decomposing the lth layer network, l=1,2,...,L D ,h=1,2,...,HD , L D Indicates the number of layers in the network.

[0167] In the embodiment, L D =12,H D =12,d D =D D / H D =768 / 12=64.

[0168] Step 4.3: Calculate the Query matrix, Key matrix, and Value matrix of each header using the method shown in formula (16):

[0169]

[0170] in, is the trainable parameter matrix.

[0171] Step 4.4: Use the self-attention mechanism on the sub-text hidden state sequence to obtain the single-head text self-attention weighted value as shown in formula (17);

[0172]

[0173] in Represents the single-head text self-attention weighted value of the h-th head of the l-th layer network.

[0174] Step 4.5: Use the method shown in formula (18) to perform multi-head fusion on the single-head text self-attention weighted values; further, splice the single-head text self-attention weighted values on the hidden layer dimension to restore the hidden layer dimension of the weighted value to the full hidden layer dimension size, and perform a linear transformation without changing the dimension;

[0175]

[0176] in, is a trainable parameter;

[0177] Step 4.6: Use the method shown in formula (19) to perform residual connection and normalization on the current layer input and the text self-attention weighted value after multi-head fusion, and then obtain the output of this layer network after passing through two layers of feedforward neural network.

[0178] s l′ =LN(s l +F l ) (19)

[0179] Among them, LN represents the layer normalization operation.

[0180] In the embodiment, the output s of the current layer network is calculated as shown in the following formula: l+1 .

[0181]

[0182] in, is a trainable parameter, D ′ D =3072.

[0183] Step 4.7: Treat steps 4.2 to 4.6 as a single layer of network calculation, and use the output of the previous layer as the input of the next layer. After multi-layer network calculation, the probability distribution of the next word is obtained as shown in formula (20);

[0184] P(w t+m+1 |w1,w2,...,w t+m )=softmax(W out h t+m +b out ) (20)

[0185] in is a trainable parameter, d voc is the vocabulary size, w i represents the i-th word of the generated sentence, h t+m Represents the hidden state at position t+m in the hidden state sequence. Based on the probability distribution of the next word, a selection strategy such as top-k or beam-search is used to select the next word, and the word embedding of the selected word is placed at the end of the input queue s.

[0186] In the embodiment, the word with the highest probability is selected as the output, that is, the beam width of the beam-search is set to 1.

[0187] Step 4.8: Iterate steps 4.1 to 4.7 in a loop until a sentence end symbol is generated or the maximum sentence length is reached, and then stop generating the generalized semantic description words for the multi-temporal remote sensing image changes;

[0188] In the embodiment, the maximum length of a sentence is set to 100 words. In addition, the trainable parameters in steps 4.3 to 4.7 are initialized using the parameters corresponding to the decoder of the GIT model fine-tuned on the video semantic description dataset MSR-VTT.

[0189] Step 5: During the training phase, the Mask matrix in Equation (21) is used to suppress attention to the generated word queues summarizing the changes in multi-temporal remote sensing images, thereby ensuring the autoregressive generation capability of the multi-layer Transformer decoder and improving the efficiency of semantic parsing of the summarization of multi-temporal remote sensing image changes in the field of surveying and mapping.

[0190]

[0191] Where T represents the length of the sequence being processed, It is a triangular matrix: the value above the diagonal is negative infinity, and the value below the diagonal is 0;

[0192] In this embodiment, the pre-trained image encoding model and text decoding model are frozen. A time series fusion model is trained based on the video keyframe-text description datasets MSRVTT and MSVD to better align its hidden states with the word embedding space. Finally, the entire model is fine-tuned end-to-end using multi-temporal remote sensing image annotation data to adapt to surveying and mapping domain knowledge.

[0193] In order to further demonstrate the practicality of the present invention, the present invention is explained in conjunction with the diagram shown in Figure (2).

[0194] Specifically, in Figure (2), a set of multi-temporal remote sensing images is constructed. Through the method proposed in this patent, after pre-training and fine-tuning of the labeled data, the result "The roads have been widened and extended, and the original buildings have been replaced with larger structures. Overall, the area has become more urbanized, featuring more buildings, roads, and less agricultural land." This result can be used to assist in the interpretation of the changing trend information of multi-temporal remote sensing images in the field of surveying and mapping.

[0195] The results show that the present invention realizes the statement that describes the semantics of changes in multi-temporal remote sensing images;

[0196] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating a generalized semantic description of multi-temporal remote sensing image changes, characterized by: The following steps are included: Step 1: Splice the multi-temporal remote sensing images acquired by the visible light sensor into a remote sensing image set in a time series manner, and extract the key frames in the remote sensing image set at equal time intervals; Step 2: Use the Transformer model to extract single-phase image spatial features from key frames in an image segmentation manner; Step 3: Use the Transformer model to fuse the spatial features of single-phase images with the temporal features of multi-phase images; Step 4: Use a multi-layer Transformer decoder to decode the temporal features of the multi-temporal image, and then obtain a summary descriptive word of the multi-temporal remote sensing image changes; Step 5: During the training phase, the Mask matrix in Equation (21) is used to suppress attention to the generated word queues summarizing the changes in multi-temporal remote sensing images, thereby ensuring the autoregressive generation capability of the multi-layer Transformer decoder and improving the efficiency of semantic parsing of the summarization of multi-temporal remote sensing image changes in the field of surveying and mapping. Where T represents the length of the sequence being processed, It is a triangular matrix: the values above the diagonal are negative infinity, and the values above the diagonal and below the diagonal are 0.

2. The method for generating a generalized semantic description of multi-temporal remote sensing image changes according to claim 1, wherein: Step 1 is implemented as follows: Step 1.1: Construct a remote sensing image set, specifically by splicing multi-temporal remote sensing images in a time series manner; Step 1.2: Extract key frames from the remote sensing image set at equal time intervals.

3. The method for generating a generalized semantic description of multi-temporal remote sensing image changes according to claim 1, wherein: Step 2 is implemented as follows: Step 2.1: Obtain the patch space sequence for image segmentation as shown in formula (1); Among them, stack represents the stacking operation of vectors, X t Represents the patch space sequence corresponding to the remote sensing image at time t, Represents the result of flattening the pixels of each patch into 1 dimension, N = HW / P 2 ; Step 2.2: Map the patch space sequence to the dimension consistent with the hidden layer of the Transformer model to obtain the embedded representation shown in formula (2); in, is a trainable parameter, D B represents the hidden layer dimension, The embedding of each patch and the embedding of the stacked patches respectively; Step 2.3: Use the method shown in formula (3) to encode the position of the patch space sequence of the embedded representation to obtain the hidden state sequence; Among them, x cls For additional learnable hidden states, Encode for position; Step 2.4: Decompose the hidden state sequence into multiple heads in the hidden layer dimension using the method shown in formula (4) to obtain multiple sub-hidden state sequences, each of which is called a head. Among them, concat means splicing on the hidden state dimension, H B represents the total number of attention heads, d B =D B / H B represents the hidden layer dimension of each head; z t l(h) represents the hth head obtained by step 2.4 in the lth layer network, l = 1, 2, ..., L, h = 1, 2, ..., H B , L represents the number of layers of the network; Step 2.5: Calculate the query matrix, key matrix, and value matrix of each header using the method shown in formula (5): in, is a trainable parameter matrix; Step 2.6: Use the self-attention mechanism on the hidden state sequence to obtain the single-head spatial self-attention weighted value as shown in formula (6); in Represents the self-attention weighted value of the h-th head of the l-th layer network; Step 2.7: Use the method shown in formula (7) to perform multi-head fusion on the single-head spatial self-attention weighted values; further, splice the single-head spatial self-attention weighted values on the hidden layer dimension to restore the hidden layer dimension of the weighted value to the full hidden layer dimension, and perform a linear transformation without changing the dimension; in, is a trainable parameter; Step 2.8: Use the method shown in formula (8) to perform residual connection and normalization on the input and the self-attention weighted value after multi-head spatial fusion, and then pass it through two layers of feedforward neural network to obtain the output of this layer network; Among them, LN represents the layer normalization operation; Step 2.9: Consider steps 2.4 to 2.8 as one layer of network calculation, and use the output of the previous layer as the input of the next layer. After multi-layer network calculation, the spatial features of the single-phase image are obtained.

4. The method for generating a generalized semantic description of multi-temporal remote sensing image changes according to claim 1, wherein: Step 3 is implemented as follows: Step 3.1: Use the method shown in formula (8) to encode the spatial features of the single-phase image into time series positions, and then obtain the hidden state sequence; Among them D C represents the hidden layer dimensions, E1, E2, ..., E T represents the single-phase image spatial features of all T phases obtained in step 2, Encode for position; Step 3.2: Decompose the hidden state sequence into multiple heads in the hidden layer dimension using the method shown in formula (9) to obtain multiple sub-hidden state sequences, each of which is called a head. Among them, concat means splicing on the hidden state dimension, H C represents the total number of attention heads, d C =D C / H C represents the hidden layer dimension of each head; e l(h) represents the hth head obtained by decomposing the lth layer network in step 3.2, l = 1, 2, ..., L C ,h=1,2,...,H C , L C Indicates the number of layers in the network; Step 3.3: Calculate the query matrix, key matrix, and value matrix of each header using the method shown in formula (10): in, is a trainable parameter matrix; Step 3.4: Use the self-attention mechanism on the sub-hidden state sequence to obtain the single-head temporal self-attention weighted value as shown in formula (11); in Represents the self-attention weighted value of the h-th head of the l-th layer network; Step 3.5: Use the method shown in formula (12) to perform multi-head fusion on the single-head temporal self-attention weighted values; further, splice the single-head temporal self-attention weighted values on the hidden layer dimension to restore the hidden layer dimension of the weighted value to the full hidden layer dimension, and perform a linear transformation without changing the dimension; in, is a trainable parameter; Step 3.6: Use the method shown in formula (13) to perform residual connection and normalization on the input and the time self-attention weighted value after multi-head fusion, and then pass it through two layers of feedforward neural network to obtain the output of this layer network; have been l′ =LN(ie l +N l ) (13) Among them, LN represents the layer normalization operation; Step 3.7: Consider steps 3.2 to 3.6 as one layer of network calculation, and use the output of the previous layer as the input of the next layer. After multi-layer network calculation, the fused multi-temporal image temporal features are obtained.

5. The method for generating a generalized semantic description of multi-temporal remote sensing image changes according to claim 1, wherein: Step 4 is implemented as follows: Step 4.1: Use the method shown in formula (14) to perform position encoding of the multi-temporal image temporal features in text order to obtain the text feature hidden state sequence; Among them, S1, S2, ..., S T The fused multi-phase image temporal features generated in step 3; S T+1 ,...,S T+m is the word embedding corresponding to the generated word, m represents its length, and is initially empty; Encode for position; Step 4.2: Use the method shown in formula (15) to perform multi-head decomposition on the hidden layer dimension of the text feature hidden state sequence to obtain multiple sub-text hidden state sequences, each of which is called a head; Among them, concat means splicing on the hidden state dimension, H D represents the total number of attention heads, d D =D D / H D represents the hidden layer dimension of each head; s l(h) represents the hth head obtained by decomposing the lth layer network, l=1,2,...,L D ,h=1,2,...,H D , L D Indicates the number of layers in the network; Step 4.3: Calculate the Query matrix, Key matrix, and Value matrix of each header using the method shown in formula (16): in, is a trainable parameter matrix; Step 4.4: Use the self-attention mechanism on the sub-text hidden state sequence to obtain the single-head text self-attention weighted value as shown in formula (17); in Represents the single-head text self-attention weighted value of the h-th head of the l-th layer network; Step 4.5: Use the method shown in formula (18) to perform multi-head fusion on the single-head text self-attention weighted values; further, splice the single-head text self-attention weighted values on the hidden layer dimension to restore the hidden layer dimension of the weighted value to the full hidden layer dimension size, and perform a linear transformation without changing the dimension; in, is a trainable parameter; Step 4.6: Use the method shown in formula (19) to perform residual connection and normalization on the current layer input and the text self-attention weighted value after multi-head fusion, and then pass through two layers of feedforward neural network to obtain the output of this layer network; s l′ =LN(s l +F l ) (19) Among them, LN represents the layer normalization operation; Step 4.7: Treat steps 4.2 to 4.6 as a single layer of network calculation, and use the output of the previous layer as the input of the next layer. After multi-layer network calculation, the probability distribution of the next word is obtained as shown in formula (20); P(w t+m+1 |w1,w2,...,w t+m )=softmax(W out h t+m +b out ) (20) in is a trainable parameter, d voc is the vocabulary size, w i represents the i-th word of the generated sentence, h t+m Represents the hidden state at position t+m in the hidden state sequence; selects the next word based on the probability distribution of the next word using a selection strategy such as top-k or beam-search, and places the word embedding of the selected word at the end of the input queue s; Step 4.8: Iterate steps 4.1 to 4.7 in a loop until a sentence end symbol is generated or the maximum sentence length is reached, and then stop generating the generalized semantic description words for the multi-temporal remote sensing image changes.

Images