A method for generating semantic description text of time series remote sensing image changes

Abstract

The present invention provides a method for generating text describing semantic descriptions of changes in time-series remote sensing images, comprising the following steps: Step A: extracting local and global features of time-series remote sensing images; Step B: fusing the extracted global and local features of single-phase images through a multi-level feature crossover method to obtain multi-level fusion features of the image; Step C: inputting the semantic description text of the changes into a third pre-trained neural network model to extract text features; Step D: projecting the multi-level fusion features and text features of the image into the same feature domain to achieve a mapping relationship between the change features of the time-series remote sensing images and the semantic features of the text; and Step E: predicting and outputting the semantic description sentence of the changes with the highest probability for test data based on the mapping relationship as the final generated text. The present invention directly converts the change detection interpretation image results into a natural language description of surface changes, eliminating the need for manual summary and understanding of the change detection results and automatically generating human-understandable text description outputs. The invention can provide support for urban planning, regional monitoring, insurance claims, and the like.

Description

Technical Field

[0001] The present invention relates to two technical fields of computer vision and natural language processing, and in particular to a method for generating text describing semantic descriptions of changes in time series remote sensing images. Background Art

[0002] Semantic description of changes in time-series remote sensing imagery lies at the intersection of computer vision and natural language processing. Change detection in time-series remote sensing imagery aims to detect and identify changes in surface cover based on multiple observations within a timeframe. Compared to traditional two- or multi-temporal change detection, change detection in time-series remote sensing imagery, due to its higher temporal resolution, can accurately extract the timing of changes, capturing long-term and slow trends in surface features and enabling analysis of how the surface changes over time.

[0003] However, existing time series remote sensing image change detection methods cannot achieve an end-to-end link from image interpretation results to natural language descriptions. It is difficult to directly convert the change detection interpretation image results into natural language descriptions of surface changes. In actual engineering applications, manual summary and understanding of the change detection results are often required to generate descriptive statements, making it impossible to directly use the time series change detection results for actual engineering applications. Summary of the Invention

[0004] (1) Technical issues to be resolved

[0005] In order to solve the problems that existing methods are unable to achieve end-to-end mapping from change detection and interpretation results of time series remote sensing images to change semantic natural language descriptions, are unable to automatically generate change semantic description text based on change detection results of time series remote sensing images, are difficult to directly convert change detection and interpretation image results into natural language descriptions of surface changes, and automatically generate human-understandable text description outputs, the present invention provides a method for generating change semantic description text for time series remote sensing images.

[0006] (2) Technical solution

[0007] The present invention provides a method for generating semantic description text of time series remote sensing image changes, comprising steps A: image feature extraction: extracting local features and global features of time series remote sensing images;

[0008] Step B: Multi-level feature fusion: The global features and local features extracted from the single-phase image are fused through a multi-level feature crossover method to obtain the multi-level fusion features of the image, which are used to represent the semantic change information between time series remote sensing images;

[0009] Step C: Text feature extraction: Input the change semantic description text into the third pre-trained neural network model to extract text features;

[0010] Step D: Image-text feature matching: Project the multi-level fusion features of the image and the text features into the same feature domain to achieve the mapping relationship between the change features of the time series remote sensing image and the semantic features of the text;

[0011] Step E: Text prediction generation: Based on the mapping relationship between the time series remote sensing image change features and text semantic features generated by the training data, the test data is predicted and the semantic description sentence with the highest probability of change is output as the final generated text.

[0012] Furthermore, in step A, local features are extracted through a first image classification pre-training model, which is built using a deep convolutional neural network, and the deep convolutional neural network is ImageNet, VGGNet, or ResNet; preferably, the Conv5_3 layer in the VGG16 network model is used as the output layer to extract image features to retain rich detail information of ground objects in remote sensing images, and the feature dimension is 512.

[0013] Furthermore, in step A, global features are extracted through a second image classification pre-training model, and the second image classification pre-training model is built using Transformer; the second image classification pre-training model is a stack of several spatiotemporal multi-head self-attention modules, and the spatiotemporal multi-head sub-attention module includes a cascaded spatial multi-head self-attention, temporal multi-head self-attention and a feedforward neural network, so that the global features include both the temporal features of the image and the spatial correlation relationship of the image.

[0014] Furthermore, in the second image classification pre-training model,

[0015] (1) Each input image is transformed into Tokens that can be processed by Transformer, the size of each Token is C1P 2 ×1;

[0016] (2) Tokens from the same image but different spatial positions are fed into the spatial multi-head self-attention module to obtain the Q (Query), K (Key), and V (Value) required for calculating self-attention, as shown below:

[0017]

[0018]

[0019]

[0020] Among them, LN (Layer Normalization) is a normalization operation at the feature level. and Then they are respectively related to the t-th image of the l-th layer and The corresponding learnable linear mapping layer; then, the new Tokens of the t-th image after spatial multi-head self-attention are obtained by the following calculation, that is Where SM is the Softmax operation.

[0021]

[0022] Then, the tokens in the same spatial position but different images are fed into the temporal multi-head self-attention module to obtain tokens with fused temporal context.

[0023] (3) Output layer, each token after spatial and temporal multi-head self-attention processing is passed to the feed-forward neural network FFN to obtain the corresponding output, with the n-th position of the t-th image in the l-th layer as the output. For example, the output of the feedforward neural network is:

[0024]

[0025] Where W1∈R D′×D and W2∈R D×D Both are learnable linear mapping layers, b1∈R D and b2∈R D is a learnable bias, and FFN is a feedforward neural network.

[0026] Furthermore, a linear mapping layer is added after Token to transform each Token from C1P 2 ×1 is mapped to D×1 size, and the nth Token of the tth image is represented as:

[0027] z (t，n) =Ex (t，n) +p (t，n)

[0028] in, is a learnable linear mapping layer, R represents the second image classification pre-training model network structure, D is the Token output dimension, P is the sliding window size, and p (t，n) It is a D-dimensional learnable positional encoding, representing the input image I t The location information in the original image; x represents the input image I t .

[0029] Furthermore, the step B includes:

[0030] Sub-step B1: Local features Zoom to

[0031] Sub-step B2: Subtract the image features of two consecutive time phases in the time series remote sensing image. The subtraction operation is defined as: |f loc 1-f loc 2|2,|f loc 2-f loc 3|2,…,|f loc k-1 -f loc k |2 and |f glo 1-f glo 2|2,|f glo 2-f glo 3|2,…,|f glo k-1 -f glo k |2;

[0032] Generated k-1 groups of local feature differences and global feature differences;

[0033] Sub-step B3: Cascade the time series remote sensing images after the “subtraction” operation to obtain the multi-level fusion feature f of the two consecutive time phase images t I =[|f loc t -f loc t+1 |2 f loc t f loc t+1 |f glo t -f glo t+1 |2 f glo t f glo t+1 ](t=1,2,…,k-1);

[0034] Sub-step B4: Use the classifier to classify the multi-level fusion feature f t I (t=1,2,…,k-1) performs binary classification, i.e. detects the changed area and outputs the binary result of change detection;

[0035] Sub-step B5: Use the Softmax function to obtain the probability distribution of the change categories of the two consecutive time-phase images, and select the category corresponding to the item with the largest probability after normalization as the change prediction category of the two consecutive time-phase images.

[0036] Furthermore, the step D comprises:

[0037] Sub-step D1: Pass two learnable linear mapping layers and Map the image and text features separately and

[0038] Sub-step D2: Based on the idea of ​​contrastive learning, two constraints are imposed within the same batch: one is to make the multi-level fusion features and text features from the same image pair as close as possible, and the other is to make the multi-level fusion features and text features from different image pairs as far apart as possible.

[0039] Furthermore, the sub-step D2 includes:

[0040] (1) Calculate the similarity between the multi-level fusion features of the continuous two-phase image and the change semantic description text, and construct the similarity matrix M∈R (T-1)×(T-1) :

[0041] M=(g CI )(g CS ) T

[0042] The size of M is (k-1)×(k-1), where k represents the number of multi-temporal remote sensing images and T represents matrix transpose;

[0043] (2) Based on the constructed M matrix, maximize its diagonal elements and minimize the elements in other positions.

[0044] Furthermore, the step E comprises:

[0045] Sub-step E1: For the time series remote sensing image to be tested, execute steps A and B to obtain the multi-level fusion features of the image pair; then, using a pre-set paradigm and a custom corpus, generate a predicted description text, and pass the generated description text through step C to obtain the text features corresponding to the image. Then, according to sub-step D2, calculate the similarity between the multi-level fusion features and the text features. The text with the highest similarity is the final generated change semantic description text.

[0046] Furthermore, the step B4 includes:

[0047] Sub-step B41: Network structure. Since the output values ​​of change detection are only two, namely "changed" and "unchanged", the value of the 0th channel at the same spatial position (x, y) of the present invention is Indicates the probability that the position has not changed, the value of the first channel Indicates the probability of the position changing;

[0048] Sub-step B42: First, use the convolution kernel Conv_k1_o128 with a step size of 1, padding of 0, and kernel size of 1 to reduce the dimension of the feature to 128, and then continuously use the convolution kernel with a step size of 1, padding of 1, and kernel size of 3 to process the feature, and output a feature map with a channel dimension of 64; then use the convolution kernel with a step size of 1, padding of 1, and kernel size of 3 to process the feature to obtain a two-dimensional feature, and use Softmax to probabilize the two-dimensional feature map to obtain the output probability map

[0049] Sub-step B43: In the inference phase, the Argmax operation is performed in the channel dimension to output the final change detection result.

[0050] (3) Beneficial effects

[0051] (1) Ability to complete change detection based on time series remote sensing images and automatically generate semantic description text of changes;

[0052] (2) It can directly convert the change detection interpretation image results into natural language descriptions of surface changes, eliminating the need for manual summary and understanding of the change detection results, and automatically generates human-understandable text description outputs;

[0053] (3) The invention can provide technical support for urban planning, regional monitoring, insurance claims and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic flow chart of a generation method according to an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of generation results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0057] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0058] The present invention proposes a method for generating semantic description text of time series remote sensing image changes. Figure 1 Shown, including:

[0059] Step A: Image feature extraction: The time series remote sensing image is input into the pre-trained neural network model to extract the global and local features of the input time series remote sensing image and complete the semantic representation of the target in the remote sensing image;

[0060] The time series remote sensing image I t (t=1,2,...,k) (k represents the total number of input time series remote sensing images) is input into the pre-trained neural network model to extract the input time series remote sensing images I t The local feature f loc and the global feature f glo , complete the semantic representation of the target in the remote sensing image, that is, use features to represent the image. Step A includes:

[0061] Sub-step A1: Local feature extraction, for each input remote sensing image Among them, C1 is the number of channels of the input remote sensing image, which can be 1 (such as SAR) or 3 (such as optical), H1 and W1 are its height and width, and the first image classification pre-training model built by deep convolutional neural network (CNN) is used to extract the input time series remote sensing image (I1, I2, ..., I k ) local image features H2 and W2 are its height and width, and C2 represents the dimension of the feature, which can be determined manually. The first image classification pre-training model can be ImageNet, VGGNet, ResNet, etc.

[0062] In one embodiment, the Conv5_3 layer in the VGG16 network model is used as the output layer to extract image features to retain the rich detail information of the ground objects in the remote sensing image, and its feature dimension is 512; the remote sensing image I of each phase is output. t The local eigenvector f of (t=1, 2, ..., k) loc t(t=1, 2, ..., k); Among them, VGG16 has a total of 16 layers, 13 convolutional layers and 3 fully connected layers, and 5 pooling layers; the 13 convolutional layers are: 2 two-dimensional convolutions with a convolution kernel size of 3 and an output channel of 64, 2 two-dimensional convolutions with a convolution kernel size of 3 and an output channel of 128, 2 two-dimensional convolutions with a convolution kernel size of 3 and an output channel of 128 and 1 two-dimensional convolution with a convolution kernel size of 1 and an output channel of 128, 2 two-dimensional convolutions with a convolution kernel size of 3 and an output channel of 512 and 1 convolution The kernel size is 1 and the output channel number is 512. There are two-dimensional convolutions with a kernel size of 3 and an output channel number of 512, and one two-dimensional convolution with a kernel size of 1 and an output channel number of 512. The last three layers are fully connected layers, including two fully connected layers with an output channel number of 4096 and one softmax layer. The pooling layers are located after the second 64-core convolution layer, the second 128-core convolution layer, the third 256-core convolution layer, the third 512-core convolution layer, and the sixth 512-core convolution layer.

[0063] Sub-step A2: Global feature extraction, using Transformer to build a second image classification pre-training model to extract the global image features of the input time series remote sensing image Where C3 is the number of feature channels, determined by the image itself, and H1 and W1 are the height and width of the image. The model uses multiple self-attention modules, including spatial multi-head self-attention (S-MHSA), temporal multi-head self-attention (T-MHSA), and a feed-forward neural network (FFN), to extract temporal and spatial characteristics of the image. The purpose is to ensure that the extracted global features contain both the temporal characteristics of the image and the spatial correlation relationship of the image. The details are as follows:

[0064] Sub-step A21: Construct a second image classification pre-training model, which contains M stacked temporal and spatial multi-head self-attention modules (TS-MHSA). Each TS-MHSA module consists of three parts connected in series: spatial multi-head self-attention (S-MHSA), temporal multi-head self-attention (T-MHSA), and a feed-forward network (FFN).

[0065] Sub-step A22: Token design, for each input image Through a sliding window with a width and height of P, it is divided into tokens that can be processed by the second image classification pre-training model. The size of each token is C1P 2 ×1; In order to reduce the number of parameters of the model, a linear mapping layer is added after Token to transform each Token from C1P 2 ×1 is mapped to D×1 size, as shown below, the nth Token of the tth image is represented as:

[0066] z (t，n) =Ex (t，n) +p (t，n)

[0067] in, is a learnable linear mapping layer, R represents the second image classification pre-training model network structure, D is the Token output dimension, P is the sliding window size, and p (t，n) It is a D-dimensional learnable positional encoding, representing the input image I t The location information in the original image; x represents the input image I t .

[0068] Sub-step A23: Self-attention module design: Tokens from the same image but different spatial positions are fed into the spatial multi-head self-attention module to obtain the Q (Query), K (Key), and V (Value) required for calculating self-attention, as shown below:

[0069]

[0070]

[0071]

[0072] Among them, LN (Layer Normalization) is a normalization operation at the feature level. and Then they are respectively related to the t-th image of the l-th layer and The corresponding learnable linear mapping layer; after that, the new Tokens of the t-th image after spatial multi-head self-attention can be obtained by the following calculation, that is Where SM is the Softmax operation.

[0073]

[0074] Similarly, tokens in the same spatial position but in different images are fed into the temporal multi-head self-attention module to obtain tokens with fused temporal context. For example, the output of the temporal multi-head attention at the nth position of the multi-temporal image in the lth layer is:

[0075]

[0076] in and The calculation method is as follows:

[0077]

[0078]

[0079]

[0080] Sub-step A24: Output layer, pass each token after spatial and temporal multi-head self-attention processing to the feed-forward neural network FFN to obtain the corresponding output, with the n-th position of the t-th image in the l-th layer as the output. For example, the output of the feedforward neural network is:

[0081]

[0082] Where W1∈R D′×D and W2∈R D×D Both are learnable linear mapping layers, b1∈R D and b2∈R D is a learnable bias, FFN is a feedforward neural network;

[0083] In an example, we select P as 16 and take 8 multi-temporal remote sensing images of size 224×224 as input, and get a total of 8×14×14, or 8×196 tokens. The dimension D of each token is 768. After stacking 4 layers of TS-MHSA modules to integrate the global features into the tokens of each image, we finally get the multi-temporal features as the global features f glo .

[0084] Step B: Multi-level feature fusion: The global features and local features extracted from the single-phase image are fused through a multi-level feature cross-processing method to represent the semantic change information between time series remote sensing images; since remote sensing images may contain both large and small target changes, in order to take into account targets of different sizes, both global features and local features are required in the feature extraction process. Only by fusing the global and local features can we ensure that small changes are not lost in the subsequent detection process.

[0085] Sub-step B1: Feature interpolation, due to local features and the input image The width and height are inconsistent. In order to ensure that the final change detection image output is consistent with the original image size, the local features are first interpolated using bilinear interpolation. Zoom to

[0086] Sub-step B11: Since the output values ​​of the change detection are only two, namely "changed" and "unchanged", the value of the 0th channel at the same spatial position (x, y) of the present invention is Indicates the probability that the position has not changed, the value of the first channel Indicates the probability of the position changing;

[0087] Sub-step B12: First, use the convolution kernel Conv_k1_o128 with a step size of 1, padding of 0, and kernel size of 1 to reduce the dimension of the local features to 128, and then continuously use the convolution kernel with a step size of 1, padding of 1, and kernel size of 3 to process the reduced dimension features, and output a feature map with a channel dimension of 64; then use the convolution kernel with a step size of 1, padding of 1, and kernel size of 3 to process the features to obtain a two-dimensional feature, and use Softmax to probabilize the two-dimensional feature map to obtain the output probability map

[0088] Sub-step B13: In the inference phase, perform an Argmax operation on the channel dimension to output the final change detection result, as shown below, where 0 indicates no change and 1 indicates a change.

[0089]

[0090] Sub-step B2: Subtract the image features of two consecutive time phases in the time series remote sensing image to generate a local feature difference group and a global feature difference group. Specifically, the local feature vector f output by sub-step A1 is loc t (t=1,2,…,k) and the global feature vector f output by sub-step A2 glo t (t=1,2,…,k) performs subtraction operation; the subtraction operation is defined as follows: |f loc 1-f loc 2|2,|f loc 2-f loc 3|2,…,|f loc k-1 -f loc k |2 and |f glo1-f glo 2|2,|f glo 2-f glo 3|2,…,|f glo k-1 -f glo k |2; where |AB|2 represents the Euclidean distance between A and B. For k multi-temporal remote sensing images, the features of two consecutive temporal images are "subtracted" to generate k-1 sets of local feature differences and global feature differences;

[0091] Sub-step B3: Feature cascade, cascade the time series remote sensing images after the "subtraction" operation to obtain the multi-level fusion feature f of the two consecutive time phase images t I =[|f loc t -f loc t+1 |2 f loc t f loc t+1 |f glo t -f glo t+1 |2 f glo t f glo t+1 ](t=1,2,…,k-1); It should be noted that for k multi-temporal remote sensing images, k-1 groups of local feature differences and global feature differences are generated. Therefore, the generated multi-level fusion feature f t I (t=1,2,…,k-1) There are only k-1 groups, and the difference between the first and last images is already included in the middle difference feature map, so they are no longer compared;

[0092] Sub-step B4: perform change region detection and output the change detection results; specifically: use the classifier to classify the multi-level fusion feature f t I (t=1,2,…,k-1) performs binary classification. The classifier can use models such as VGGNet and ResNet. The top network is composed of multiple fully connected layers. The feature dimension output by the last fully connected layer is consistent with the number of change detection output categories, that is, the number of categories is 2. The last layer is the Softmax layer. The Softmax function predicts the output change detection category, that is, the change class / no change class.

[0093] Sub-step B5: The Softmax function obtains the probability distribution of the change categories of the two consecutive time-phase images, and selects the category corresponding to the item with the largest probability after normalization as the change prediction category of the two consecutive time-phase images.

[0094] The change detection result is a binary image with the same width and height as the original input image. Each pixel in the image is either 0 or 1, where 0 indicates no change and 1 indicates a change.

[0095] Step C: Text Feature Extraction: Each set of time-series remote sensing images contains a text document, the semantic description document, which uses natural language to describe the changes occurring between two consecutive time-phase images. In this step, the third pre-trained neural network model is used to extract text features from the document and complete the semantic representation of the words in the text.

[0096] The third pre-trained neural network model is built using a convolutional neural network based on an attention mechanism. The text classification pre-training model can be word2vec, glove, BERT and other models; each text S is output t (t=1,2,…,k-1) text feature vector f t S (t=1,2,…,k-1); the output text can be in English, Chinese, Japanese and other languages.

[0097] In one embodiment, the BERT model is used as the third pre-trained neural network model for English text descriptions. First, the text is pre-processed using BertTokenizer, including word segmentation and indexing. Second, a self-supervised learning method is run on a large amount of open source corpus to pre-train a basic model, namely BERT-base. Third, the input semantic description text is used to fine-tune the basic model, that is, to optimize and adjust some parameters of BERT-base. Finally, it can be used to extract the input English text S. t (t=1,2,…,k-1) describes the features and selects cls_token to represent the entire sentence, that is Among them, D s It is 768.

[0098] First, [CLS] and [SEP] markers are added to the entire front-end and back-end respectively to indicate the beginning and end of the sentence; secondly, the sentence is divided into several words (tokens) according to the spaces between the words in the sentence; thirdly, according to the vocabulary (vocab.txt) provided by the pre-trained model BERT, each word is converted into a number indicated in the vocabulary (such as: a→10, an→11...), and complex words may be divided into multiple segments represented by multiple numbers (for example: aaaa is divided into "a, ##a, ##a, ##a", ## indicates connection with the previous word). In this way, an entire sentence is encoded as a vector composed of numbers; in addition, in addition to encoding sentences, since multiple sentences are often input into the model during training, and the lengths of the sentences are different, these sentences are padded in the encoding process to become vectors of the same length and input into the model. Therefore, the encoded vector inevitably has invalid positions, so it is also necessary to set a mask (i.e., attention_mask) vector to instruct the model to perform calculations only at valid positions in the sentence;

[0099] Fine-tuning: This invention uses a pre-trained model as the backbone of its own model. Although BERT can generate a vector representation of an entire sentence, this is done on large-scale data that is irrelevant to the current task. Therefore, during model training, BERT, as the base model, will still be trained in gradient backpropagation for its model parameters. However, generally speaking, the learning rate set for training the BERT model will be lower than that of other parts of the model.

[0100] Step D: Image-text feature matching: Multi-level fusion feature f t I and text feature vector f t S Projecting to the same feature domain (this feature domain can be any one of the image or text features, or a third domain) realizes the mapping between the time series remote sensing image change features and the text semantic features, that is, generates the similarity matrix M; specifically, it includes:

[0101] Sub-step D1: Feature mapping. In order to compare the features of text and image in the same feature space, the present invention uses two learnable linear mapping layers and Map the multi-level fusion features and text feature vectors respectively to obtain and As shown below:

[0102]

[0103]

[0104] In addition, in order to ensure the scale consistency of the two values, L2 normalization is required after mapping. The final calculation result is:

[0105]

[0106]

[0107] Sub-step D2: Obtain the mapped image features in sub-step D1 and text features Based on the idea of ​​contrastive learning, within the same batch, two main constraints are made: one is to make the image features and text features from the same image pair as close as possible, and the other is to make the image features and text features from different image pairs as far apart as possible.

[0108] First, calculate the similarity between the two consecutive time-phase image pairs and the change semantic description text, and construct the similarity matrix M∈R (T-1)×(T-1) :

[0109]

[0110] Note that the size of M is (k-1)×(k-1), where k represents the number of multi-temporal remote sensing images and T represents matrix transpose.

[0111] Secondly, based on the constructed M matrix, maximize its diagonal elements and minimize the elements in other positions;

[0112] Finally, after obtaining the similarity matrix M, the present invention minimizes the similarity matrix through the binary cross entropy (BCE) loss function. The final loss function J is as follows:

[0113] J=BCE(Flatten(M),Flatten(I))

[0114] Where I represents the expanded diagonal matrix, and Flatten means forcibly transforming the multi-dimensional feature into a one-dimensional feature.

[0115] Step E: Text prediction generation: Based on the mapping relationship between the time series remote sensing image change features and text semantic features generated by the training data, the test data is predicted and the semantic description sentence with the highest probability of change is output as the final generated text.

[0116] Input the time series remote sensing image to be tested into the trained network model, and obtain the multi-level fusion features of the image pair through steps A and B; then generate the predicted description text through the pre-set paradigm (i.e. text output format) and the customized corpus, generally output according to the specified format, and pass the generated description text through step C to obtain the text features corresponding to the image, and then calculate the similarity between the multi-level fusion features of the image and the features of each text according to sub-step D2. The text with the highest similarity is the final generated semantic description text of the change. The final expected result is as follows Figure 2 As shown in the figure, a set of descriptive text can be automatically generated from a time series of remote sensing images. This text can provide a natural language description of the changes between two consecutive time phases. This method is not only fast and eliminates the need for iterative data generation similar to RNN methods, but also provides a more accurate description of the scene and is more suitable for practical application scenarios.

[0117] It should be noted that any implementation methods not shown or described in the drawings or the main text of the specification are known to those skilled in the art and are not described in detail. In addition, the above definitions of the various elements are not limited to the various methods mentioned in the embodiments. Those skilled in the art can easily modify or replace them. For example:

[0118] (1) The image local feature extraction network in step A can be replaced by other deep neural network models, such as VGG21, ResNet, etc. These deep neural network models are well-known methods in the art and will not be described in detail here;

[0119] (2) The text feature extraction network in step C can be replaced by other models, such as word2vec, glove, etc. These models are well-known methods in the art and will not be described in detail here;

[0120] (3) The construction method of the deep convolutional neural network can be empirically set according to the application purpose, including the number of network layers, the number of convolutional layers, the number of pooling layers, the number of convolution filters, the convolution kernel size, and the pooling scale; these parameter settings are well-known methods in the field and will not be described in detail here.

[0121] (4) The time series remote sensing images in the present invention are not limited to homologous image sequences, but can also be heterologous image sequences;

[0122] (5) The above embodiments can be mixed and matched with each other or with other embodiments based on design and reliability considerations. That is, the technical features of different embodiments can be freely combined to form more embodiments.

[0123] (6) Although the deep learning method of the present invention adopts a deep convolutional neural network model, the present invention is not limited to this. There are many options for deep learning methods, and the present invention can also adopt other deep learning methods.

[0124] The specific embodiments described above provide a detailed description of the objectives, technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating semantic description text of time series remote sensing image changes, characterized in that: include: Step A: Image feature extraction: extract local and global features of time series remote sensing images; Step B: Multi-level feature fusion: The global features and local features extracted from the single-phase image are fused through a multi-level feature crossover method to obtain the multi-level fusion features of the image, which are used to represent the semantic change information between time series remote sensing images; Step C: Text feature extraction: Input the change semantic description text into the third pre-trained neural network model to extract text features; Step D: Image-text feature matching: Project the multi-level fusion features of the image and the text features into the same feature domain to achieve the mapping relationship between the change features of the time series remote sensing image and the semantic features of the text; Step E: Text prediction and generation: Based on the mapping relationship between the time series remote sensing image change features generated by the training data and the text semantic features, the test data is predicted and the semantic description sentence with the highest probability of change is output as the final generated text; The step B comprises: Sub-step B1: Local features Zoom to Sub-step B2: Subtract the image features of two consecutive time phases in the time series remote sensing image. The subtraction operation is defined as: |f loc 1-f loc 2|2,|f loc 2-f loc 3|2,…,|f loc k-1 -f loc k |2 and |f glo 1-f glo 2|2,|f glo 2-f glo 3|2,…,|f glo k-1 -f glo k |2; Generated k-1 groups of local feature differences and global feature differences; Sub-step B3: Cascade the time series remote sensing images after the "subtraction" operation to obtain the multi-level fusion feature f of the two consecutive time phase images t I =[|f loc t -f loc t+1 |2 f loc t f loc t+1 |f glo t -f glo t+1 |2 f glo t f glo t+1 ](t=1,2,…,k-1); Sub-step B4: Use the classifier to classify the multi-level fusion feature f t I (t=1,2,…,k-1) performs binary classification, i.e. detects the changed area and outputs the binary result of change detection; Sub-step B5: Use the Softmax function to obtain the probability distribution of the change categories of the two consecutive time-phase images, and select the category corresponding to the item with the largest probability after normalization as the change prediction category of the two consecutive time-phase images.

2. The generation method according to claim 1, characterized in that In step A, local features are extracted through a first image classification pre-training model, which is built using a deep convolutional neural network, such as ImageNet, VGGNet, or ResNet.

3. The generation method according to claim 2, characterized in that The Conv5_3 layer in the VGG16 network model is used as the output layer to extract image features to retain the rich detail information of the ground objects in the remote sensing image. The feature dimension is 512.

4. The generation method according to claim 1, characterized in that In step A, global features are extracted through a second image classification pre-training model, and the second image classification pre-training model is built using Transformer; the second image classification pre-training model is a stack of several spatiotemporal multi-head self-attention modules, and the spatiotemporal multi-head sub-attention module includes a cascaded spatial multi-head self-attention, temporal multi-head self-attention and a feedforward neural network, so that the global features include both the temporal features of the image and the spatial correlation relationship of the image.

5. The generation method according to claim 4, characterized in that In the second image classification pre-training model, (1) Each input image is transformed into It is divided into Tokens that can be processed by Transformer. The size of each Token is C1P 2 ×1; (2) Tokens from the same image but different spatial positions are fed into the spatial multi-head self-attention module to obtain the Q (Query), K (Key), and V (Value) required for calculating self-attention, as shown below: Among them, LN (Layer Normalization) is a normalization operation at the feature level. and Then they are respectively related to the t-th image of the l-th layer and The corresponding learnable linear mapping layer, z is the Token representation; then, the new Tokens of the t-th image after spatial multi-head self-attention are obtained by the following calculation, expressed as Where SM is the Softmax operation; Then, the tokens in the same spatial position but different images are fed into the temporal multi-head self-attention module to obtain tokens with fused temporal context. (3) Output layer, each token after spatial and temporal multi-head self-attention processing is passed to the feed-forward neural network FFN to obtain the corresponding output, with the n-th position of the t-th image in the l-th layer as the output. For example, the output of the feedforward neural network is: Where W1∈R D′×D and W2∈R D×D′ Both are learnable linear mapping layers, b1∈R D′ and b2∈R D is a learnable bias, and FFN is a feedforward neural network.

6. The generation method according to claim 5, characterized in that Token is followed by a linear mapping layer to transform each Token from C1P 2 ×1 is mapped to D×1 size, and the nth Token of the tth image is represented as: z (t,n) =Ex (t,n) +p (t,n) in, is a learnable linear mapping layer, R represents the second image classification pre-training model network structure, D is the Token output dimension, P is the sliding window size, and p (t,n) It is a D-dimensional learnable positional encoding, representing the input image I t The location information in the original image; x represents the input image I t .

7. The generation method according to claim 1, characterized in that The step D comprises: Sub-step D1: Pass two learnable linear mapping layers and Map the image and text features separately and Sub-step D2: Based on the idea of ​​contrastive learning, two constraints are imposed within the same batch: one is to make the multi-level fusion features and text features from the same image pair as close as possible, and the other is to make the multi-level fusion features and text features from different image pairs as far apart as possible.

8. The generation method according to claim 7, characterized in that The sub-step D2 comprises: (1) Calculate the similarity between the multi-level fusion features of the continuous two-phase image and the change semantic description text, and construct the similarity matrix M∈R (T-1)×(T-1) : M=(g CI )(g CS ) T The size of M is (k-1)×(k-1), where k represents the number of multi-temporal remote sensing images and T represents matrix transpose; (2) Based on the constructed M matrix, maximize its diagonal elements and minimize the elements in other positions.

9. The generation method according to claim 8, characterized in that After obtaining the similarity matrix M, the similarity matrix is ​​minimized through the binary cross entropy loss function. The binary cross entropy loss function J is: J=BCE(Flatten(M),Flatten(I)) Where I represents the expanded diagonal matrix, and Flatten means forcibly transforming the multi-dimensional feature into a one-dimensional feature.

10. The generation method according to claim 8, characterized in that The step E comprises: Sub-step E1: For the time series remote sensing image to be tested, execute steps A and B to obtain the multi-level fusion features of the image pair; then, using a pre-set paradigm and a custom corpus, generate a predicted description text, pass the generated description text through step C to obtain the text features of the image, and then calculate the similarity between the multi-level fusion features and the text features according to sub-step D2. The text with the highest similarity is the final generated change semantic description text.

11. The generation method according to claim 1, characterized in that The step B4 comprises: Sub-step B41: At the same spatial position (x, y), the value of the 0th channel Indicates the probability that the position has not changed, the value of the first channel Indicates the probability of the position changing; Sub-step B42: First, use the convolution kernel Conv_k1_o128 with a step size of 1, padding of 0, and kernel size of 1 to reduce the dimension of the feature to 128, and then continuously use the convolution kernel with a step size of 1, padding of 1, and kernel size of 3 to process the feature, and output a feature map with a channel dimension of 64; then use the convolution kernel with a step size of 1, padding of 1, and kernel size of 3 to process the feature to obtain a two-dimensional feature, and use Softmax to probabilize the two-dimensional feature map to obtain the output probability map Sub-step B43: Perform Argmax operation in the channel dimension and output the final change detection result.

Images