Method for generating remote sensing image from text

Abstract

The invention discloses a method for generating a remote sensing image based on a text. NLP (Natural Language Processing) and CV (Computer Vision) technologies are fused. According to the method, a high-quality remote sensing image is generated by analyzing text description and carrying out hierarchical prototype learning and dynamic prototype learning by utilizing a dynamic hierarchical prototype block. Specifically, a text image pair is processed by an encoder to obtain a text image mark, a joint mark formed by splicing the text image pair and the text image mark is input into a dynamic hierarchical prototype layer to extract features, and the method enhances the memory and information retrieval capability of a model by combining a Hopfield network and a self-attention mechanism. Therefore, the richness and accuracy of remote sensing image feature representation are improved. Besides, a dynamic prototype learning strategy is adopted, so that the model can gradually learn and adapt to more prototypes, and higher robustness and accuracy are shown when complex data are processed. Finally, the method can generate a remote sensing image which is visually consistent with the text description and has high-quality details.

Description

Technical Field

[0001] The present invention relates to the intersecting field of remote sensing technology and artificial intelligence, and in particular to a method for generating remote sensing images from text. Background Art

[0002] Remote sensing images are regarded as "photos" of the earth and are important earth observation data. They can generate a wide range of real-time data from a remote location and play a key role in land and urban planning, environmental monitoring, military target identification and other fields. With the advancement of science and technology, substantial progress has been made in the understanding of remote sensing images, and the advancement of geographic information systems has promoted the visualization of remote sensing data. In addition, the development of machine learning and artificial intelligence has transformed text descriptions into highly semantically relevant images, connecting natural language with computer vision, and also promoted the development of artificial intelligence in "understanding".

[0003] As widely recognized in the field, there are significant differences in views and contents between natural images and remote sensing images. Remote sensing images present complex and diverse data, requiring a broad understanding of the interactions between various geographical aspects and phenomena. Natural images are recorded in a horizontal perspective, with the foreground and background clearly visible in human sight, and the protagonists in the image often occupy most of the area and are easy to identify. On the other hand, remote sensing images are recorded in a vertical view (God's view), contain a large number of objects and more complex situations, with the foreground and background difficult to distinguish, and the protagonists in the image are similar in size to the background or even hidden in the background, making it difficult even for humans to detect. These factors make generating remote sensing images more difficult than generating natural images. In addition, with the explosive growth of remote sensing data, it remains a challenge for traditional remote sensing image processing techniques to create the desired content without tedious manual operations.

[0004] Despite recent successes in generating natural images from text, generating high-resolution remote sensing images from text remains challenging. The first work to address text-to-remote sensing image generation was presented by Bejiga et al., where conditional GANs were applied to generate grayscale remote sensing images of extremely low spatial resolution from textual descriptions of ancient geographic areas. They later improved text encoding by using a pre-trained Doc2Vec encoder to capture different levels of information in the input text, however, the grayscale remote sensing images generated by these methods had very low spatial resolution and missed many details. Chen et al. further proposed a text-based deep supervised GAN to generate satellite images with a spatial size of 128×128, and used the generated images to enhance the training set for the change detection task. Zhao et al. proposed a structured GAN to synthesize remote sensing images from text with a spatial size of 256×256, emphasizing that structural information is an important factor in evaluating the fidelity of generated images. Xu et al. proposed using a modern Hopfield network to generate high-resolution remote sensing images from text. However, given that foreground elements in remote sensing images are difficult to identify and understand, the spatial distribution of different objects is complex and diverse, and text descriptions and remote sensing images are completely different modal information, the performance of the above methods is limited, and it is still very difficult to generate high-quality and realistic remote sensing images from text descriptions. Summary of the invention

[0005] In order to overcome the shortcomings of the above prior art, the purpose of the present invention is to provide a method for generating remote sensing images from text, which improves the richness and accuracy of remote sensing image feature representation by introducing a hierarchical prototype layer and a confidence-driven dynamic prototype learning strategy. The model can gradually learn and adapt to more prototypes, so that the model can show higher robustness and accuracy when processing complex data. In order to achieve the above technical purpose, the present invention adopts the following technical solution.

[0006] A method for generating remote sensing images from text, specifically comprising the following steps:

[0007] S1: Prepare the training dataset RSICD and obtain text descriptions and corresponding real remote sensing images;

[0008] S2: Train a Vector Quantization Generative Adversarial Network (VQGAN);

[0009] S3: Encode the text-image pair through the text and image encoder;

[0010] S4: splicing text and image tags;

[0011] S5: Input the joint concatenation sequence into the dynamic hierarchical prototype block to extract features;

[0012] S6: The training process adopts a dynamic prototype learning strategy;

[0013] S7: Use the trained model to generate remote sensing images.

[0014] Furthermore, the step S1 is specifically as follows: extract the text description of each image into a separate txt file, input the original json file of the RSICD dataset, and output a txt file containing all text descriptions, all training sample file names, and all test sample file names.

[0015] Furthermore, the step S2 is specifically as follows: pre-generate a codebook of discrete values ,in .exist Each encoding position goes to Find the nearest code in the discretized form and generate variables of the same dimension. Based on this, CNN Encoder is used for encoding:

[0016]

[0017] The self-supervised loss is as follows:

[0018]

[0019] Among them, the first term in the above formula To rebuild the losses, is the gradient termination operation. In addition, the adversarial loss in GAN is also added, and its loss function can be expressed as:

[0020]

[0021] In summary:

[0022]

[0023] Furthermore, the step S3 is specifically as follows: (1) First, each character in the text is regarded as an independent token, the frequency of all character pairs is counted, the most frequent characters are selected to merge, and a new token is formed. Finally, the above steps are repeated until the predetermined number of tokens is reached. Let n be the maximum length of the input sentence. When the number of words in the input text description is less than n, 0 is used as a placeholder to fill the empty token. The input text is converted into text tokens using Byte Pair Encoding (BPE), and then the text tokens are converted into vector representations using a pre-trained word embedding model. (2) The input image is converted into a series of image tokens using a pre-trained image encoder, and then the image tokens are converted into vector representations. Through the above steps, the text and image information are converted into a unified token representation.

[0024] Furthermore, the step S4 is specifically as follows: the text tags and image tags are concatenated together in the order of text tags first and image tags later to form a joint tag sequence, and then a position code is added to each tag to help the model understand the relative position relationship between the tags.

[0025] Furthermore, the step S5 is specifically as follows: the joint label sequence is input into the LayerScale and PreNorm layers in the first dynamic hierarchical prototype block, the label sequence after standardization and scaling is input into the Hopfield layer which introduces the temperature parameter, the Hopfield layer stores and retrieves the representative prototype by minimizing the energy function, and then the label sequence processed by the Hopfield layer is input into the hierarchical prototype layer.

[0026] After the labeled sequence enters the hierarchical prototype layer, it first enters the first Hopfield layer to store and retrieve information, then enters the first self-attention layer to capture the dependencies between different positions in the sequence, then enters the second Hopfield layer to further refine the feature representation, and finally enters the second self-attention layer to enhance the interaction between features. During the forward propagation process, the input data is processed through these hierarchical structures in turn, and the final output combines the feature representations of each level, as shown in the following formula:

[0027]

[0028] in, and Represent the first and second Hopfield layers respectively, and Represent the first and second self-attention layers respectively.

[0029] Multiple hierarchical prototype layers are introduced in each dynamic hierarchical prototype block. Each dynamic hierarchical prototype block contains two hierarchical prototype layers and two standard Hopfield layers and self-attention layers. The formula is as follows:

[0030]

[0031] Among them, P represents the dynamic hierarchical prototype block, LS represents LayerScale, PN represents PreNorm, and HL represents the Hopfiled layer. Represents a hierarchical prototype layer, represents the self-attention layer, Indicates the number of prototypes.

[0032] The token sequence processed by the hierarchical prototype layer is input to the LayerScale and PreNorm layers again for normalization and scaling. The normalized and scaled token sequence is then input to the self-attention layer to calculate the similarity between the query, key, and value vectors, capturing the long-range dependencies in the token sequence, and then the token sequence processed by the first dynamic hierarchical prototype block is sequentially input to the subsequent dynamic hierarchical prototype blocks.

[0033] Furthermore, the step S6 is specifically as follows: During the training process, the model will periodically (every 100 steps) generate images and calculate their confidence. The confidence of the generated image is the average of the maximum value of the model's predicted probability of the generated image. First, the model generates an image based on the input text, and then the model calculates the Logits value of the generated image, and converts the Logits value into a probability distribution through a normalized exponential function. The formula of the normalized exponential function is:

[0034]

[0035] in, Refers to the i-th logits value. Take the maximum value in the probability distribution as the confidence and calculate its average value. The confidence calculation formula is:

[0036]

[0037] in, is the sample size, is the logits value of the i-th sample.

[0038] Furthermore, the step S7 is specifically as follows: after being processed by multiple dynamic hierarchical prototype blocks, the model generates a predicted image tag These tags represent the visual features of the generated image, and the generation process can be expressed as: .in, is the generating function of the model, is a joint concatenation sequence. Then the cross entropy loss between the predicted image tag and the original tag is calculated. The cross entropy loss is used to measure the difference between the predicted tag and the true tag. The cross entropy loss is defined as:

[0039]

[0040] in, is the probability distribution of the original image label, is the probability distribution of the predicted image tags. The predicted image tags are input to the decoder network, which converts the image tag sequence into actual image pixel values ​​through the learned features, and then generates the final high-quality remote sensing image through a series of post-processing steps.

[0041] Through the above technical solution, the method for generating remote sensing images from text provided by the present invention has the following advantages and effects:

[0042] 1. The method of the present invention designs a hierarchical prototype layer that combines multiple Hopfield layers and self-attention layers to capture richer feature representations. Specifically, these hierarchical structures of the hierarchical prototype layer can capture more complex features and relationships. In particular, when multiple hierarchical prototype layers are added, the impact on the model generation ability is particularly significant, which is manifested in the significant improvement of the quality and details of the generated images.

[0043] 2. The method of the present invention designs a dynamic prototype learning strategy to overcome the drawback of the traditional method that the number of prototypes is fixed and cannot adapt to the model requirements during the training process. Our method enables the model to have the ability of adaptive learning by dynamically adjusting the number of prototypes according to the confidence and training stage during the training process. The experimental results show that the method of dynamically adjusting the number of prototypes is superior to the method of fixing the number of prototypes in multiple evaluation indicators, and the quality of the generated images is significantly improved.

[0044] 3. After the method of the present invention introduces the temperature parameter into the Hopfield layer, it is possible to more flexibly adjust the smoothness of the normalized exponential function, thereby performing the memory and retrieval process more stably. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is an overall flow chart of a method for generating remote sensing images from texts of the present invention;

[0046] Figure 2 It is an algorithm flow chart of a method for generating remote sensing images from texts of the present invention;

[0047] Figure 3 It is a different variant of the dynamic hierarchical prototype block of a text-generated remote sensing image method of the present invention;

[0048] Figure 4 The invention discloses a remote sensing image generated by a method for generating remote sensing images from text. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the following will further describe in detail the specific implementation of the present invention with reference to examples or prior art descriptions and the accompanying drawings.

[0050] The present invention provides a text-based image generation algorithm, the overall flow chart is as follows: Figure 1 , the algorithm flow is as follows Figure 2 , the specific implementation is as follows:

[0051] Step 1: Prepare the training dataset RSICD and obtain remote sensing images and corresponding detailed sentence descriptions;

[0052] The RSICD dataset was originally collected for the remote sensing image captioning task. There are 30 scene classes in the dataset, containing a total of 10,921 aerial remote sensing images of different resolutions. The size of each image is 224×224 pixels, and each image is equipped with 5 text descriptions. 8,734 text-image pairs segmented from it are used as the training set of this method, and the remaining 2,187 text-image pairs are used as the test set of this method, and all images are adjusted to a resolution of 256×256.

[0053] Step 2: Train the Vector Quantization Generative Adversarial Network (VQGAN). The model consists of four parts: CNNEncoder, CNN Decoder, Codebook, and CNN Discriminator.

[0054] In step 2, we use the Adam optimizer with an initial learning rate of 1e-3 to train the VQGAN model. In order to optimize the learning process, an "exponential" learning rate decay strategy is adopted, that is, at the end of each training cycle (epoch), the learning rate is multiplied by 0.996, and the entire training process is carried out for a total of 1000 epochs. Once the VQGAN training is completed, the weight parameters in the encoder network, decoder network, and codebook will be fixed.

[0055] Step 3, encode the text-image pair through the text and image encoder, and then concatenate the text and image tags;

[0056] Text and image embedding: Create embedding layers for text and images respectively to convert them into high-dimensional vector representations. Assume that the number of text tokens is , the number of image tags is , the embedding dimension is , the embedding matrices are:

[0057]

[0058]

[0059] Position embedding: Create position embeddings for text and images to capture the position information in the sequence. Assume that the length of the text sequence is , the image sequence length is , then the position embedding matrices are:

[0060]

[0061]

[0062] VQGAN Freeze: Freeze VQGAN (Vector Quantization Generative Adversarial Network) to prevent it from being updated during training:

[0063]

[0064] Step 4: Input the joint splicing sequence into the dynamic hierarchical prototype block to extract features;

[0065] First, a dynamic hierarchical prototype block is initialized for hierarchical prototype learning in the latent space. Assume that the number of prototypes is ( ), then the parameters of the dynamic hierarchical prototype block module are:

[0066]

[0067] The hierarchical prototype layer consists of four main parts: Hopfield layer: used to store and retrieve information; self-attention layer: used to capture the dependencies between different positions in the sequence; the second Hopfield layer: further refines the feature representation; the second self-attention layer: enhances the interaction between features. During the forward propagation process, the input data is processed through these hierarchical structures in turn, and the final output combines the feature representations of each level.

[0068] Combining these definitions, we can express the forward propagation process of the hierarchical prototype layer as follows: First, the input data , processed by the first Hopfield layer:

[0069]

[0070] At this step, Represents the number of prototypes. The Hopfield layer maps the input data to a high-dimensional space and stores it as prototypes.

[0071] Next, the output of the first Hopfield layer After the first attention layer:

[0072]

[0073] The self-attention layer The dimensional head is used to calculate the similarity between each token in the input sequence and other tokens to aggregate the contextual information of the text.

[0074] Then, the outputs of the first Hopfield layer and the first sub-attention layer are added and passed to the second Hopfield layer:

[0075]

[0076] The second Hopfield layer further processes the input data to capture more complex patterns and features.

[0077] Finally, the output of the second Hopfield layer After the second self-attention layer:

[0078]

[0079] The second self-attention layer further aggregates contextual information to enhance the model’s expressiveness. Finally, the outputs of all layers are added together to get the final output:

[0080]

[0081] Step 5: Generate images and calculate confidence during and after training;

[0082] Text processing: Ensure that the input text is within the specified length range and convert it into an embedded representation. Assume that the input text is , where Represent the position of each word in the text:

[0083]

[0084] Generation process: The model generates image tags step by step. At each step, logits are calculated based on the current text and image tags, and new tags are generated by sampling.

[0085]

[0086] The above formula means that the current sequence Through the weight matrix The logits are calculated and converted to a probability distribution through a normalized exponential function, and then the next tag is sampled and added to the sequence. This process is repeated until a complete sequence of image tags is generated.

[0087] Image decoding: Generate new data by learning the potential distribution of the data. The generated image tags are converted into the final image through the decoder. The decoding process is:

[0088]

[0089] in, Represents the decoder function of VAE. The role of the decoder is to transform the representation in the latent space Convert to final image .

[0090] Calculate probability distribution: Evaluate the confidence of the generated image by calculating the logits of the generated image and calculating the probability using the normalized exponential function.

[0091] First, by calling the compute_logits function, the text and image data are passed in, and the calculation formula of Logits is:

[0092]

[0093] Represents the input text data, which is usually embedded into a vector space after preprocessing. Represents the input image data, which can be a single image or a collection of multiple images, and is usually converted into a feature vector after preprocessing.

[0094] Logits is the original score before the activation function is applied to the output layer of the neural network. Assume that logits is , converted to a probability distribution through a normalized exponential function , then the calculation formula for the probability distribution is:

[0095]

[0096] Evaluate confidence: Confidence indicates how certain the model is that the generated image belongs to a certain category. By choosing a probability distribution The maximum value in is used to evaluate the confidence, and the formula is:

[0097]

[0098] The proposed method for generating remote sensing images from text is trained on one NVIDIA RTX A6000 and one NVIDIA RTX 4090 graphics card. The number of dynamic hierarchical prototype blocks is fixed to 10, the model is trained for 1000 epochs, and the batch size is set to 16. The results show that Figure 4 ,The remote sensing images generated by the proposed method have better quality, and the details and structure of the objects in the images are improved, and the image content is closer to the real remote sensing images.

[0099] Table 1

[0100]

[0101] Figure 3Various variants of the dynamic hierarchical prototype block are shown. As can be seen from Table 1, the hierarchical prototype layer of the present invention significantly improves the performance of the model, and can improve the accuracy and robustness of the model no matter at which layer of the basic prototype block it is introduced. In particular, after adding multiple hierarchical prototype layers, the quality and details of the generated images are significantly improved, verifying its effectiveness and applicability in the task of text generation from remote sensing images.

[0102] Table 2

[0103]

[0104] Table 2 shows the quantitative results of the impact of different dynamic prototype learning strategies on the model. We classify dynamic prototype learning strategies into four categories: 1. Dynamic prototype learning strategy-a: a prototype adjustment strategy based on confidence. 2. Dynamic prototype learning strategy-b: a prototype adjustment strategy based on training progress. 3. Dynamic prototype learning strategy-c: a prototype adjustment strategy based on fixed intervals. 4. Dynamic prototype learning strategy-d: a staged prototype adjustment strategy. The results show that the model with the dynamic prototype learning strategy outperforms the baseline model in all evaluation indicators. Specifically, the model has a lower cross entropy loss, indicating that it converges better during training, and the IS and FID scores are also greatly improved.

[0105] Aiming at the difficulty of generating high-quality and realistic remote sensing images from text descriptions, the present invention proposes a method for generating remote sensing images from text. Experiments were conducted on the RSICD dataset, and the IS index, FID index and zero-shot classification accuracy OA were significantly improved. Overall, our model performs well in generating image quality and processing unseen data, especially in zero-shot classification accuracy. In addition, our model is different from the traditional GAN-based method, but adopts the Transformer-based method. The Transformer-based method has unique advantages in processing sequence data and capturing long-distance dependencies, which may be one of the reasons why our model performs well in zero-shot classification tasks.

[0106] The above embodiments are only used to illustrate the present invention, but not to limit the present invention. A person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the present invention, and the patent protection scope of the present invention shall be defined by the claims.

[0107] The technical contents not described in detail in the present invention are all known technologies.

Claims

1. A method for generating remote sensing images from text, characterized in that: The steps include: S1, prepare the training dataset RSICD, obtain text descriptions and corresponding real remote sensing images; S2. Train a vector quantization generative adversarial network (VQGAN); S3, encoding the text-image pair by a text and image encoder; S4, splicing text and image tags; S5, inputting the joint splicing sequence into the dynamic hierarchical prototype block to extract features; S6, the training process adopts a dynamic prototype learning strategy; S7. Use the trained model to generate remote sensing images.

2. The method for generating remote sensing images from text according to claim 1, characterized in that: In step S2, a codebook of discrete values ​​is generated in advance. , ,exist Each encoding position goes to Find the code closest to it, and then use CNN Encoder to encode it: .

3. The method for generating remote sensing images from text according to claim 1, characterized in that: In the step S3, each character in the text is first regarded as an independent token, the frequencies of all character pairs are counted, the most frequent characters are selected for merging to form a new token, and the above steps are repeated until a predetermined number of tokens is reached; let n be the maximum length of the input sentence, and when the number of words in the input text description is less than n, 0 is used as a placeholder to fill the empty token, and then the pre-trained word embedding model is used to convert the text token into a vector representation; Use a pre-trained image encoder to convert the input image into image tokens, and then convert the image tokens into vector representations.

4. The method for generating remote sensing images from text according to claim 1, characterized in that: In step S4, the text mark and the image mark are concatenated together in the order of the text mark first and the image mark second to form a joint mark sequence, and then a position code is added to each mark.

5. The method for generating remote sensing images from text according to claim 1, characterized in that: In step S5, the joint tag sequence is input into the first dynamic hierarchical prototype block for standardization and scaling; the standardized and scaled tag sequence is input into the Hopfield layer that introduces the temperature parameter; and then the tag sequence processed by the Hopfield layer is input into the hierarchical prototype layer, and the formula of the hierarchical prototype layer is as follows: ,in and Represent the first and second Hopfield layers respectively, and Represent the first and second self-attention layers respectively; Each dynamic hierarchical prototype block contains two hierarchical prototype layers and two standard Hopfield layers and self-attention layers, and the formula is as follows: , where P represents the dynamic hierarchical prototype block, LS represents LayerScale, PN represents PreNorm, and HL represents the Hopfiled layer. Represents a hierarchical prototype layer, represents the self-attention layer, Indicates the number of prototypes; then the tag sequence processed by the first dynamic hierarchical prototype block is input into the subsequent dynamic hierarchical prototype blocks in sequence.

6. The method for generating remote sensing images from text according to claim 1, characterized in that: In step S6, during the training process, the model will periodically generate images and calculate their confidences; first, the model generates images based on the input text, then the model calculates the Logits value of the generated image, and then converts the Logits value into a probability distribution through a normalized exponential function. The formula of the normalized exponential function is: ,in, Refers to the i-th logits value, takes the maximum value in the probability distribution as the confidence, and calculates its average value. The confidence calculation formula is: ,in, is the sample size, is the logits value of the i-th sample; The number of prototypes is dynamically adjusted according to the confidence and training stage: if the confidence of the generated image is lower than the set confidence threshold, the number of prototypes is increased; when the number of prototypes has reached the maximum value and the training has reached a certain stage, the number of prototypes is reduced.

7. The method for generating remote sensing images from text according to claim 1, characterized in that: In step S7, after processing through multiple hierarchical prototype layers, the model generates a predicted image tag ; The generation process can be expressed as: ,in, is the generating function of the model, The concatenated sequences are then combined and the cross entropy loss between the predicted image tags and the original tags is calculated. The cross entropy loss is defined as: ,in, is the probability distribution of the original image labels, It is the probability distribution of predicted image labels; the predicted image labels are converted into image pixel values ​​through the decoder network to generate the final high-quality remote sensing image.

Images