A remote sensing camouflage image editing method based on text prompts

Abstract

The application provides a remote sensing camouflage image editing method based on a text prompt and belongs to the field of remote sensing image generation. First, a three-mode CLIP model is constructed and trained. Second, a distribution generation network is built. Then, the distribution generation network is trained. Next, the decoder is trained. Finally, according to application requirements, an original remote sensing image that needs to be edited is selected, and a natural language description is written, which is used as a prompt for image editing to generate a remote sensing camouflage image. The application can still regenerate objects and regions in the image without image masks.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image generation, and specifically refers to a method for editing remote sensing camouflage images based on text prompts. Background Technology

[0002] Currently, researchers have developed various extended models based on improvements to image generation models, such as in-image rendering models. These models can define the image region to be generated based on a mask and generate content only within the mask based on prompts. However, such in-image rendering models require an additional mask as input, which undoubtedly reduces the interactivity between the model and the user. To address this, a text-based remote sensing camouflage image editing method can regenerate specified objects and regions in an image using only text prompts, thereby improving the efficiency and flexibility of image editing. Furthermore, compared to editing natural or cartoon images, remote sensing image editing demands higher image detail, is more sensitive to the relationships and constraints between pixels, and has a lower tolerance for noise and erroneous pixels. These characteristics significantly increase the difficulty of remote sensing image generation, necessitating targeted model improvements. Summary of the Invention

[0003] In view of this, the present invention proposes a remote sensing camouflage image editing method based on text prompts. This method can reduce the number of input modalities for image editing, and can regenerate objects and regions in the image using only text prompts, while improving the performance of the image editing model in remote sensing images.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] A method for editing remote sensing camouflage images based on text prompts includes the following steps:

[0006] Step 1: Construct a three-mode CLIP model, which includes two image encoders and one text encoder. The image encoders convert spoofed images into image embedding vectors, and the text encoders convert text into text embedding vectors. The image encoders all use ViT networks, and the text encoders use GPT2 models. A dataset grouping training strategy is adopted, and the three-mode CLIP model is trained by minimizing the loss function through an optimization function.

[0007] Step 2: Construct a distributed generation network. The input to the distributed generation network is random noise, image embedding vector, text embedding vector, and sampling time t. After adding the random noise and the image embedding vector, the network passes through a multilayer perceptron to obtain an image transformation vector. The image transformation vector then undergoes a random masking operation to obtain an image mask vector. The text embedding vector passes through a multilayer perceptron to obtain a text transformation vector. The text transformation vector then undergoes a random masking operation to obtain a text mask vector. The sampling time t passes through a multilayer perceptron to obtain a time transformation vector. The image mask vector, text mask vector, and time transformation vector are then input into the Transformer module of the causal relationship, and the mean and variance are output.

[0008] Step 3: Train the distributed generation network. Input the first camouflaged image, the second camouflaged image, and the text description into the three-mode CLIP model to obtain the first image embedding vector, the second image embedding vector, and the text embedding vector. Randomly generate noise and set the sampling time t. Input the random noise, the second image embedding vector, the text embedding vector, and the sampling time t into the distributed generation network to obtain the variance and mean. Obtain a normal distribution based on the variance and mean. Sample the target image embedding vector in the obtained distribution through the sampling function. Calculate the L2 loss between the target image embedding vector and the first image embedding vector as the loss function. Minimize the loss function through the optimization algorithm to update the parameters of the multilayer perceptron and the Transformer module parameters of the causal relationship in the distributed generation network. After training converges, the trained distributed generation network is obtained.

[0009] Step 4: Train the decoder. Construct a decoder using a convolutional neural network. The decoder is a 5-layer fully convolutional network. Extract all training images from the dataset used in Step 1 to obtain multiple camouflaged images, which form the decoder training dataset. Input the data from the decoder training dataset into the image encoder of the triple CLIP model. Embed the encoded image into a vector and input it into the decoder to obtain the decoded image. Calculate the similarity between the decoded image and the input image, which is used as the first part of the loss function. Classify the decoded image using an image classification network, and use the classification confidence as the second part of the loss function. Add the first and second parts of the loss function to form the decoder training loss function. Use an optimization algorithm to minimize the loss function to train the parameters of the decoder. Iterate through the training to obtain the trained decoder.

[0010] Step 5: The camouflaged image and text description are processed through the three-mode CLIP model to obtain image embedding vectors and text embedding vectors. The image embedding vectors and text embedding vectors are then input into the trained distributed generation network. After distributed generation and sampling, the target image embedding vector is obtained. This vector is then input into the trained decoder to obtain the edited remote sensing camouflaged image.

[0011] Further, the dataset grouping training strategy in step 1 is as follows: The training dataset contains 1200 training data points. Each training data point contains three types of data: a first camouflaged image, a text description, and a second camouflaged image. The difference between the first and second camouflaged images is recorded using the text description. During dataset grouping training, the 1200 data points are divided into 300 groups, with each group containing 4 data points. The first camouflaged image from the 4 data points is input into the image encoder to obtain the first image embedding vector, and the 4 first image embedding vectors are concatenated to form the first loss vector. The second camouflaged image from the 4 data points is input into the image encoder to obtain the second image embedding vector, and the 4 second image embedding vectors are concatenated to form the second loss vector. The text description from the 4 data points is input into the text encoder to obtain the text embedding vector, and the 4 text embedding vectors are concatenated to form the third loss vector. The similarity between the i-th position in the first loss vector, the j-th position in the second loss vector, and the k-th position in the third loss vector is calculated and recorded as L. ijk L represents a tensor of size 4×4×4, ijk is the index, and the loss function is set as the mean square error between L and the three-dimensional unit tensor.

[0012] Furthermore, there is a reciprocal relationship between the first camouflaged image and the second camouflaged image in step 3. At least one of the three elements—background type, target type, and camouflage method—must be consistent, and inconsistent factors are converted into each other through natural language text description.

[0013] Furthermore, the classification confidence in step 4 is the classification confidence of any category, that is, the classification confidence value is larger when there is no category ambiguity in the image than the classification confidence value when there is category ambiguity.

[0014] The advantages of this invention over the prior art are as follows:

[0015] (1) The present invention can reduce the number of input modalities for image editing and can still regenerate objects and regions in an image without the need for an image mask.

[0016] (2) The present invention can perform stable, controllable and realistic editing of remote sensing images.

[0017] (3) The present invention can edit pixels in an image by relying solely on text prompts. Attached Figure Description

[0018] Figure 1 This is an architecture diagram of the three-mode CLIP model proposed in this invention.

[0019] Figure 2 This is a schematic diagram of the dataset grouping training strategy proposed in this invention.

[0020] Figure 3This is a structural diagram of the distributed generation network proposed in this invention.

[0021] Figure 4 This is a schematic diagram of the training process of the distributed generative network proposed in this invention. Detailed Implementation

[0022] The specific embodiments and basic principles of the present invention will be further explained below with reference to the accompanying drawings.

[0023] A method for editing remote sensing camouflage images based on text prompts includes the following steps:

[0024] Step 1, Construction and training of the three-mode CLIP model: (e.g.) Figure 1 As shown, the three-mode CLIP model includes two image encoders and one text encoder. The image encoders convert spoofed images into image embedding vectors, and the text encoder converts text into text embedding vectors. Both image encoders use the ViT network, and the text encoder uses the GPT2 model. The three-mode CLIP model is trained using a dataset grouping strategy, such as... Figure 2 As shown, the training dataset contains 1200 training data points. Each training data point contains three types of data: camouflaged image 1, text description, and camouflaged image 2. The difference between camouflaged image 1 and camouflaged image 2 is recorded using the text description. During dataset grouping training, the 1200 data points are divided into 300 groups, with each group containing 4 data points. The camouflaged image 1 data point is input into the image encoder to obtain image embedding vector 1, and the four image embedding vectors 1 are concatenated to form loss vector 1. The camouflaged image 2 data point is input into the image encoder to obtain image embedding vector 2, and the four image embedding vectors 2 are concatenated to form loss vector 2. The text description data point is input into the text encoder to obtain text embedding vectors, and the four text embedding vectors are concatenated to form loss vector 3. The similarity between the i-th position in loss vector 1, the j-th position in loss vector 2, and the k-th position in loss vector 3 is calculated and recorded as L. ijk Let L represent a tensor of size 4×4×4, ijk be the index, and the loss function be set as the mean squared error between L and the three-dimensional unit tensor. The three-mode CLIP model is trained by minimizing the loss function through an optimization function.

[0025] Step 2, Distributed generation network setup: such as Figure 3As shown, the inputs of the distributed generation network are random noise, image embedding vector, text embedding vector, and sampling time t. After the random noise and image embedding vector are added, the image transformation vector is obtained through a multilayer perceptron. The image transformation vector is then subjected to a random masking operation to obtain an image mask vector. The text embedding vector is then subjected to a multilayer perceptron to obtain a text transformation vector. The text transformation vector is then subjected to a random masking operation to obtain a text mask vector. The sampling time t is then subjected to a multilayer perceptron to obtain a time transformation vector. The image mask vector, text mask vector, and time transformation vector are input into the Causal Transformer module, which outputs the mean and variance.

[0026] Step 3, Distributed Generative Network Training: (e.g.) Figure 4 As shown, camouflaged image 1, camouflaged image 2, and text description are input into the three-mode CLIP model. There is a reciprocal relationship between camouflaged image 1 and camouflaged image 2. At least one of the three elements of background type, target type, and camouflage method is consistent. Inconsistent factors can be converted to each other through simple natural language text descriptions to obtain image embedding vector 1, image embedding vector 2, and text embedding vector. Random noise is generated, and a sampling time t is set. Random noise, image embedding vector 2, text embedding vector, and sampling time t are input into the distributed generation network to obtain variance and mean. A normal distribution is obtained based on variance and mean. The target image embedding vector is obtained by sampling in the obtained distribution through a sampling function. The L2 loss between the target image embedding vector and image embedding vector 1 is calculated as the loss function. The multilayer perceptron parameters and CausalTransformer parameters in the distributed generation network are updated by minimizing the loss function through an optimization algorithm. After training convergence, the trained distributed generation network is obtained.

[0027] Step 4, Decoder Training: Construct a decoder using a convolutional neural network. The decoder is a 5-layer fully convolutional network. Extract all training images from the dataset used in Step 1 to obtain 2400 camouflaged images, forming the decoder training dataset. Input the data from the decoder training dataset into the image encoder of the triple CLIP model. Embed the encoded image into a vector and input it into the decoder to obtain the decoded image. Calculate the similarity between the decoded image and the input image as the first part of the loss function. Classify the decoded image using an image classification network, and use the classification confidence as the second part of the loss function. The classification confidence is the classification confidence of any category, that is, the classification confidence value is larger when the image has no category ambiguity than the classification confidence value when there is category ambiguity. Add the first and second parts of the loss function to form the decoder training loss function. Use an optimization algorithm to minimize the loss function to train the parameters of the decoder. Iterate through the training to obtain the trained decoder.

[0028] Step 5, Remote sensing camouflage image editing: The camouflage image and text description are processed through a three-mode CLIP model to obtain image embedding vectors and text embedding vectors. The image embedding vectors and text embedding vectors are then input into a trained distributed generation network. After distributed generation and sampling, the target image embedding vector is obtained and input into a trained decoder to obtain the edited remote sensing camouflage image.

[0029] This invention can regenerate objects and regions in an image without requiring an image mask.

Claims

1. A method for editing remote sensing camouflage images based on text prompts, characterized in that, Includes the following steps: Step 1: Construct a three-mode CLIP model, which includes two image encoders and one text encoder. The image encoders convert spoofed images into image embedding vectors, and the text encoders convert text into text embedding vectors. The image encoders all use ViT networks, and the text encoders use GPT2 models. A dataset grouping training strategy is adopted, and the three-mode CLIP model is trained by minimizing the loss function through an optimization function. Step 2: Construct a distributed generation network. The input to the distributed generation network is random noise, image embedding vector, text embedding vector, and sampling time t. After adding the random noise and the image embedding vector, the network passes through a multilayer perceptron to obtain an image transformation vector. The image transformation vector then undergoes a random masking operation to obtain an image mask vector. The text embedding vector passes through a multilayer perceptron to obtain a text transformation vector. The text transformation vector then undergoes a random masking operation to obtain a text mask vector. The sampling time t passes through a multilayer perceptron to obtain a time transformation vector. The image mask vector, text mask vector, and time transformation vector are then input into the Transformer module of the causal relationship, and the mean and variance are output. Step 3: Train the distributed generation network. Input the first camouflaged image, the second camouflaged image, and the text description into the three-mode CLIP model to obtain the first image embedding vector, the second image embedding vector, and the text embedding vector. Randomly generate noise and set the sampling time t. Input the random noise, the second image embedding vector, the text embedding vector, and the sampling time t into the distributed generation network to obtain the variance and mean. Obtain a normal distribution based on the variance and mean. Sample the target image embedding vector in the obtained distribution through the sampling function. Calculate the L2 loss between the target image embedding vector and the first image embedding vector as the loss function. Minimize the loss function through the optimization algorithm to update the parameters of the multilayer perceptron and the Transformer module parameters of the causal relationship in the distributed generation network. After training converges, the trained distributed generation network is obtained. Step 4: Train the decoder. Construct a decoder using a convolutional neural network. The decoder is a 5-layer fully convolutional network. Extract all training images from the dataset used in Step 1 to obtain multiple camouflaged images, which form the decoder training dataset. Input the data from the decoder training dataset into the image encoder of the triple CLIP model. Embed the encoded image into a vector and input it into the decoder to obtain the decoded image. Calculate the similarity between the decoded image and the input image, which is used as the first part of the loss function. Classify the decoded image using an image classification network, and use the classification confidence as the second part of the loss function. Add the first and second parts of the loss function to form the decoder training loss function. Use an optimization algorithm to minimize the loss function to train the parameters of the decoder. Iterate through the training to obtain the trained decoder. Step 5: The camouflaged image and text description are processed through the three-mode CLIP model to obtain image embedding vectors and text embedding vectors. The image embedding vectors and text embedding vectors are then input into the trained distributed generation network. After distributed generation and sampling, the target image embedding vector is obtained. This vector is then input into the trained decoder to obtain the edited remote sensing camouflaged image.

2. The remote sensing camouflage image editing method based on text prompts according to claim 1, characterized in that, The dataset grouping training strategy in Step 1 is as follows: The training dataset contains 1200 training data points. Each training data point contains three types of data: a first camouflaged image, a text description, and a second camouflaged image. The difference between the first and second camouflaged images is recorded using the text description. During dataset grouping training, the 1200 data points are divided into 300 groups, with each group containing 4 data points. The first camouflaged image from the 4 data points is input into the image encoder to obtain the first image embedding vector, and the 4 first image embedding vectors are concatenated to form the first loss vector. The second camouflaged image from the 4 data points is input into the image encoder to obtain the second image embedding vector, and the 4 second image embedding vectors are concatenated to form the second loss vector. The text description from the 4 data points is input into the text encoder to obtain the text embedding vector, and the 4 text embedding vectors are concatenated to form the third loss vector. The similarity between the i-th position in the first loss vector, the j-th position in the second loss vector, and the k-th position in the third loss vector is calculated and recorded as L. ijk L represents a tensor of size 4×4×4, ijk is the index, and the loss function is set as the mean square error between L and the three-dimensional unit tensor.

3. The remote sensing camouflage image editing method based on text prompts according to claim 1, characterized in that, There is a reciprocal relationship between the first and second camouflaged images in step 3. At least one of the three elements—background type, target type, and camouflage method—must be consistent. Inconsistent factors are converted to each other through natural language text descriptions.

4. The remote sensing camouflage image editing method based on text prompts according to claim 1, characterized in that, The classification confidence in step 4 is the classification confidence for any category, that is, the classification confidence value is greater when there is no category ambiguity in the image than when there is category ambiguity.

Images