A remote sensing image extrapolation method for camouflage assessment

Abstract

The present invention provides a remote sensing image extrapolation method for camouflage assessment, belonging to the field of remote sensing image generation. The present invention first establishes a two-stage image extrapolation framework; secondly, it optimizes text encoding and updates the text encoder parameters; then, it constructs a remote sensing image extrapolation dataset for camouflage assessment, forming "original remote sensing image-canvas-cropped remote sensing image-prompt text" data pairs, and introduces human preferences into the dataset; then, it conducts two-stage image extrapolation training; finally, based on application requirements, it selects an original remote sensing image to be extrapolated, sets an extrapolation canvas, and writes a natural language description, which serves as a prompt for image extrapolation to generate an extrapolated remote sensing image. The present invention can automatically generate a background around the camouflaged object based on the prompt word, thereby quickly obtaining a camouflage assessment result.

Description

Technical Field

[0001] The invention belongs to the field of remote sensing image generation, and in particular relates to a remote sensing image extrapolation method for camouflage assessment. Background Art

[0002] Camouflaged images are widely used in the fields of adversarial recognition and privacy protection. Camouflage evaluation is an effective means to evaluate the camouflage effect. The main purpose of camouflage measures is to eliminate the differences between the camouflaged object and the background and improve the similarity between the two in dimensions such as texture, structure, brightness, and color. However, the mobility of the camouflaged object and the time-varying nature of the background require the camouflage measures to be adjusted on the fly. The adjusted camouflage measures need to be promptly evaluated.

[0003] Currently, the common method for camouflage assessment is to select an 8-neighborhood or 4-neighborhood area around the camouflaged object from the background and perform similarity calculations. This process of "camouflage deployment - target background selection - camouflage effectiveness evaluation" is constrained by multiple steps and is inefficient. Therefore, a remote sensing image extrapolation method for camouflage assessment is adopted. This method automatically generates a background around the camouflaged object based on a prompt word. The similarity between the generated background and the background of the planned target is then directly calculated, quickly obtaining camouflage assessment results and improving the speed and flexibility of camouflage assessment. Summary of the Invention

[0004] In view of this, the present invention proposes a remote sensing image extrapolation method for camouflage evaluation, which can improve the efficiency of the traditional camouflage evaluation process and evaluate the camouflage effect under different backgrounds without moving the camouflaged target.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A remote sensing image extrapolation method for camouflage assessment includes the following steps:

[0007] Step 1: Build a two-stage image extrapolation framework. The framework contains four models with trainable parameters: image encoder, denoising network, image decoder, and text encoder. The input of the extrapolation framework includes remote sensing images, canvas, prompt text, and random noise. The remote sensing images in the training stage are cropped from the original remote sensing images, and in the prediction stage are the remote sensing images to be extrapolated. The size of the canvas in the training stage is consistent with the original remote sensing image, and the size in the prediction stage is manually set. The content of the prompt text is a natural language description of the newly generated content of the extrapolated image. Random noise is used to increase the randomness and diversity of the generated image.

[0008] Step 2: After cropping the original remote sensing image, a cropped remote sensing image is obtained. The cropping amount is set to 0, that is, the original remote sensing image is consistent with the cropped remote sensing image, and the canvas size is consistent with the size of the cropped remote sensing image. The trainable parameters in the image encoder, denoising network, and image decoder are frozen, and the loss function is set to the image similarity between the original remote sensing image and the generated image. Through iterative optimization, the model parameters in the text encoder are updated to achieve text encoding optimization;

[0009] Step 3: Use optical remote sensing satellites and UAV platforms to collect data on the scene or target where remote sensing image extrapolation is required to obtain the original remote sensing image. Then, crop the image to obtain the cropped remote sensing image. Set all pixel values ​​of the original remote sensing image to 0 as the canvas. Use text to describe the original remote sensing image and the cropped content in natural language to form a "original remote sensing image-canvas-cropped remote sensing image-prompt text" data pair. Repeat the above steps to construct 2000 sets of data pairs to obtain the training dataset.

[0010] Step 4: Use the image encoder and image decoder to encode and decode the original remote sensing images collected in step 3. For the decoded images, manually select images that meet human preferences, and generate canvases, crop, and describe them in natural language according to the dataset construction process in step 3 to form data pairs. The data pairs are added to the training dataset to obtain a new dataset.

[0011] Step 5: extract training data from the data set obtained in step 4, input the cropped remote sensing image and canvas into the image encoder to obtain image features, input the prompt text into the text encoder to obtain text features, input the image features, text features, and random noise into the denoising network to obtain denoised image features, input the denoised image features into the image decoder, and generate an image after decoding. Freeze the trainable parameters of the image encoder, image decoder, and text encoder, and update the model parameters of the denoising network by optimizing the similarity between the generated image and the original remote sensing image.

[0012] Step 6: Select a remote sensing image that needs to be extrapolated, set the first canvas according to its size, and the horizontal and vertical dimensions of the first canvas are 80 pixels larger than the remote sensing image. Write a natural language description as the prompt text, randomly generate random noise, and input the remote sensing image, the first canvas, the random noise, and the prompt text into the image extrapolation network to generate the first-stage generated image. Then, use the first-stage generated image as the remote sensing image that needs to be extrapolated, set the second canvas, and extend the canvas size by 80 pixels. Input the first-stage generated image, the second canvas, the random noise, and the prompt text into the extrapolation network to generate the second-stage generated image, that is, the final generated image.

[0013] Furthermore, the image encoder and image decoder in step 1 respectively adopt the encoder part and decoder part in the VAE network, the text encoder adopts the BERT model, and the denoising network adopts UNet.

[0014] Furthermore, the natural language description in step 1 includes the target, background, relative positions between targets, attributes of the target, and attributes of the background that need to be newly added.

[0015] Furthermore, the specific encoding and decoding method in step 4 is: set the canvas size to be the same as the remote sensing image size, input the remote sensing image and the canvas into the image encoder, obtain the image features, skip the denoising network, directly input the image features into the image decoder, and obtain the generated image through decoding.

[0016] The beneficial effects of the present invention compared to the prior art are:

[0017] (1) The present invention can simplify the process of disguise evaluation and improve the efficiency of disguise evaluation.

[0018] (2) The present invention can evaluate the camouflage effect under different backgrounds without moving the camouflage target.

[0019] (3) The present invention can generate any realistic background based on the extrapolated canvas and prompt words. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the principle of an embodiment of the present invention.

[0021] Figure 2 Schematic diagram of the structure of the extrapolation network in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The specific implementation manner and basic principle of the present invention will be further described below with reference to the accompanying drawings.

[0023] A remote sensing image extrapolation method for camouflage assessment. The principle of this method is as follows Figure 1 The specific steps are as follows:

[0024] Step 1: Build a two-stage image extrapolation framework: Figure 2The framework contains four models with trainable parameters, namely image encoder, denoising network, image encoder, and text encoder. The image encoder and image decoder use the encoder part and decoder part of the VAE network respectively. The text encoder uses the BERT model, and the denoising network uses UNet. The input of the extrapolation framework includes remote sensing images, canvases, prompt texts, and random noise. The remote sensing images are the remote sensing images that have been cropped from the original remote sensing images during the training phase, and the remote sensing images that need to be extrapolated during the prediction phase. The size of the canvas is consistent with the original remote sensing image during the training phase, and the size is set manually during the prediction phase. The content of the prompt text is a natural language description of the newly generated content of the extrapolated image, including the newly added targets, backgrounds, the relative positions between targets, the attributes of the targets, and the attributes of the background. Random noise as the input of the denoising network can increase the randomness and diversity of the generated images.

[0025] Step 2, text encoding optimization: The original remote sensing image is cropped to obtain a cropped remote sensing image, and the cropping amount is set to 0, that is, the original remote sensing image is consistent with the cropped remote sensing image, and the canvas size is consistent with the size of the cropped remote sensing image. The trainable parameters in the image encoder, denoising network, and image decoder are frozen, and the loss function is set to the image similarity between the original remote sensing image and the generated image. Through iterative optimization, the model parameters in the text encoder are updated.

[0026] Step 3: Construct a remote sensing image extrapolation dataset for camouflage assessment: Use optical remote sensing satellites and UAV platforms to collect data on the scene or target for remote sensing image extrapolation to obtain the original remote sensing image. Then, crop the image to obtain the cropped remote sensing image. Set all pixel values ​​of the original remote sensing image to 0 as the canvas. Use text to describe the original remote sensing image and the cropped content in natural language, forming a "original remote sensing image-canvas-cropped remote sensing image-prompt text" data pair. Repeat the above steps to construct 2,000 data pairs.

[0027] Step 4: Human preferences are introduced into the dataset: Since the results produced by image extrapolation are non-deterministic, that is, the multiple outputs obtained by the model based on a certain input can all be considered reasonable and correct, in order to constrain the output results, human preferences need to be introduced. The specific process is: set the canvas size to the same as the remote sensing image size, input the remote sensing image and canvas into the image encoder, obtain image features, skip the denoising network, and directly input the image features into the image decoder, and obtain the generated image through decoding. Since the encoder and decoder parts of the VAE network are used, the encoding and decoding process requires distribution generation and sampling processes. Sampling from the distribution is random, that is, after decoding, a variety of images with different content and style can be generated. By manually selecting images that meet human preferences, and following the dataset construction process in step 3 for canvas generation, cropping, and natural language description, data pairs are formed and added to the training dataset.

[0028] Step 5, two-stage image extrapolation training: crop the original remote sensing image to obtain a cropped remote sensing image, set the cropping amount to 80 pixels, and set a canvas with the same size as the original remote sensing image. Input the cropped remote sensing image and the canvas into the image encoder to obtain image features. Input the prompt text into the text encoder to obtain text features. Input the image features, text features, and random noise into the denoising network to obtain denoised image features. Input the denoised image features into the image decoder. After decoding, the generated image is output. Freeze the trainable parameters of the image encoder, image decoder, and text encoder. Update the model parameters of the denoising network by optimizing the similarity between the generated image and the original remote sensing image.

[0029] Step 6, image extrapolation: Select a remote sensing image that needs to be extrapolated according to application requirements, set the first canvas according to its size, and the horizontal and vertical dimensions of the first canvas are 80 pixels larger than the remote sensing image. Write a natural language description as the prompt text, randomly generate random noise, and input the remote sensing image, the first canvas, the random noise, and the prompt text into the image extrapolation network to generate the first-stage generated image. Then, use the first-stage generated image as the remote sensing image that needs to be extrapolated, set the second canvas, and extend the canvas size by 80 pixels. Input the first-stage generated image, the second canvas, the random noise, and the prompt text into the extrapolation network to generate the second-stage generated image, that is, the final generated image.

[0030] The present invention can automatically generate the background around the camouflage object according to the prompt word, and then quickly obtain the camouflage evaluation result.

Claims

1. A remote sensing image extrapolation method for camouflage assessment, characterized in that: The following steps are involved: Step 1: Build a two-stage image extrapolation framework. The framework contains four models with trainable parameters: image encoder, denoising network, image decoder, and text encoder. The input of the extrapolation framework includes remote sensing images, canvas, prompt text, and random noise. The remote sensing images in the training stage are cropped from the original remote sensing images, and in the prediction stage are the remote sensing images to be extrapolated. The size of the canvas in the training stage is consistent with the original remote sensing image, and the size in the prediction stage is manually set. The content of the prompt text is a natural language description of the newly generated content of the extrapolated image. Random noise is used to increase the randomness and diversity of the generated image. Step 2: After cropping the original remote sensing image, a cropped remote sensing image is obtained. The cropping amount is set to 0, that is, the original remote sensing image is consistent with the cropped remote sensing image, and the canvas size is consistent with the size of the cropped remote sensing image. The trainable parameters in the image encoder, denoising network, and image decoder are frozen, and the loss function is set to the image similarity between the original remote sensing image and the generated image. Through iterative optimization, the model parameters in the text encoder are updated to achieve text encoding optimization; Step 3: Use optical remote sensing satellites and UAV platforms to collect data on the scene or target where remote sensing image extrapolation is required to obtain the original remote sensing image. Then, crop the image to obtain the cropped remote sensing image. Set all pixel values ​​of the original remote sensing image to 0 as the canvas. Use text to describe the original remote sensing image and the cropped content in natural language to form the "original remote sensing image-canvas-cropped remote sensing image-prompt text" data pair. Repeat the above steps to construct 2000 sets of data pairs to obtain the training dataset. Step 4: Use the image encoder and image decoder to encode and decode the original remote sensing images collected in step 3. For the decoded images, manually select images that meet human preferences, and generate canvases, crop, and describe them in natural language according to the dataset construction process in step 3 to form data pairs. The data pairs are added to the training dataset to obtain a new dataset. Step 5: extract training data from the data set obtained in step 4, input the cropped remote sensing image and canvas into the image encoder to obtain image features, input the prompt text into the text encoder to obtain text features, input the image features, text features, and random noise into the denoising network to obtain denoised image features, input the denoised image features into the image decoder, and generate an image after decoding. Freeze the trainable parameters of the image encoder, image decoder, and text encoder, and update the model parameters of the denoising network by optimizing the similarity between the generated image and the original remote sensing image. Step 6: Select a remote sensing image that needs to be extrapolated, set the first canvas according to its size, and the horizontal and vertical dimensions of the first canvas are 80 pixels larger than the remote sensing image. Write a natural language description as the prompt text, randomly generate random noise, and input the remote sensing image, the first canvas, the random noise, and the prompt text into the image extrapolation network to generate the first-stage generated image. Then, use the first-stage generated image as the remote sensing image that needs to be extrapolated, set the second canvas, and extend the canvas size by 80 pixels. Input the first-stage generated image, the second canvas, the random noise, and the prompt text into the extrapolation network to generate the second-stage generated image, that is, the final generated image.

2. The remote sensing image extrapolation method for camouflage assessment according to claim 1, characterized in that: The image encoder and image decoder in step 1 use the encoder part and decoder part of the VAE network respectively, the text encoder uses the BERT model, and the denoising network uses UNet.

3. The remote sensing image extrapolation method for camouflage assessment according to claim 1, characterized in that: The natural language description in step 1 includes the newly added target, background, relative positions between targets, attributes of the target, and attributes of the background.

4. The remote sensing image extrapolation method for camouflage assessment according to claim 1, characterized in that: The specific encoding and decoding method in step 4 is: set the canvas size to the same as the remote sensing image size, input the remote sensing image and canvas into the image encoder to obtain image features, skip the denoising network, directly input the image features into the image decoder, and obtain the generated image through decoding.

Images