Remote sensing scene generation method based on autoregression model and diffusion model

Abstract

According to the remote sensing scene generation method based on the autoregression model and the diffusion model provided by the invention, the reasonable layout prediction of the remote sensing target based on the autoregression model is carried out, and the reasonable position, size and orientation of the target are predicted by adopting an autoregression thought, that is, the position, size and orientation of only one target are predicted each time; and all previous prediction results become condition input of next prediction. According to the method, a complex multivariate problem (targets can influence each other) is simplified, so that the problem is easier to solve. And then a remote sensing background image is excavated based on the predicted position, size and orientation, and a controllable remote sensing target is generated based on the remote sensing background image. The method comprises the following four steps of: 1, collecting and processing a remote sensing scene image; 2, predicting the position, size and orientation of a single target; and step 3, carrying out autoregression sampling. And 4, generating and embedding a remote sensing target in the background.

Description

Technical Field

[0001] The invention relates to a remote sensing scene generation method based on an autoregressive model and a diffusion model, and belongs to the field of remote sensing. Background Art

[0002] Mainstream remote sensing image target detection, tracking and trajectory prediction models are data-driven. However, remote sensing image data in real application scenarios often presents a long-tail distribution. The training of existing deep learning models can be regarded as a maximum likelihood / posterior estimation process, which is not effective for such long-tail distribution data and lacks accuracy in critical and scarce scenarios such as wartime or post-disaster. Therefore, it is necessary to generate various remote sensing scenarios to train and test deep learning models.

[0003] Autoregressive Model (AR Model for short) is a time series analysis model, whose main feature is to use the past values ​​of the time series itself to predict future values.

[0004] Unet is a popular convolutional neural network architecture, mainly used for biomedical image segmentation tasks. It was originally proposed by Olaf Ronneberger et al. in 2015, aiming to obtain accurate segmentation results on a small amount of training data through an efficient end-to-end training method. Unet adopts a symmetrical U-shaped structure, and the name of Unet comes from its symmetrical U-shaped architecture. The network is divided into two parts: encoder (downsampling path) and decoder (upsampling path). The encoder is responsible for gradually reducing the image size and extracting features, while the decoder gradually restores the image size and performs fine pixel-level segmentation.

[0005] A major feature of Unet is the skip connection, which connects the feature map in the encoder to the feature map of the corresponding layer in the decoder. These connections help the network retain high-resolution information, thereby improving the accuracy of segmentation, especially in the processing of boundary details.

[0006] Unet has a simple and efficient architecture, and its unique design makes it perform well in small dataset tasks that require high-precision segmentation, making it the preferred model in many image segmentation applications. It also has important applications in diffusion models.

[0007] In the field of image restoration, a method called LaMa solves key problems such as large missing areas, complex geometric structures, and high-resolution images. LaMa's method contains the following three main innovations:

[0008] 1. Repair the network architecture and use Fast Fourier Convolutions (FFCs) to achieve image-wide receptive fields, which is crucial for capturing the global structure of the image.

[0009] 2. High receptive field perceptual loss, based on the perceptual loss of the semantic segmentation network, improves the consistency of the global structure and shape in the restoration results.

[0010] 3. Large training masks, adopt an aggressive mask generation strategy to fully utilize the high receptive field of the network and loss function.

[0011] The LaMa model performs well on various datasets, especially in handling periodic structures, and generalizes well to high-resolution images even when only low-resolution data is used during training. In addition, the LaMa model has significant advantages in terms of parameter and time cost compared to competing baselines.

[0012] The K-means clustering algorithm is a classic unsupervised learning method for partitioning a data set into K different clusters. The basic steps of the algorithm include: randomly selecting K initial centroids, assigning each data point to the cluster with the nearest centroid, calculating the new centroid of each cluster, and repeating the above process until the centroids are stable and no longer change or a predetermined number of iterations is reached. The algorithm is widely used due to its simplicity and high computational efficiency, but its effectiveness depends on the selection of initial centroids and requires the number of clusters K to be determined in advance.

[0013] Diffusion Model is the current mainstream image generation model. The most classic and basic model is Denoising Diffusion Probabilistic Models (DDPM). The general idea is to assume that the sample is x0, add noise to x0, and after multiple steps of adding noise, the sample will become pure noise. The inference process is to add pure noise to the network and turn it into the required image. Assuming a certain x t , which is formed by adding t times of noise to the original image, then x t The noise prediction network is added at the same time as t. The noise prediction network will predict the noise that should be subtracted in this step, x t After subtracting the noise, we get x t-1 For a clearer picture. Repeated denoising will eventually produce a clear image.

[0014] During training, the real x t-1 As the ground truth.

[0015] The DDPM-based Text-to-Image model Stable Diffusion also adopts roughly the same idea. However, they do not directly generate the final image, but generate a latent space representation (Imagen is a small 64*64 image) and restore it to an image through a decoder. The idea of ​​doing this is similar to that of the variational autoencoder (VAE), and the decoder can be trained separately. At the same time, Stable Diffusion uses a text encoder to encode the language into latent variables and feed the latent variables into the generator at each step of the generation.

[0016] Consider the DDPM forward process. Given a real image x0~q(x), the diffusion model forward process adds Gaussian noise to it through T times of accumulation to obtain x1, x2, ..., x T Here we need to give a series of Gaussian distribution variance hyperparameters {β t ∈0,1t=1T. Since each moment t in the forward process is only related to the moment t-1, it can also be regarded as a Markov process:

[0017]

[0018] In this process, as t increases, x t is getting closer and closer to pure noise. When T→∞, x T is a completely Gaussian noise. And in practice, β t As it increases, it increases, that is, β1<β2<…<β T In the GLIDE code, β t It is a linear interpolation from 0.0001 to 0.02 (with T = 1000 as the benchmark, T increases, β t corresponding reduction).

[0019] Since each Gaussian noise sampling is independent, we can directly write x t The relationship between α and x0. t =1-β t but;

[0020]

[0021] According to the Bayesian formula,

[0022]

[0023] in,

[0024] Because it is known but;

[0025]

[0026] The noise in this formula is what we want to predict through the neural network, ∈ = ∈ θ (x t , t),

[0027] During the training process, the loss function used continues the idea of ​​the variational autoencoder (VAE) loss function. The DDPM algorithm for training and sampling inference is as follows:

[0028]

[0029] A common image restoration technique based on diffusion model is to add noise to the missing part of the image and restore it through step-by-step denoising.

[0030] Guided Diffusion refers to the process of using the diffusion model to generate data, by adding a classifier as a guide, without changing the training process, only changing the sampling process to control the generation condition and improve the generation effect. θ，φ (x t |x t+1 , y) for sampling, where θ and φ are neural network parameters. By simple Bayesian formula derivation, we can get p θ，φ (x t |x t+1 , y) = Zp θ (x t |x t+1 ) φ (y|x t ), where Z is a constant. By training DDPM, p θ (x t |x t+1 ) is what we already know, Through Taylor expansion, we can further estimate logp φ (y|x t ), expanding around the predicted mean, we have

[0031]

[0032] Here C1 is a constant. In summary, we have

[0033]

[0034] We thus find that the guiding effect of the classifier is reflected in the addition of ∑g to the mean of each step during sampling. Became The pseudo code is as follows:

[0035]

[0036] Classifier-Free Diffusion Guidance essentially adds condition y to the noise prediction process, that is, ∈ θ (x t , y, t). By ∈ θ (x t , y, t) can directly derive μ(x t , y, t), let u t '=μ(x t , y, t)

[0037] Here μ t 'Equivalent to μ+∑g in the classifier guide

[0038] Considering the scaling factor s, we have μ+s∑g=s(μ+∑g)-μ(s-1). Let ω=s-1 and we have μ+s∑g=-μω+(ω+1)(μ+∑g). For the non-classifier guidance, it is equivalent to (ω+1)μ′. t -ωμ(x t ,φ,t).φ represents unconditional input. Summary of the invention

[0039] The purpose of the present invention is to provide a remote sensing scene generation method based on an autoregressive model and a diffusion model. The method first predicts the reasonable layout of remote sensing targets based on the autoregressive model, that is, according to a remote sensing background image input, predicts the reasonable position, size and orientation of a specified number of remote sensing targets in the remote sensing background image. Then, the corresponding part of the background image is extracted according to the obtained layout information, and on this basis, the controllable remote sensing target generation based on the diffusion model is realized. The target generation adopts the idea of ​​combining virtual and real generation, that is, the virtual target is superimposed on the real background, and the generated remote sensing target has good authenticity, controllability and diversity. Specifically, the authenticity of the remote sensing target is reflected in the fact that the light and shadow and edge of the generated target are consistent with the original remote sensing background; the controllability is reflected in the ability to specify the target orientation; the diversity is reflected in the ability to generate various types of remote sensing targets. The present invention can automatically generate a large amount of training and test data required for remote sensing target detection, tracking and trajectory prediction models, greatly alleviates the current problem of insufficient data, and improves the generalization performance and robustness of remote sensing target detection, tracking and trajectory prediction models.

[0040] The present invention is achieved through the following technical solutions:

[0041] The present invention is a remote sensing scene generation method based on an autoregressive model and a diffusion model. It predicts the reasonable layout of remote sensing targets based on the autoregressive model, and adopts the idea of ​​autoregression to predict the reasonable position, size and orientation of the target, that is, only predicts the position, size and orientation of one target each time, and all previous prediction results will become the conditional input for the next prediction. This method simplifies a complex multivariate problem (targets will affect each other), making it easier to solve the problem. Then, based on the predicted position, size and orientation, a remote sensing background image is extracted, and controllable remote sensing target generation is performed on this basis. The method is divided into four steps: Step 1: remote sensing scene image collection and processing; Step 2: predicting the position, size and orientation of a single target; Step 3: autoregressive sampling. Step 4: generation and embedding of remote sensing targets in the background; the specific steps are as follows:

[0042] Step 1: Remote sensing scene image collection and processing

[0043] Regulation:

[0044] 1. All coordinates used in the image are in pixel coordinate system.

[0045] 2. The coordinates of a pixel are usually x or (x, y) T .

[0046] 3. The location where the target appears refers to its center coordinates.

[0047] 4.size refers to the target size. In order to align with the method of generating targets, it means twice the length of the largest side of the smallest horizontal box that encloses the target.

[0048] 5. θ refers to the direction of the target, which is in degrees. Its definition is consistent with the angle defined in the general sense of the coordinate system (i.e., the horizontal rightward direction is the x-axis, and the horizontal upward direction is the y-axis), in order to align with the method of generating the target.

[0049] 6. The location, angle and size data of the target are called metadata.

[0050] Obtain remote sensing scene images from the public remote sensing image target detection dataset, use the LaMa model to remove targets in the image, and record the location, size, and orientation of these targets in a csv file. Perform data enhancement on the image and cluster the size of the target.

[0051] Step 2: Predict the location, size, and orientation of a single target

[0052] In order to predict the position, size and orientation of a single target, the problem can be modeled as: predict the probability of a target of size size and orientation θ appearing at each pixel on an image. Specifically, consider the coordinates x = (x, y) T , predict the probability p(x) of the target appearing in the pixel, and under the condition of the target appearing, predict the distribution of direction θ and size size p(θ, size|x)=p(θ|x)p(size|x) (it is reasonable to assume that the two conditions are independent), thus obtaining the joint probability distribution p(θ, size, x). The obtained distribution is called the layout graph L. From the layout graph L, take Take again The prediction result (x * ,θ * size * ).

[0053] Step 3: Autoregressive Sampling

[0054] Specifying that n targets should be embedded in a remote sensing background image, first predict the first target layout L1 and get For the i-th (2≤i≤n) target, the prediction of its layout is based on the prediction results of the previous i-1 targets. Further prediction results

[0055] The values ​​of θ and size are continuous. In order to simplify the problem, they are discretized and the prediction of θ and size is treated as a classification problem, each discretized into 12 categories.

[0056] Step 4: Generate and embed the remote sensing target in the background

[0057] After sampling the position, size and orientation of the target, the corresponding image block is extracted from the background image according to the position and size of the target, and the center of this part is replaced with random noise. The target is integrated into the background image through the diffusion model. In order to control the generated angle of the target, a conditional guidance graph is connected to the channel dimension of the input, and the angle information is embedded on the basis of the time step embedding. The classifier-free diffusion guidance (CFG) is used to better control the angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is the image before LaMa removes the target.

[0059] Figure 2 It is the image after LaMa removes the target.

[0060] Figure 3 is the enhanced input image.

[0061] Figure 4 It is condition_map.

[0062] Figure 5 is the probability graph.

[0063] Figure 6 It is an occurrence probability graph (3D graph).

[0064] Figure 7 It is a schematic diagram of target embedding remote sensing background training.

[0065] Figure 8 It is a schematic diagram of target embedded remote sensing background sampling.

[0066] Fig. 9 It is a conditional control chart.

[0067] Fig.10 It is the target generation effect diagram.

[0068] Fig.11 It is the rendering of the layout diagram.

[0069] Fig.12 It is the overall generated effect diagram. DETAILED DESCRIPTION

[0070] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0071] Step 1: Collect and process remote sensing target data sets;

[0072] 1. Obtain remote sensing scene images from the public remote sensing image target detection dataset, use the LaMa model to remove targets in the image, and record the position, size and orientation of these targets in a csv file. Figure 1 and Figure 2 shown.

[0073] 2. The specific method of obtaining the target orientation The first step is to train a classifier that can distinguish the four orientations of the target: "up, down, left, and right". These four orientations correspond to 90°, 270°, 180°, and 360° respectively. Classifier training:

[0074] 1.1 Use the oriented bounding box in the remote sensing target dataset to rotate the target to the horizontal.

[0075] 1.2 Preliminary screening: the longest side of the oriented bounding box of the retained targets should be greater than 50, and the aspect ratio should be between 0.8-1.2.

[0076] 1.3 Manually select the targets facing right and resize them to 224*224.

[0077] 1.4 Rotate the right-facing target to the four directions of “up, down, left, and right”.

[0078] 1.5 Based on this training dataset, fine-tune a backbone network, which is a VGG pre-trained network on ImageNet with four categories of direction discrimination networks.

[0079] 3. The specific method of obtaining the target orientation In the second step, find the target according to the label in the dataset and use the OpenCV library to rotate it to the horizontal. The rotation angle is β r The rotated target is cropped and resized to 224*224, and the angle classified by the trained classifier is β c , then the angle of the final target is β=β c -β r .

[0080] 4. For the remote sensing background image, cut out the largest square image from the upper left corner and adjust its size to 256*256. Correspondingly, the content in the csv file should also be adjusted.

[0081] 5. Perform random data augmentation on the input image, considering horizontal and vertical flipping, counterclockwise rotation of 90°, 180°, 270°, and gamma transformation in the range of [0.5, 1.5]. During the flipping and rotation process, the angle and position data matching the image should also be transformed accordingly.

[0082] 6. The size value is obtained by performing K-means clustering with 12 categories on the sizes of all targets recorded in the csv file, and the 12 value categories of θ are obtained from the interval of [30, 360] with an interval of 30.

[0083] Step 2: Predict the location, size and orientation of a single target;

[0084] Consider the training process to predict the location, size, and orientation of a single target.

[0085] 1. For each remote sensing background image, specify the maximum number of targets max_num that can be considered during training. This is done to facilitate parallel computing.

[0086] 2. For each image input of a training batch, determine the original number of all targets as n, consider valid_num = min(n, max_num), and randomly select the metadata input of valid_num targets.

[0087] 3. Sample t∈{0,...,valid_num-1}. This process is similar to DDPM.

[0088] 4. Randomly select t targets as known targets and predict the layout graph with the remaining valid_num-t targets.

[0089] 5. The prediction network is Unet, and the output layer uses sigmoid activation function.

[0090] 6. How does the metadata of the first t targets affect the layout map prediction of the t+1th target as a condition?

[0091] 6.1Unet input is x of shape 4×H×W t ,x t The original input data x is composed of 3×H×W data The condition map condition_map of 1×H×W is obtained by connecting it in the channel dimension. For the condition map condition_map, when t=0, that is, when there is no prediction result as a condition, all its elements are 0. Otherwise, assuming that the i-th target position is (x i ,y i ) T , size is size i The coordinates in the conditional graph are (x, y) T The value (x, y) of the pixel point is:

[0092]

[0093] The visualization results are as follows Figure 3 and Figure 4 As shown:

[0094] 6.2 linearly maps the value of t from [0, max_num-1] to [0, 4000], and then embeds it into the Unet prediction process in the same way as DDPM.

[0095] 7. Target value setting of layout diagram:

[0096] 7.1 Consider the background image of the remaining valid_num-t targets, with coordinates x = (x, y) T For pixels, the probability of the target appearing is Among them, p i (x) represents the influence of each real target position on it. i =(x i ,y i ) T , define p i The distribution form of (x) is a Gaussian distribution with it as the expectation. However, in order to avoid the problem of large targets with small probabilities, the coefficients in front of the Gaussian distribution are removed, and finally In order to make the area with high probability density of target as close to the original target center as possible, take σ i，1 =σ i，2 =size i / 3, the visualization result is as follows Figure 5 and Figure 6 As shown:

[0097] 7.2 Based on the above i (x) can obtain the conditional probability distribution of angle and direction. Let the size and angle of the i-th target be θ i , size i , for coordinates x = (x, y) T For pixels, the target values ​​of angle and size are θ(x), size(x)

[0098]

[0099] Discretize it and assume that the standard categories of angle and size are T = {θ s，i , 1≤i≤12}, S={size s，i , 1≤i≤12}, then its discretization target value is:

[0100]

[0101] The category number is cls(θ g (x)) and cls(size g (x)), then the target distribution is:

[0102]

[0103] 8. The loss function is set to cross entropy loss or mean square loss. Set the batch size Batch_size to B and the image size to H×H. Let the i-th predicted layout image be I i , where the coordinate x i The predicted occurrence probability, target size and angle values ​​are p ω (x i ), p ω (size|x i ), p ω (θ|x i ), the losses are L1, L2, and L3 respectively.

[0104] If mean square loss is used:

[0105]

[0106] If cross entropy loss is used,

[0107]

[0108] Assuming the weights of the three losses are w1, w2, and w3, the final loss is

[0109] The training algorithm is as follows:

[0110]

[0111] Step 3: Autoregressive sampling;

[0112] 1. Specify the number of targets n that need to be predicted.

[0113] 2. For t = 1, ..., n, construct the condition_map in the same way as mentioned in the training, except that the target data used here are all based on the predictions in the previous step. Input the background image and condition_map into the network to obtain the layout map L t .

[0114] 3. Consider the coordinate x in condition_map t The pixel value is value(x t ), let the predicted position probability be p(x t ), take p′(x t )=(1-value(x t ))p(x t ),Pick Take again You can get the prediction results The reason why an operation equivalent to the Hadimer product of the predicted position probability map and 1-condition_map is performed here is to prevent overlap between the predicted targets.

[0115] 4. Note that, whether training or sampling, the input original image x data Random noise is added to enhance robustness, and most importantly, to prevent the neural network from learning LaMa to remove the high-frequency fine traces left by small targets. Here, the normal noise is σ = 0.02.

[0116] The sampling algorithm is as follows:

[0117]

[0118] Step 4: Generate and embed the remote sensing target in the background.

[0119] The first step is training data processing: obtain target images that meet certain conditions from the public remote sensing image target detection dataset, rotate them to various specified orientations, crop and resize the images to 128*128.

[0120] The specific steps are as follows:

[0121] 1. The classifier uses the previously trained classifier.

[0122] 2. Rotate to the specified angle.

[0123] 2.1 There are 12 types of controllable angles: 0°~330°, with one type set every 30°.

[0124] 2.2 Initial screening, same as above.

[0125] 2.3 Rotate the target to a horizontal position and determine its direction. Determine the direction of the original target based on the determined direction. Once the original target direction is known, it can be rotated to a specified angle.

[0126] 2.4 Get the target after rotating to the specified angle, frame it with the smallest horizontal frame, expand it to 2 times with the center of the horizontal frame as the center, crop it, and then perform a second screening to keep the target image with a longer side greater than 110 and a pure black area ratio less than 0.05, and then resize it to 128*128.

[0127] 3. Naming of training data: the scene number to which the target belongs_target category_the number of the category_target orientation.

[0128] The training process, such as Figure 7 shown.

[0129] Based on the palette model, a mask is first generated, which is a square in the center of the input image with a side length of 3 / 5 of the side length of the original training image. Assuming that the shape of image x0 is 3×H×W, the shape of the mask is H×W, the value of the pixel at the damaged position that needs to be repaired is 1, and the value of the pixel at the other positions is 0. The mask is used to represent the part that needs to be repaired.

[0130] When training, follow these steps:

[0131] 1. Sample x0 from training data

[0132] 2. Replace the mask part in x0 with standard Gaussian noise ∈ 0 to obtain conditional x cond , that is, x cond =mask⊙∈0+(1-mask)⊙x0 (where ⊙ represents the Hadimer product, element-by-element product, taking into account the broadcast mechanism)

[0133] 3. Perform the same noise adding process as DDPM, but only add noise to the mask part

[0134] 4. Generate condition y

[0135] 5. With probability p, the condition is randomly set to φ to train a model without conditions.

[0136] 6. x t , x cond , y,t is sent to the network to predict the noise, and only the mask part is taken as the mean square loss.

[0137] The algorithm is as follows:

[0138]

[0139] Condition y uses a simple stripe diagram with a shape of 1×H×W. The width of the stripe is d, the length is L, and the value in the axial direction (length direction) satisfies the uniform distribution from [-1,1]. The place with a value of 1 is the head, and the place with a value of -1 is the tail. The direction of the stripe is the direction of the target to be controlled θ, and the pixels outside the stripe are 0. The visualization effect is as follows: Figure 8 shown.

[0140] For unconditional φ, directly take a zero tensor of shape 1×H×W.

[0141] Concatenate the first dimension of the tensor with the channel dimension of the input background image and input it into the U-net as a condition.

[0142] In addition, the angle information is mapped to [1, T], and then the same operation as the time step embedding is performed, and then added to the time step embedding.

[0143] Setting different ω values ​​during sampling can achieve different conditional guidance intensities.

[0144] Sampling process, such as Fig. 9 shown.

[0145] 1. Get the background image x of the target to be generated T (The background tile corresponding to the previously predicted position and size);

[0146] 2. Replace x with standard Gaussian noise ∈ 0 T The mask part gets the condition x cond , and this condition will also be used as a new input x T ;

[0147] 3. Obtain condition y (obtained according to the predicted angle);

[0148] 4. Based on x t , x cond ,y,t predicts the noise, and the guiding strength is determined by ω;

[0149] 5. By noise ∈ tand x t Sample x t-1 ;

[0150] 6. We only use x t-1 The mask part of t ;

[0151] The algorithm is as follows:

[0152]

[0153] Fig.10 It is the target generation effect diagram. Fig.11 It is the rendering of the layout diagram. Fig.12 It is the overall generated effect diagram.

Claims

1. A remote sensing scene generation method based on autoregressive model and diffusion model, characterized in that: The steps include: Step 1: Collect and process remote sensing scene images; Obtain remote sensing scene images from a public remote sensing image target detection dataset, use the LaMa model to remove targets in the image, and record the location, size, and orientation of these targets in a csv file; perform data enhancement on the image and cluster the size of the targets; Step 2: Predict the location, size and orientation of a single target; In order to predict the position, size and orientation of a single target, the model is: predict the probability of a target of size size and orientation θ appearing at each pixel on an image; specifically, consider the coordinates x = (x, y) T , predict the probability p(x) of the target appearing. Under the condition of the target appearing, predict the distribution p(θ, size|x)=p(θ|x)p(size|x) of the direction θ and size size, and get the joint probability distribution p(θ, size, x). The obtained distribution is called the layout graph L. From the layout graph L, take Take again Get the prediction result (x * ,θ * , size * ); Step 3: Autoregressive sampling; Specifying that n targets should be embedded in a remote sensing background image, first predict the first target layout L1 and get For the i-th target, 2≤i≤n, the prediction of its layout is based on the prediction results of the previous i-1 targets Further prediction results Step 4: Generate and embed the remote sensing target in the background; After sampling the position, size and orientation of the target, the corresponding image block is extracted from the background image according to the position and size of the target, and the center of this part is replaced by random noise. The target is integrated into the background image through the diffusion model. In order to control the generation angle of the target, a conditional guidance graph is connected to the channel dimension of the input, and the angle information is embedded on the basis of the time step embedding. The diffusion guided CFG without classifier is used to control the angle.

2. The remote sensing scene generation method based on the autoregressive model and the diffusion model according to claim 1, characterized in that: All coordinates used in images are in pixel coordinate system; the coordinates of a pixel are in x or (x, y) T ; The location where the target appears refers to its center coordinates; size refers to the target size. To align with the target generation method, it means twice the length of the smallest horizontal frame that encloses the target. θ refers to the orientation of the target, in degrees, and its definition is consistent with the angle defined in the coordinates, in order to align with the method of generating targets.

3. A remote sensing scene generation method based on an autoregressive model and a diffusion model according to claim 1 or 2, characterized in that: The values ​​of θ and size are continuous. In order to simplify the problem, they are discretized and the prediction of θ and size is treated as a classification problem, each discretized into 12 categories.

4. The remote sensing scene generation method based on the autoregressive model and the diffusion model according to claim 1, characterized in that: In step 2, the specific steps are: 2.1 For each remote sensing background image, specify the maximum number of targets max_num that can be considered during training; 2.2 For each image input of a training batch, determine the number of all original objects as n, consider valid_num = min(n, max_num), and randomly select the metadata input of valid_num objects; the metadata includes the position, angle and size data of the object; 2.3 Sampling t∈{0,...,valid_num-1}; 2.4 Randomly select t targets as known targets and predict the layout of the remaining valid_num-t targets; 2.5 The prediction network is Unet, and the output layer uses the sigmoid activation function; 2.6 The metadata of the first t targets is used as a condition to influence the layout map prediction of the t+1th target: 2.7 Target value setting of layout diagram: 2.8 The loss function is set to cross entropy loss or mean square loss; set the batch size Batch_size to B, the image size to H×H; let the i-th predicted layout image be I i , where the coordinate x i The predicted occurrence probability, target size and angle values ​​are p ω (x i ), p ω (size|x i ), p ω (θ|x i ), the losses are L1, L2, and L3 respectively.

5. The remote sensing scene generation method based on the autoregressive model and the diffusion model according to claim 4 is characterized in that: In step 2.6, the input of Unet is x of shape 4×H×W t ,x t The original input data x is composed of 3×H×W data Connect the 1×H×W condition map condition_map in the channel dimension to obtain; for the condition map condition_map, when t=0, that is, when there is no prediction result as a condition, all its elements are 0, otherwise, assuming that the i-th target position is (x i ,y i ) T , size is size i The coordinates in the conditional graph are (x, y) T The value (x, y) of the pixel point is: The value of t is linearly mapped from [0, max_num-1] to [0, 4000] and then embedded into the Unet prediction process in the same way as DDPM.

6. The remote sensing scene generation method based on the autoregressive model and the diffusion model according to claim 4, characterized in that: In step 2.7, consider the background image of the remaining valid_num-t targets, with coordinates x = (x, y) T The probability of the target appearing is Among them, p i (x) represents the influence of each real target position on it; for each target position x i =(x i ,y i ) T , define p i The distribution form of (x) is the expected Gaussian distribution, and the coefficients in front of the Gaussian distribution are removed, and finally Take σ i，1 =σ i，2 =size i / 3; By p i (x) obtains the conditional probability distribution of angle and orientation; let the size and angle of the i-th target be θ i , size i , for coordinates x = (x, y) T The target values ​​of the angle and size of the pixel are θ(x), size(x) Discretize it and assume that the standard categories of angle and size are T = {θ s，i , 1≤i≤12}, S={size s，i , 1≤i≤12}, then the discretization target value is: The category number is cls(θ g (x)) and cls(size g (x)), then the target distribution is:

7. The remote sensing scene generation method based on the autoregressive model and the diffusion model according to claim 4, characterized in that: If mean square loss is used:

8. The remote sensing scene generation method based on the autoregressive model and the diffusion model according to claim 4, characterized in that: If cross entropy loss is used, 9. The remote sensing scene generation method based on the autoregressive model and the diffusion model according to claim 1, characterized in that: In step three, the specific steps are: 3.

1. Specify the number of targets n that need to be predicted; 3.2.for t=1,...,n, construct condition_map as mentioned in the training, input the background image and condition_map into the network to obtain the layout map L t ; 3.

3. Consider the coordinate x in condition_map t The pixel value is value(x t ), let the predicted position probability be p(x t ), take p′(x t )=(1-value(x t ))p(x t ),Pick Take again Get the prediction results 3.

4. Whether training or sampling, the input original image x data Random noise is added to enhance robustness and prevent the neural network from learning LaMa to remove high-frequency fine traces left by small targets; the normal noise is taken as σ = 0.

02.

10. The remote sensing scene generation method based on the autoregressive model and the diffusion model according to claim 1, characterized in that: In step 4, the specific steps are: 4.

1. Classifier Use the trained classifier; 4.

2. Rotate to the specified angle; There are 12 controllable angles: 0°~330°, and a category is set every 30° and the first screening is performed; rotate the target to the horizontal and determine the direction, determine the direction of the original target according to the determined direction, and rotate it to the specified angle when the original target direction is known; obtain the target after being rotated to the specified angle, frame it with the smallest horizontal frame, expand it to 2 times with the center of the horizontal frame as the center, crop it, and then perform the second screening, leaving the target image with a long side greater than 110 and a pure black area ratio less than 0.05, and then resize it to 128*128; 4.

3. Naming of training data: the scene number to which the target belongs_target category_the number of the category_target orientation.

Images