A remote sensing scene generation method based on an autoregressive model and a diffusion model

Abstract

The application provides a remote sensing scene generation method based on an autoregressive model and a diffusion model, reasonable remote sensing target layout prediction based on the autoregressive model, autoregressive ideas for predicting reasonable positions, sizes and orientations of targets, that is, only the positions, sizes and orientations of one target are predicted each time, and all previous prediction results become conditional inputs for next prediction. This mode simplifies a complex multivariate problem (targets will affect each other) and makes the problem easier to solve. Then, a remote sensing background image is excavated based on the predicted positions, sizes and orientations, and controllable remote sensing target generation is carried out on the basis. The method comprises four steps: step one, remote sensing scene image collection and processing; step two, prediction of positions, sizes and orientations of single targets; step three, autoregressive sampling; and step four, generation and embedding of remote sensing targets in the background.

Description

TECHNICAL FIELD

[0001] The application relates to a remote sensing scene generation method based on an autoregressive model and a diffusion model, and belongs to the field of remote sensing. BACKGROUND

[0002] Mainstream remote sensing image target detection, tracking and trajectory prediction models are based on data driving, however, remote sensing image data in real application scenarios often presents a long-tail distribution, and the training of an existing deep learning model can be regarded as a process of maximum likelihood / posterior estimation, and the effect of such long-tail distribution data is poor, and in the face of such key scarce scenarios as wartime or post-disaster, the precision is insufficient. Therefore, various remote sensing scenes need to be generated to train and test the deep learning model.

[0003] An autoregressive model (AR model for short) is a time series analysis model, and its main feature is to use the past values of a time series itself to predict future values.

[0004] Unet is a popular convolutional neural network architecture, mainly used for biomedical image segmentation tasks. It was first proposed by Olaf Ronneberger et al. in 2015, aiming to obtain accurate segmentation results on a small amount of training data through an efficient end-to-end training method. Unet adopts a symmetrical U-shaped structure, and the name Unet comes from its symmetrical U-shaped architecture. The network is divided into an encoder (downsampling path) and a decoder (upsampling path). The encoder is responsible for gradually reducing the image size and extracting features, while the decoder gradually restores the image size and performs fine pixel-level segmentation.

[0005] One of the main features of Unet is skip connections. Unet connects the feature maps in the encoder with the corresponding layers in the decoder through skip connections. These connections help the network retain high-resolution information, thereby improving the accuracy of segmentation, especially in handling boundary details.

[0006] The architecture of Unet is simple and efficient, and its unique design makes it perform well in small data set tasks that require high-precision segmentation, so it has become the preferred model in many image segmentation applications. It also has important applications in diffusion models.

[0007] In the field of image inpainting, a method called LaMa solves key problems such as large-area missing regions, complex geometric structures and high-resolution images. The LaMa method contains the following three main innovations:

[0008] 1. The repair network architecture uses Fast Fourier Convolution (FFCs) to achieve a receptive field across the image, which is crucial for capturing the global structure of the image.

[0009] 2. High-receptive-field perceptual loss, based on the perceptual loss of a semantic segmentation network, improves the consistency of global structure and shape in the repair results.

[0010] 3. Large training mask, using an aggressive mask generation strategy, fully utilizes the high receptive field of the network and loss function.

[0011] The LaMa model performs well on various datasets, especially in handling periodic structures, and can also generalize well to high-resolution images even when only low-resolution data is used during training. In addition, compared with competitive benchmarks, the LaMa model has significant advantages in terms of parameter and time cost.

[0012] K-means clustering algorithm is a classic unsupervised learning method, which is used to divide a dataset into K different clusters. The basic steps of the algorithm include: randomly selecting K initial centroids, assigning each data point to the cluster with the nearest centroid, calculating the new centroid of each cluster, and repeating the above process until the centroid is stable or the predetermined number of iterations is reached. The algorithm is widely used due to its simplicity and high computational efficiency, but its effectiveness depends on the selection of initial centroids and the need to pre-determine the number of clusters K.

[0013] Diffusion Model is the current mainstream image generation model, among which the most classic and basic model is Denoising Diffusion Probabilistic Models (DDPM). The general idea is to assume that the sample is x0, add noise to x0, and after multiple steps of noise addition, the sample will become pure noise. The inference process is to add pure noise to the network to become the required image. Assuming that a certain x t is formed by adding t times of noise to the original image, then add noise to x t at the same time, the noise prediction network will predict the noise that should be subtracted at this step, x t After subtracting the noise, x t-1 is a clearer image. Repeat the denoising process to get a clear image.

[0014] During training, the real x t-1 is used as the ground truth.

[0015] The DDPM-based Text-to-Image model Stable Diffusion also takes roughly this approach. Instead of directly generating the final image, it generates a latent space representation (Imagen is a 64*64 small image) and restores it to an image through a decoder. The idea is similar to Variational Autoencoder (VAE), and the training of the decoder can be separate. At the same time, Stable Diffusion uses a text encoder to encode language into latent variables and feed them into the generator at each step of generation.

[0016] Consider the DDPM forward process. Given a real picture x0~q(x), the diffusion model forward process adds Gaussian noise to it by T times accumulation to get x1, x2,..., x T Here we need to give a series of Gaussian distribution variance hyperparameters {β t ∈0,1t=1T. The forward process can also be regarded as a Markov process because each time t is only related to t-1:

[0017]

[0018] In this process, as t increases, x t is closer and closer to pure noise. When T→∞, x T is completely Gaussian noise. And in practice, β t is increasing, that is, β1<β2<…<β T In the code of GLIDE, β t is linearly interpolated from 0.0001 to 0.02 (based on T=1000, T increases, β t corresponding to decrease).

[0019] Since the sampling of Gaussian noise at each time is independent, we can directly write the relationship between x t and x0. Let α t =1-β t Then;

[0020]

[0021] From Bayes' theorem we have;

[0022]

[0023] where,

[0024] Since we know then;

[0025]

[0026] The noise in this equation is what we want to predict through the neural network, ∈ = ∈ θ (x t , t),

[0027] In the process of training, the loss function adopted continues the idea of the variational autoencoder (VAE) loss function, and the loss function The DDPM algorithm obtained by training and sampling inference is as follows:

[0028]

[0029] A common image inpainting technology based on diffusion model is to add noise to the missing part of the image and repair it by step-by-step denoising.

[0030] Guided diffusion model refers to adding a classifier as a guide in the process of generating using the diffusion model, which can conditionally control the generation and improve the generation effect without changing the training process, only changing the sampling process. That is, the guided diffusion model uses p θ，φ (x t |x t+1 , y) for sampling, where θ and φ are neural network parameters. Through simple Bayesian formula derivation, we can get p θ，φ (x t |x t+1 , y) = Zp θ (x t |x t+1 )p φ (y|x t ), where Z is a constant. From the training of DDPM, p θ (x t |x t+1 ) is known, Through Taylor expansion, we can further estimate logp φ (y|x t ), which is expanded around the predicted mean, and we have

[0031]

[0032] Here C1 is a constant, and we have

[0033]

[0034] We find that the guidance effect of the classifier is reflected in that the mean of each step is added with ∑g when sampling. The pseudo code is as follows:

[0035]

[0036] Classifier-free diffusion guidance is essentially to add the condition y into the noise prediction process, that is, ∈ θ (x t , y, t). From ∈ θ (x t , y, t), μ(x t , y, t) can be directly obtained, and let u t '= μ(x t , y, t)

[0037] Here μ t ' is equivalent to μ+∑g in the classifier guidance

[0038] Considering the scaling factor s, μ+s∑g=s(μ+∑g)-μ(s-1), let ω=s-1, μ+s∑g=-μω+(ω+1)(μ+∑g), for classifier-free guidance, it is equivalent to (ω+1)μ' t -ωμ(x t , φ, t). φ represents the unconditional input. SUMMARY

[0039] The purpose of the present application is to provide a remote sensing scene generation method based on an autoregressive model and a diffusion model. The method first predicts the reasonable layout of remote sensing targets based on an autoregressive model, that is, according to an input of a remote sensing background image, the reasonable position, size and orientation of a specified number of remote sensing targets in the remote sensing background image are predicted. Then, according to the obtained layout information, the corresponding part of the background image is extracted, and controllable remote sensing target generation based on a diffusion model is realized. The target generation adopts the idea of virtual-real combination generation, that is, the virtual target is superimposed on the real background, and the generated remote sensing target has good reality, controllability and diversity. Specifically, the reality of the remote sensing target is reflected in that the light and edge of the generated target are consistent with the original remote sensing background; the controllability is reflected in that the target orientation can be specified; and the diversity is reflected in that various types of remote sensing targets can be generated. The present application can automatically generate a large amount of training and test data required for remote sensing target detection, tracking and trajectory prediction models, greatly alleviating the current data shortage problem, and improving the generalization performance and robustness of remote sensing target detection, tracking and trajectory prediction models.

[0040] The present application is realized by the following technical solutions:​

[0041] The present application is a remote sensing scene generation method based on an autoregressive model and a diffusion model. The autoregressive model is used to predict the reasonable layout of remote sensing targets. The autoregressive method is used to predict the reasonable position, size and orientation of the target. That is, only the position, size and orientation of one target are predicted each time, and all previous prediction results will become the conditional input of the next prediction. This method simplifies a complex multi-element problem (targets will affect each other) and makes the problem easier to solve. Then, based on the predicted position, size and orientation, the remote sensing background image is extracted, and controllable remote sensing target generation is performed on this basis. The method includes four steps: Step 1: remote sensing scene image collection and processing; Step 2: predicting the position, size and orientation of a single target; Step 3: autoregressive sampling; Step 4: generation and embedding of remote sensing targets in the background; the specific steps are as follows:

[0042] Step 1: remote sensing scene image collection and processing

[0043] Provisions:

[0044] 1. All coordinates used in the image are in the pixel coordinate system.

[0045] 2. The coordinates of a certain pixel are generally represented by x or (x, y) T .

[0046] 3. The position of the target refers to the center coordinates.

[0047] 4. size refers to the size of the target. In order to align with the method of generating the target, its meaning is twice the length of the larger side of the horizontal frame that encloses the target.

[0048] 5. θ refers to the orientation of the target. The angle is used, which is consistent with the definition of the angle in the general sense (i.e., the horizontal right is the x-axis, and the horizontal up is the y-axis). In order to align with the method of generating the target.

[0049] 6. The position, angle and size data of the target are referred to as metadata.

[0050] Obtain remote sensing scene images from public remote sensing image target detection data sets, use the LaMa model to remove targets in the image, and record the positions, sizes and orientations of these targets in the csv file. Perform data augmentation on the image and cluster the sizes of the targets.

[0051] Step 2: Predict the position, size and orientation of a single target

[0052] In order to predict the location, size and orientation of a single target, the problem can be modeled as: predicting the probability of a target of size size and orientation θ appearing at each pixel in an image. Specifically, consider the coordinates x = (x, y) T , predict the probability p(x) of the target appearing in the pixel. Under the condition of the target appearing, predict the distribution of direction θ and size size p(θ, size|x) = p(θ|x)p(size|x) (it is reasonable to assume that the two conditions are independent). From this, we can get the joint probability distribution p(θ, size, x). The obtained distribution is called the layout graph L. From the layout graph L, take Take again The prediction result (x * ,θ * size * ).

[0053] Step 3: Autoregressive Sampling

[0054] Specify that n targets should be embedded in a remote sensing background image. First, predict the first target layout L1 and get For the i-th (2≤i≤n) target, the prediction of its layout is based on the prediction results of the previous i-1 targets Further prediction results

[0055] The values ​​of θ and size are continuous. In order to simplify the problem, they are discretized and the prediction of θ and size is treated as a classification problem, each discretized into 12 categories.

[0056] Step 4: Generate and embed the remote sensing target in the background

[0057] After sampling the target's position, size, and orientation, a corresponding image patch is extracted from the background image based on the target's position and size. The center of this patch is replaced with random noise, and a diffusion model is used to blend the target into the background image. To control the generated angle of the target, a conditional guidance graph is attached to the input channel dimension, and angle information is embedded in the time step embedding. A classifier-free diffusion guidance (CFG) is used to better control the angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is the image before LaMa removes the target.

[0059] Figure 2 It is the image after LaMa removes the target.

[0060] Figure 3 is the enhanced input image.

[0061] Figure 4 is condition_map.

[0062] Figure 5 is appearance probability map.

[0063] Figure 6 is appearance probability map (3D map).

[0064] Figure 7 is target embedding remote sensing background training schematic diagram.

[0065] Figure 8 is target embedding remote sensing background sampling schematic diagram.

[0066] Figure 9 is condition control map.

[0067] Figure 10 is target generation effect diagram.

[0068] Figure 11 is layout diagram generation effect diagram.

[0069] Figure 12 is overall generation effect diagram. DETAILED DESCRIPTION

[0070] The present application will be further described below in conjunction with the drawings and specific embodiments.

[0071] Step one: remote sensing target data set collection and processing;

[0072] 1. Obtain remote sensing scene images from public remote sensing image target detection data sets, use LaMa model to remove targets in the images, record the positions, sizes and orientations of these targets in csv files. As shown in Figure 1 and Figure 2 .

[0073] 2. The specific way to obtain the target orientation is the first step, train a classifier that can distinguish the four orientations of the target "up, down, left, right". The four orientations correspond to 90°, 270°, 180°, 360° respectively. The training of the classifier:

[0074] 1.1 Rotate the target to horizontal using the directional bounding box in the remote sensing target data set.

[0075] 1.2 Preliminary screening, the remaining targets should have a longest side of the directional bounding box greater than 50 and an aspect ratio between 0.8-1.2.

[0076] 1.3 Manually screen the targets oriented to the right and adjust their size to 224*224.

[0077] 1.4 Rotate the right-oriented target to "up, down, left, right" four directions.

[0078] 1.5 Fine-tune a backbone network to VGG pre-trained on ImageNet, with a four-class orientation discrimination network.

[0079] 3. The specific way to get the target orientation is the second step, find the target according to the label in the dataset and rotate it to horizontal using OpenCV library, record the rotation angle as β r Cut out the rotated target and adjust the size to 224*224, and the angle classified by the trained classifier is β c , then the final target angle is β = β c -β r .

[0080] 4. For the remote sensing background image, crop the largest square image from the top left corner, and adjust its size to 256*256. Correspondingly, the content in the csv file also needs to be adjusted.

[0081] 5. Random data augmentation is done on the input image, considering horizontal and vertical flipping, counterclockwise rotation by 90°, 180°, 270°, and γ transformation with a value range of [0.5, 1.5]. During flipping and rotation, the angle and position data matching the image also need to be transformed accordingly.

[0082] 6. The size value is obtained by K-means clustering of all target sizes in the csv file with a class number of 12, while the 12 value categories of θ are obtained from the interval [30, 360] with an equal interval of 30.

[0083] Step 2: Predict the position, size, and orientation of a single target.

[0084] Consider the training process to predict the position, size, and orientation of a single target.

[0085] 1. For each remote sensing background image, specify the maximum number of targets allowed during training max_num. This is done to facilitate parallel computing.

[0086] 2. For each image input in a training batch, determine the original number of targets n, consider valid_num = min(n, max_num), and randomly select valid_num target metadata inputs.

[0087] 3. Sample t ∈ {0,..., valid_num-1}, this process is similar to DDPM.

[0088] 4. Randomly select t targets as known targets and predict the layout graph with the remaining valid_num-t targets.

[0089] 5. The prediction network is Unet, and the output layer uses sigmoid activation function.

[0090] 6. How does the metadata of the first t targets affect the layout prediction of the t+1th target as a condition?

[0091] 6.1Unet’s input is x of shape 4×H×W t ,x t The original input data x is composed of 3×H×W data The condition map condition_map of 1×H×W is obtained by connecting it in the channel dimension. For the condition map condition_map, when t=0, that is, when there is no prediction result as a condition, all its elements are 0. Otherwise, assuming that the i-th target position is (x i ,y i ) T , size is size i The coordinates in the conditional graph are (x, y) T The value (x, y) of the pixel point is:

[0092]

[0093] The visualization results are as follows Figure 3 and Figure 4 As shown:

[0094] 6.2 linearly maps the value of t from [0, max_num-1] to [0, 4000], and then embeds it into the Unet prediction process in the same way as DDPM.

[0095] 7. Target value setting of layout diagram:

[0096] 7.1 Consider the background image of the remaining valid_num-t targets, with coordinates x = (x, y) T For the pixel, let the probability of the target appearing be Among them, p i (x) represents the influence of each real target position on it. i =(x i ,y i ) T , define p i The distribution form of (x) is the Gaussian distribution with it as the expectation, but in order to avoid the problem of large target and small probability, the coefficient in front of the Gaussian distribution is removed, and finally In order to make the area with high probability density of target as close to the original target center as possible, take σ i，1 =σ i，2 =size i / 3, the visualization results are as follows Figure 5 and Figure 6 As shown:

[0097] 7.2 Based on the above i (x) can obtain the conditional probability distribution of angle and direction. Let the size and angle of the i-th target be θ i , size i , for coordinates x = (x, y) T For pixels, the target values ​​of angle and size are θ(x), size(x)

[0098]

[0099] Discretize it and set the standard categories of angle and size to be T = {θ s，i , 1≤i≤12}, S={size s，i , 1≤i≤12}, then its discretization target value is:

[0100]

[0101] The category number is cls(θ g (x)) and cls(size g (x)), then the target distribution is:

[0102]

[0103] 8. The loss function is set to cross entropy loss or mean square loss. Let the batch size Batch_size be B, and the image size be H×H. Let the i-th predicted layout image be I i , where the coordinate x i The predicted probability of occurrence, target size and angle are p respectively. ω (x i ), p ω (size|x i ), p ω (θ|x i ), the losses are L1, L2, and L3 respectively.

[0104] If mean square loss is used:

[0105]

[0106] If cross entropy loss is used,

[0107]

[0108] Assume the weights of the three losses are w1, w2, w3, then the final loss

[0109] The training algorithm is as follows:

[0110]

[0111] Step three: autoregressive sampling;

[0112] 1. Specify the target number n that needs to be predicted.

[0113] 2. for t = 1,..., n, construct condition_map in the way mentioned in the training, but here the data of the target used are all based on the prediction of the previous step. Input the background image and condition_map into the network to get the layout map L t .

[0114] 3. Consider the pixel with coordinates x t in condition_map, whose value is value(x t ), let the predicted position probability be p(x t ), take p'(x t ) = (1 - value(x t )) p(x t ), take and take The prediction result can be obtained The reason why we do an operation equivalent to Hadamard product of the predicted position probability map and 1-condition_map is to prevent the predicted targets from overlapping.

[0115] 4. Note that, whether in training or sampling, the input original image x data is randomly added with noise to enhance robustness, and most importantly, to prevent the neural network from learning the high-frequency fine traces left by LaMa when removing small targets. Here we take normal noise with σ = 0.02.

[0116] The sampling algorithm is as follows:

[0117]

[0118] Step four: generation and embedding of remote sensing targets in the background.

[0119] First, the training data processing: obtain target images that meet certain conditions from the public remote sensing image target detection dataset, rotate them to each specified orientation, crop and adjust the image size to 128*128.

[0120] Specifically, the following steps are included:

[0121] 1. The classifier uses the aforementioned trained classifier.

[0122] 2. Rotate to a specified angle.

[0123] 2.1 There are 12 controllable angles: 0°-330°, with a class set every 30°.

[0124] 2.2 Preliminary screening, as above.

[0125] 2.3 Rotate the target to the horizontal and determine the orientation. Determine the orientation of the original target according to the determined orientation. Knowing the orientation of the original target, it can be rotated to the specified angle.

[0126] 2.4 After rotating to the specified angle, obtain the target, frame it with the smallest horizontal frame, and expand it by 2 times with the horizontal frame center as the center. Crop, then perform a second screening, leaving the target image with a longer side greater than 110 and a pure black area ratio less than 0.05, and then adjust the size to 128*128.

[0127] 3. Naming of training data: scene number of target belonging to _ target category _ the first of this category _ target orientation.

[0128] The training process is shown in Figure 7 .

[0129] Based on the palette model, a mask is first generated, which is a square centered on the input image with a side length of 3 / 5 of the original training image side length. Assuming the image x0 shape is 3xHxW, the mask shape is HxW, and the pixel value of the damaged position that needs to be repaired is 1, and the pixel value of the remaining position is 0. Use mask to represent the part that needs to be repaired.

[0130] During training, the following steps are followed:

[0131] I. Sample x0 from training data

[0132] II. Replace the mask part of x0 with standard Gaussian noise ∈0 to get the condition x cond , that is, cond x

[0133] III. Perform the same noise addition process as DDPM, but only add noise to the mask part

[0134] IV. Generate condition y

[0135] V. Set the condition as φ randomly with probability p to train a model without condition.

[0136] VI. Send x t , x cond , y, t into the network to predict the noise, only take the part of mask to do the mean square loss.

[0137] The algorithm is as follows:

[0138]

[0139] The condition y is a simple stripe pattern, which is 1xHxW in shape. The width of the stripe is d, and the length is L, which satisfies the uniform distribution from [-1, 1] in the axial direction (length direction). The place with value 1 is the head, and the place with value -1 is the tail. The orientation of the stripe is the orientation θ of the target to be controlled. The pixels outside the stripe are 0, and the visualization effect is as shown in Figure 8 .

[0140] For the unconditional φ, directly take a zero tensor with a shape of 1xHxW.

[0141] The first dimension of the tensor is combined with the channel dimension of the input background image, and is input as a condition into the U-net.

[0142] In addition, the angle information will also be mapped between [1, T], and then the same operation as the time step embedding will be done, and then added to the time step embedding.

[0143] Setting different ω values during sampling can achieve different condition guiding intensities.

[0144] The sampling process is as shown in Figure 9 .

[0145] I. Obtain the background image x T of the target to be generated (the background image block corresponding to the predicted position and size in advance);

[0146] II. Replace the part of x T with standard Gaussian noise ∈0 to obtain the condition x cond , and this condition will also be used as the new input x T ;

[0147] III. Obtain the condition y (according to the predicted angle);

[0148] IV. Predict the noise based on x t , x cond , y, t, and the guiding intensity is determined by ω;

[0149] V. From the noise ∈ tand x t sample x t-1 ;

[0150] VI. We only update x t-1 with the mask part of x t ;

[0151] The algorithm is as follows:

[0152]

[0153] Figure 10 is the target generated effect map. Figure 11 is the layout map generated effect map. Figure 12 is the overall generated effect map.

Claims

1. A remote sensing scene generation method based on an autoregressive model and a diffusion model, characterized in that, The steps include the following: Step one: remote sensing scene image collection and processing; Remote sensing scene images are obtained from a public remote sensing image target detection dataset, LaMa model is used to remove targets in the image, and the positions, sizes and orientations of these targets are recorded in a csv file; the image is data-augmented and the size of the target is clustered; Step two: predicting the position, size and orientation of a single target; In order to predict the position, size and orientation of a single object; model as: predict the probability of an object of size and orientation appearing at each pixel in an image; specifically, consider a pixel with coordinates , predict the probability of an object appearing at the pixel, and under the condition of an object appearing, predict the distribution of orientation and size , obtaining a joint probability distribution ; the obtained distribution is called a layout map ; from the layout map , take , and then take , , to obtain a prediction result ; Step three: autoregressive sampling; Specifies that a remote sensing background image should be embedded targets, first predict the first target layout ,get , for goals, , whose layout is predicted based on the previous The prediction results of the target , and further obtain the prediction results ; Step four: generation and embedding of remote sensing targets in the background; After sampling the position, size and orientation of the target, the corresponding image block is dug out in the background image according to the position and size of the target, the center of this part is replaced with random noise, and the target is obtained by diffusing the model into the background; in order to control the generation angle of the target, a conditional guide image is connected to the channel dimension of the input, and angle information embedding is added on the basis of time step embedding; CFG without classifier is used to control the angle.

2. The remote sensing scene generation method based on autoregressive model and diffusion model according to claim 1, characterized in that: All the coordinates used in the images are in pixel coordinate system; the coordinates of a certain pixel are denoted as ; The position where the target appears refers to the center coordinate thereof; Refers to the size of the target, in order to align with the method of generating the target, it means twice the length of the largest horizontal frame that frames the target; Refers to the orientation of the target, using the angle system, its definition is consistent with the angle defined under the coordinate, in order to align with the method of generating the target.

3. The remote sensing scene generation method based on autoregressive model and diffusion model according to claim 1 or 2, characterized in that: and are continuous, for simplicity, they are discretized into and are classified into 12 classes respectively.

4. The remote sensing scene generation method based on autoregressive model and diffusion model according to claim 1, characterized in that: In step two, the specific steps are as follows: (2.1) for each remote sensing background image, specifying a maximum value of the number of targets allowed to be considered in the training process ; (2.2) For each image input of a training batch, determine its original all-target number as , consider , randomly select the metadata of targets to input; the metadata includes the location, angle and size data of the targets; (2.3) Sampling ; (2.4) randomly select one of the targets as a known target, and predict the layout map of the remaining targets targets​ (2.5) The prediction network is Unet, and the output layer uses sigmoid activation function; (2.6) The metadata of the preceding target as a condition affects the layout graph prediction of the target: (2.7) Set the target value of the layout diagram: (2.8) The loss function is set as cross-entropy loss or mean square loss; let the batch size Batch_size be , the image size be ; let the th predicted layout be , where the appearance probability, target size and angle distribution at coordinate are , and the loss is .

5. The remote sensing scene generation method based on autoregressive model and diffusion model according to claim 4, characterized in that: In step 2.6, the input of Unet is of shape of , is The original input data Connecting in the channel dimension Conditional graph of Get; for the conditional graph , exist That is, when there is no prediction result as a condition, all its elements are 0. Otherwise, assuming that the i-th target position is , the size is Then the coordinates in the conditional graph are The pixel value of for: ； The values of are linearly mapped from to , and subsequently embedded into the Unet prediction process in the same way as the DDPM.

6. The remote sensing scene generation method based on autoregressive model and diffusion model according to claim 4, characterized in that: In step 2.7, considering the background map of the remaining targets, for the pixel with coordinates , let the probability of the presence of a target be ; where represents the influence of each real target position on it; for each target position , define the distribution form as a Gaussian distribution with its expectation, remove the coefficient in front of the Gaussian distribution, and finally take , take ; Depend on Get the conditional probability distribution of angle and direction; let The size and angle of the target are , the coordinates are The target value of the pixel, its angle and size ; ； Discretize it, set angle and size standard categories are , then the discretization target value is: ； The category number is recorded as and The target distribution is: ； 7. The remote sensing scene generation method based on autoregressive model and diffusion model according to claim 6, characterized in that: If using mean square loss: ； 8.The remote sensing scene generation method based on autoregressive model and diffusion model according to claim 6, characterized in that: If using cross entropy loss, there is ； 9. The remote sensing scene generation method based on autoregressive model and diffusion model according to claim 5, characterized in that: In step three, the specific steps are as follows: (3.1) specifying a target number of predictions required ; (3.2) Construct in the way mentioned in the training , and input the network to get the layout map ; (3.3) Consider The pixel whose coordinate is has a value of Let the predicted position probability be Take Take Take , , and get the prediction result ; (3.4) For both training and sampling, the original image randomly added to enhance robustness, preventing the neural network from learning the high-frequency fine traces left by LaMa when removing small targets; take normal noise.

10. The remote sensing scene generation method based on autoregressive model and diffusion model according to claim 1, characterized in that: In step four, the specific steps are as follows: (4.1) The classifier uses a trained classifier; (4.2) Rotate to the specified angle; There are 12 controllable angles: 0°~330°, set a class every 30° and perform the first screening; Rotate the target to horizontal and determine the orientation, determine the original target orientation according to the determined orientation, rotate the target to the specified angle; Get the target after rotating to the specified angle, box it with the smallest horizontal box, and expand it to 2 times with the horizontal box center as the center, crop, then perform the second screening, leave the target image with a long side greater than 110 and a pure black area ratio less than 0.05, and then adjust the size to 128*128; (4.3) Naming of training data: target scene sequence number_target category_It is the first of this category_target orientation.

Images