An image generation method based on a diffusion model

Abstract

The application discloses an image generation method based on a diffusion model, and relates to the field of computer vision and remote sensing image processing. Through a direction and size perception attention mechanism, the application realizes the cooperative and accurate control of the target position, direction and size in remote sensing image generation, and the geometric consistency of the generated image and the input layout is significantly improved. Based on a complete layout rebalancing, conditional generation and quality screening process, the application can automatically and high-quality synthesize a data set, and the accuracy of a downstream rotating target detector is obviously improved on multiple benchmark data sets, effectively solving the problems of data scarcity, uneven distribution and high labeling cost in remote sensing detection tasks, and providing reliable data support for remote sensing target detection tasks. The method can be used for generating synthetic remote sensing data sets with consistent layout and accurate geometric properties, and is especially suitable for generating remote sensing target detection training data containing various orientations and sizes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and remote sensing image processing, and in particular to an image generation method based on a diffusion model. Background Technology

[0002] Remote sensing image target detection has significant application value in fields such as urban planning and environmental monitoring; however, its performance is largely limited by the availability of large-scale, high-quality labeled data. Currently, high data acquisition costs, cumbersome manual annotation processes, and uneven distribution of target categories in datasets generally restrict the training and practical deployment of detection models. To alleviate the challenge of scarce training data, synthetic data augmentation methods have emerged as an effective solution. The core of these methods is to expand the dataset and balance the sample distribution by generating realistic images, thereby improving the generalization performance and robustness of downstream detection models. The technological evolution in this direction has roughly progressed from general image generation methods to remote sensing-specific generation methods.

[0003] In terms of general image generation methods, technological evolution has progressed from traditional data augmentation to generative adversarial networks (GANs) to diffusion models. Traditional data augmentation methods, such as flipping and scaling, are essentially transformations of existing images and cannot create new objects or complex layouts, resulting in limited diversity gains. Subsequently, while GANs can generate richer content, they generally suffer from inaccurate target location and pattern collapse, leading to unstable generation quality. Current mainstream diffusion models achieve higher image fidelity through a gradual denoising process and can accept conditional inputs such as text or layout for control. However, these general models have a fundamental flaw: a lack of precise spatial control. When applied to remote sensing scenarios requiring strict geometric consistency, they often generate targets that are misaligned with the specified layout, have incorrect orientation, or distorted dimensions, failing to meet the stringent requirements of remote sensing target detection for position, orientation, and scale.

[0004] To address the shortcomings of general-purpose methods, the field of remote sensing has developed specialized data generation methods aimed at improving the localization accuracy of generated targets by introducing spatial conditions such as layout and bounding boxes. Early studies, such as LostGAN, initially verified the feasibility of generating remote sensing images based on layout conditions. In recent years, specialized frameworks based on diffusion models have become mainstream, with representative works such as CC-Diff, MMO-IG, CRS-Diff, and AeroGen emerging. While these methods have made progress in basic position control, their core attention mechanisms and feature sampling methods still do not fully consider the essential geometric characteristics of remote sensing targets. They generally neglect modeling arbitrary orientations and precise dimensions of targets, resulting in generated images that deviate from the real scene in terms of orientation and scale, and exhibit poor layout consistency. This lack of control over orientation and scale makes it difficult for synthetic data generated by existing specialized methods to effectively improve the performance of rotating target detection models.

[0005] For example, in patent application CN116051683A, entitled "A Remote Sensing Image Generation Method, Storage Medium, and Device Based on Style Self-Organization," while the proposed solution can quickly generate labeled remote sensing images, it has inherent flaws: the method heavily relies on predefined, precise target mask contours, resulting in poor flexibility and difficulty in generating targets with natural geometric shapes; more importantly, it completely lacks a fine-grained control mechanism for target orientation and size, failing to guarantee geometric consistency between the generated target and the real scene layout. Furthermore, the strict limitation on the number of denoising iterations in pursuit of generation speed also restricts the improvement of the final image quality, affecting the practical value of the synthesized data in downstream tasks.

[0006] The patent application CN116486077A, entitled "Method and Apparatus for Generating Sample Sets of Semantic Segmentation Model for Remote Sensing Images," constructs pseudo-labels using unsupervised semantic segmentation technology, optimizes the sample labels based on a knowledge-guided intelligent interactive interpretation method, and constructs the sample dataset using an iterative training approach. While this method improves the efficiency of remote sensing sample set production, it relies on an unsupervised semantic segmentation network to generate initial pseudo-labels, and the quality of these pseudo-labels limits the upper limit of the final sample dataset's quality. Furthermore, this method requires multiple iterative optimizations and manual corrections, making the entire process complex and time-consuming. Moreover, error accumulation during iteration can affect the accuracy of the final sample dataset.

[0007] Existing processing methods suffer from poor layout consistency in generated images because their core mechanisms fail to accurately model the target's orientation and size. This severely limits the effectiveness of synthetic data in improving the performance of rotating target detectors. Summary of the Invention

[0008] This invention provides an image generation method based on a diffusion model, which aims to solve the shortcomings of existing remote sensing image generation methods in controlling target orientation and size, as well as the resulting problems of poor consistency in the layout of synthetic data and difficulty in effectively improving the detection performance of rotating targets.

[0009] The technical solution adopted in this invention is: an image generation method based on a diffusion model, which includes the following steps:

[0010] Step 1: Perform distribution statistics on the layout labels of the original remote sensing dataset to extract the distribution of categories, target locations, target sizes, and target orientation angles (i.e., rotation angles);

[0011] The layout labels are adjusted to obtain a balanced layout label distribution: for the category distribution, the category sampling probability is adjusted using inverse frequency weighting; for the target location distribution (i.e., spatial location distribution), the spatial location is sampled using an approximately uniform distribution; the target size distribution remains unchanged; and the target orientation angle distribution is sampled based on a uniform distribution.

[0012] Step 2: Randomly determine the number of targets for each synthetic image to be generated. Based on the obtained balanced layout label distribution, sample the category, position coordinates, target size and target orientation angle of each target in sequence to obtain the layout label of each target in the synthetic image to be generated.

[0013] Step 3: Convert the layout tags of each target obtained in Step 2 into spatial layout embeddings and semantic text embeddings, and use them as multimodal conditional embeddings.

[0014] Step 4: Based on multimodal conditional embedding, Gaussian noise is gradually converted into latent variables (also known as latent variables) of the remote sensing image through the constructed direction and angle (scale) sensitive diffusion model (OSA-Diff). Then, the latent variables of the final remote sensing image are converted into the remote sensing image (e.g., the latent variables are decoded into the remote sensing image through an image decoder) to obtain the generated synthetic image.

[0015] The orientation and angle-sensitive diffusion model is a U-shaped network based on residual modules and layout cross-attention modules. It includes an encoder part composed of alternating stacks of residual modules and layout cross-attention modules, and a decoder part composed of alternating stacks of residual modules and layout cross-attention modules. The output of the decoder part is output by the orientation and scale attention module to output the latent variables obtained by denoising at each step. The input of the orientation and angle-sensitive diffusion model is Gaussian noise or the latent variables obtained by denoising in the previous step.

[0016] Step 5: The quality of the synthetic images generated in Step 4 is evaluated based on semantic-layout alignment and target classification confidence. Synthetic images that simultaneously meet the semantic-layout alignment threshold and target classification confidence threshold are considered high-quality synthetic images. An enhanced remote sensing dataset is obtained based on the original remote sensing dataset and all high-quality synthetic images.

[0017] Furthermore, step 1, the specific process of adjusting the distribution of layout tags, includes:

[0018] By using inverse frequency weighting to adjust the category sampling probabilities, we obtain the category distribution after balance. :

[0019] in, This represents the extracted category distribution. and The sizes of the original dataset and the synthetic dataset are respectively. This represents the total number of categories;

[0020] By sampling spatial locations using an approximately uniform distribution, the equilibrium position distribution is obtained. :

[0021]

[0022] in, Represents the total area of ​​the image. These are the width and height of a single image, respectively. Represents pixel coordinates;

[0023] Based on uniformly distributed sampling of direction angles, the balanced direction angle distribution is obtained. ,in, Indicates the direction angle.

[0024] Furthermore, in step 2, the range of the target number for each synthesized image to be generated is set to... .

[0025] Furthermore, in step 3, the spatial layout is embedded as follows:

[0026]

[0027] in, These are position parameters used to characterize the coordinates of the four corner points of the target's rotated bounding box, determined based on the target's position coordinates, target size, and target orientation angle. Indicates the positional encoding of sine and cosine. This represents the text encoder of the contrastive language-image pre-trained CLIP model. This represents a multilayer perceptron.

[0028] Furthermore, in step 3, semantic text embedding... for:

[0029] .

[0030] in, This indicates the selected large language model. This represents the layout label of the target obtained in step 2. This represents the prompt template set for the large language model.

[0031] Furthermore, in step 4, the cross-attention module is laid out as follows:

[0032]

[0033] in, This indicates the layout cross-attention feature. Output characteristics of the residual module The mapping features, i.e. , , The layout embeds the mapped values ​​and key features. , ; , The values ​​and key features are embedded in the text. , ,in, This is a query mapping matrix for residual features. , These are the value mapping matrix and key mapping matrix for layout embedding, respectively. , These are the value mapping matrix and key mapping matrix for text embedding, respectively. The scaling factor set is used in the layout cross-attention module to limit the numerator part of the expression. The calculation results are within the normal range.

[0034] Furthermore, in step 4, the direction and scale attention module is set as follows:

[0035] Three prediction networks based on multi-layer convolutional neural networks were used to obtain information about height. ,width and direction angle convolution kernel parameters The input to the multi-layer convolutional neural network includes the output of a direction- and angle-sensitive diffusion model. and spatial layout embedding ; Subscript Used to indicate height ,width and direction angle ;

[0036] convolution kernel parameters Scale to the preset value range to obtain the scaled parameters. That is, dynamically variable parameters; and based on height and width Corresponding parameters Construct a sampling grid for the current feature sampling points; then based on the orientation angle Corresponding parameters Rotate the sampling grid and perform a convolution operation on the rotated sampling grid to obtain the corresponding output feature map. ;

[0037] The layout labels of the target obtained in step 2 The rotated bounding box of the defined target is converted into a binary space mask. The mask of pixels located within the rotated bounding box is set to 1, and the mask of the rest is set to 0;

[0038] based on Obtain the output of the direction and scale attention module ,in, The scaling factor is set. Indicates the target number. Query the target number of a single image. ,key ,value They are respectively: , , , , , These are mapping matrices for queries, keys, and values, respectively.

[0039] Furthermore, the network structure of the prediction network includes, in sequence: convolutional layer 1, activation function 1, dropout layer 1, convolutional layer 2, activation function 2, dropout layer 2, convolutional layer 3, and activation function, wherein the activation function is preferably the tanh function.

[0040] Furthermore, the sampling grid for the current feature sampling points is constructed as follows:

[0041]

[0042] in, Represents the coordinates of the sampling grid. , They are respectively height and width Corresponding parameters .

[0043] Furthermore, in step 5, the semantic-layout alignment is set to:

[0044]

[0045] in, The layout labels for the target obtained in step 2 The text description, This represents the image encoder in the CLIP model. This represents the text encoder in the CLIP model. () indicates the calculation of cosine similarity.

[0046] Furthermore, in step 5, the high-quality synthesized image The criteria are: semantic-layout alignment greater than or equal to 0.7, and target classification confidence greater than or equal to 0.85.

[0047] The technical solution provided by this invention brings at least the following beneficial effects:

[0048] This invention implements an image synthesis method for remote sensing image target detection with orientation and size awareness capabilities. It is used to generate synthetic remote sensing datasets with consistent layout and accurate geometric attributes, solving the problems of low consistency between the generated results and layout and inaccurate control of target orientation and size in existing methods. It provides a high-quality remote sensing image dataset synthesis scheme that is sensitive to target orientation and scale and has class balance, thereby improving the performance of downstream target detection models. It is especially suitable for generating remote sensing target detection training data containing diverse orientations and sizes.

[0049] This invention achieves precise and coordinated control of target position, orientation, and size in remote sensing image generation through a proposed orientation and size-aware attention mechanism, significantly improving the geometric consistency between the generated image and the input layout. Based on a complete layout rebalancing, conditional generation, and quality screening process, it can automatically synthesize high-quality datasets, significantly improving the accuracy of downstream rotating target detectors on multiple benchmark datasets. This effectively addresses the core pain points of data scarcity, uneven distribution, and high annotation costs in remote sensing detection tasks, providing reliable data support for remote sensing target detection tasks. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1This is a schematic diagram illustrating the processing steps of an image generation method based on a diffusion model proposed in this invention.

[0052] Figure 2 The overall architecture of the method of this invention is shown, illustrating the complete process from layout sampling to final dataset generation. It can be divided into three stages: layout distribution rebalancing and sampling, target orientation and scale-sensitive remote sensing image generation, and quality screening.

[0053] Figure 3 This section describes the Orientation and Angle-Sensitive Diffusion Model (OSA-Diff) in this invention. (a) shows the framework of the OSA-Diff model, whose input is noise, and whose conditional inputs are layout embeddings and text embeddings. OSA-Diff predicts the noise, and after denoising and decoding, a clearer synthetic image is obtained for the next step. (b) shows the detailed structure of the Orientation and Scale-Aware Attention (OSA) module, which achieves accurate modeling of oriented targets through rotation feature sampling. (c) shows the Layout Cross-Attention (LCA) module, which provides unified guidance for the generation process by fusing layout embeddings and text embeddings.

[0054] Figure 4 To compare the generation results of the method of this invention with existing technologies on the DIOR-R dataset, the figure shows that the method proposed in this invention surpasses existing technologies in terms of the consistency between the generated objects and the target labels across multiple categories. This comparison figure verifies the significant advantage of this invention in maintaining layout consistency.

[0055] Figure 5 To compare the performance of the synthesized data generated by this invention with the state-of-the-art aerial image diffusion model (AeroGen) in enhancing target detector performance, the detection model trained on the dataset synthesized by this invention outperforms the model trained on the original dataset in most categories, demonstrating the superiority of this invention in improving detector performance. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0057] This invention provides an image generation method based on a diffusion model, which is a direction- and size-aware remote sensing image synthesis method. It addresses the problems of low consistency between the generated image and the original layout, and insufficient precision in controlling the target direction and size, in existing remote sensing image generation methods that rely on an input layout. These issues make the generated data difficult to directly apply to downstream target detection tasks. This invention, through an image synthesis method using a direction- and size-aware attention mechanism, can generate high-quality remote sensing datasets that are sensitive to target direction and scale, and are class-balanced.

[0058] In one embodiment, see Figure 1 and Figure 2 The image generation method based on a diffusion model provided in this invention can be divided into three core stages: layout rebalancing, geometrically perceptive image generation, and refined data filtering, to automatically generate a synthetic dataset that can significantly improve the performance of a rotating target detector. Its specific implementation process includes:

[0059] Phase 1: Layout distribution rebalancing and sampling.

[0060] The core task of this stage is to analyze and adjust the distribution characteristics of layout labels in the original dataset. Layout labels refer to the category, location, size, and orientation information of all labeled targets in a single image. The analysis object is the statistical distribution of all layout labels in the entire dataset across these dimensions (category, location, scale, and orientation). The original distribution of these dimensions is significantly imbalanced at the dataset level. To address this, this invention designs a step-by-step rebalancing sampling mechanism: first, a balanced category distribution is constructed at the dataset level, and target categories are sampled; based on this, a balanced location and orientation distribution are then sampled sequentially. Through this step-by-step, ordered sampling strategy, more balanced layout labels across key attributes such as category, location, and orientation can be obtained. This allows for the generation of a dataset with good statistical characteristics based on these layout labels, which is beneficial for improving the training effect of the object detector. The specific processing steps of the first stage include:

[0061] Step 1: Data distribution analysis and rebalancing strategy formulation.

[0062] First, statistical analysis was performed on the layout labels of the original remote sensing dataset, extracting the distributions of four key dimensions: category distribution, target location distribution, target size distribution, and target orientation distribution. These distributions reflect: the frequency differences of different target categories; the location preference of targets in the image; the statistical characteristics of the target size range; and the distribution pattern of target orientation angles, respectively. Analysis of these distributions revealed that the original category distribution exhibited a long-tailed distribution, the original location distribution showed a central clustering phenomenon, and the original target orientation distribution lacked diversity. To address these issues, a rebalancing strategy was developed: adjusting the category sampling probability through inverse frequency weighting to alleviate the long-tailed distribution problem; using an approximately uniform distribution to sample spatial locations to eliminate central bias; maintaining the true size distribution of categories unchanged; and sampling orientation angles based on a uniform distribution to improve rotational diversity.

[0063] Step 2, layout label sampling.

[0064] In step 1, an adjusted and balanced layout label distribution is obtained. In this step, samples are taken from this adjusted layout label distribution to obtain the layout labels needed for the subsequent image synthesis process. Specifically, the number of targets is first randomly determined for each image to be generated (usually between 1 and 15), and then the category, position coordinates, size, and rotation angle of each target are sampled sequentially according to the rebalanced distribution probability. This process ensures that the generated layout retains the statistical characteristics of the real scene while effectively compensating for the imbalance in the original data distribution through the rebalancing mechanism, providing geometrically reasonable and evenly distributed layout labels for subsequent image generation.

[0065] Second stage: Generation of remote sensing images that are sensitive to target orientation and scale.

[0066] This stage is the core of the entire image generation system, achieving precise orientation and scale synthesis of remote sensing images through the attention mechanism set up in this invention. Specifically, it includes the following steps:

[0067] Step 3: Spatial layout embedding and semantic text embedding generation.

[0068] The layout labels obtained in the first stage are converted into two complementary conditional representations: spatial layout embedding and semantic text embedding. First, the original layout labels are converted into categories and four location coordinates. This conversion is lossless and facilitates operation. In obtaining the spatial layout embedding, the categories are encoded using CLIP (Contrastive Language-Image Pre-training), and the location coordinates are encoded using Fourier transform. These two encodings are then fused using a simple multilayer perceptron to obtain the spatial layout embedding. Second, the layout labels are input into a large language model, which generates a text sentence with a rough positional relationship based on these layout labels. This text is then encoded using CLIP to obtain the semantic text embedding. These two embeddings together constitute the multimodal conditional embeddings in the subsequent remote sensing image generation process.

[0069] Step 4: Generation of orientation and scale-sensitive remote sensing images.

[0070] This step is a progressive denoising process. It begins with Gaussian noise, which is gradually transformed into a latent variable of the remote sensing image using a direction- and angle-sensitive diffusion model (OSA-Diff) based on conditional embeddings (spatial layout embedding and semantic text embedding). This noise is then decoded into the corresponding remote sensing image. Specifically, this step first uses a layout cross-attention module (i.e., layout cross-attention U-Ne) to deeply fuse spatial layout embedding and semantic text embedding into multiple feature layers of the U-Net, achieving coordinated semantic and spatial guidance. Subsequently, a key adaptive sampling operation is performed through an orientation and scale-aware attention module: this module first adaptively predicts the corresponding convolutional kernel parameters based on each target layout and generates a rotated rectangular sampling grid accordingly; then, a mask attention mechanism is executed within the local region defined by the layout. This step precisely controls the orientation, size, and position of the generated target by resampling the features, ensuring strict alignment with the geometric properties of the input layout.

[0071] Phase 3: Quality screening.

[0072] This stage aims to filter the generated images, removing samples with inaccurate positioning, semantic inaccuracies, and poor image quality, thereby ensuring the practical value of the synthesized data. Specifically, it includes the following two steps:

[0073] Step 5, Quality Assessment and Quantification.

[0074] Rigorous quality assessments were performed on all images generated in the second stage, examining semantic consistency and layout consistency. First, a semantic consistency score between the generated image and the input layout was calculated based on the CLIP model to quantify the semantic matching degree between the layout bounding box and the generated object. Simultaneously, a ResNet classifier pre-trained on real remote sensing data was used to identify targets in the generated image, and the average classification confidence score was used as a target layout consistency index. These two scores were used to evaluate the quality of the synthesized image from the perspectives of semantic consistency and layout consistency, respectively.

[0075] Step 6: Threshold filtering and final dataset construction.

[0076] Based on the quality assessment results, a selection threshold was set to retain only high-quality samples that simultaneously meet the requirements of semantic consistency and layout consistency. Specifically, for semantic consistency, the CLIP alignment score must be no less than 0.75. For layout consistency, the classification confidence score must be no less than 0.85. The selected samples were integrated with the original dataset to form a larger, more balanced, and higher-quality augmented dataset, which can be directly used to train downstream rotating object detection models, effectively improving detection performance.

[0077] In existing dataset generation processes, the input labels are processed using a dual cross-attention U-Net, along with layout and text conditions, to perform mask-guided denoising and output synthetic images. However, some of these synthetic images contain targets whose orientation and scale do not match the labels. In the method of this invention, as shown in the example... Figure 1 As shown, based on the set layout cross attention U-Ne, and the layout and cross conditions, orientation and size-aware attention processing is realized. Combined with adaptive convolution kernel prediction (including size, aspect ratio and orientation), the corresponding sampling grid is dynamically constructed (i.e., dynamic sampling). With the help of the layout mask attention mechanism, the operation is strictly limited to the foreground area, realizing precise spatial control of the generated image. This method can perform precise spatial control and obtain a synthetic image with target orientation and scale matching the label.

[0078] In one embodiment, see Figure 3 The direction- and angle-sensitive diffusion model (OSA-Diff) used in this embodiment specifically includes: the framework of the direction- and angle-sensitive diffusion model, such as... Figure 3 As shown in (a), the input is a noisy image or pure noise, and the conditional inputs are layout embedding and text embedding. OSA-Diff predicts the noise, and after denoising and decoding, a clearer synthetic image is obtained in the next step. Among them, the orientation and scale-aware attention (OSA) module, as shown in (a), is used to generate the image. Figure 3 As shown in (b), this module achieves accurate modeling of the oriented target through rotational feature sampling; the layout cross-attention (LCA) module, as shown in (b), Figure 3 As shown in (c), it provides unified guidance for the generation process by integrating layout embedding and text embedding. Based on the diffusion model OSA-Diff, the overall implementation process of this embodiment includes:

[0079] First, the layout distribution rebalancing and sampling in the first stage. The core task of this stage is to balance the distribution of layout labels in the original dataset. First, the statistical distribution of all targets in the entire dataset in terms of category, location, size, and orientation is analyzed, revealing a significant imbalance. Therefore, based on the step-by-step rebalancing sampling mechanism designed in this invention, more balanced layout labels across all dimensions are obtained, specifically including steps S1 to S2:

[0080] Step S1: Data distribution analysis and rebalancing strategy formulation.

[0081] First, the original remote sensing dataset... A comprehensive statistical analysis of the annotation information was conducted to obtain the categories. ,Location ,scale The distribution of angle θ (direction angle) , . Represents the original remote sensing dataset The image in the image, subscript Number the images. and For the width and height of the image, each image The corresponding layout tag is ,in, For image The target number in , For image The target quantity, , , , Images The first in The location, scale, angle, and category label of each target.

[0082] In order to solve the problem of the original remote sensing dataset based on the rebalancing strategy set by this invention. To address the data distribution imbalance, the following steps should be taken;

[0083] First, an inverse frequency weighting method is used to balance the original class distribution. :

[0084]

[0085] in, This represents the balanced class distribution. and The sizes of the original dataset and the synthetic dataset are respectively. This represents the total number of categories.

[0086] Secondly, replace the original target location distribution with a uniform location distribution:

[0087]

[0088] in, For the equilibrium position distribution, Represents the total area of ​​the image. These represent the width and height of a single image, respectively.

[0089] Then, the orientation angle sampling also adopts a uniform distribution, according to the formula The equilibrium directional angle distribution is obtained;

[0090] Finally, the size distribution retains the size distribution of the original data. To maintain authenticity.

[0091] Step S2: Generate diverse layouts.

[0092] After achieving a balanced distribution, an automated layout generation process is executed. For each composite image, the target number is first generated. Then sample in the following order Parameters for each target:

[0093] (1) Based on the balanced category distribution Sampling categories ;

[0094] (2) Based on the positional distribution after equilibrium Sampling center location ;

[0095] (3) Based on size distribution Sampling size ;

[0096] (4) Based on the directional angle distribution after equilibrium Sampling direction angle .

[0097] By combining these parameters, you can obtain the desired layout tags. This process ensures the generated layout. It maintains the statistical characteristics of real-world scenarios while effectively compensating for the imbalance in the distribution of the original data.

[0098] The second stage performs orientation and scale-sensitive remote sensing image generation. This stage aims to generate remote sensing image targets whose geometric attributes (including orientation and scale) are strictly aligned with the input layout conditions. Specifically, it includes steps S3 to S4:

[0099] Step S3: Layout embedding and text embedding generation.

[0100] The layout tags obtained in step S2 This is converted into two complementary conditional representations: spatial layout embedding and text layout embedding.

[0101] like Figure 3 As shown in (a), the spatial layout is embedded. Generated through Fourier position encoding:

[0102]

[0103] Among them, position parameter , by parameters This is obtained through conversion, which is for computational convenience and without information loss. Among them, The eight parameters are used to characterize the coordinates of the four corner points, i.e. , , and Four corner points, Indicates the positional encoding of sine and cosine. The text encoder representing the CLIP model. This represents a multilayer perceptron.

[0104] Semantic text embedding The generation process is as follows:

[0105] .

[0106] in, This indicates the selected large language model. This represents the prompt template set for the large language model.

[0107] Layout tags Input a large language model, and have it output a text description that roughly corresponds to the layout tags. Finally, use a text encoder. Encode this text into a text embedding. Prompt Template It can guide large language models to generate natural language descriptions that include spatial relationships. A complete prompt template can be set as follows:

[0108] "You are an expert in describing aerial image layouts. Transform the given layout information into a concise, fluent sentence. Layout input (layout_info) instructions: (1) Describe the scene based on object categories and their relative spatial relationships; (2) Infer approximate locations using directed bounding box coordinates, but do not mention specific numerical coordinates; (3) Focus on the most prominent objects and their relationships; (4) Keep the description as a sentence. Output only the generated description." In this embodiment, Deepseek-R2 is used as the large language model (LLM).

[0109] Step S4, diffusion generation and geometry refinement.

[0110] In this step, spatial layout embedding is used. and semantic text embedding Guide the direction and angle-sensitive diffusion model OSA-Diff to generate images.

[0111] like Figure 3 As shown in (a), the spatial layout is embedded. and semantic text embedding The OSA-Diff diffusion model, which is sensitive to direction and angle, is used for iterative denoising. At each time step... The model performs the following calculations: First, through Extract layout-aware features:

[0112]

[0113] in, Represents Gaussian noise or the latent variable obtained from the previous denoising step, where Indicates layout cross attention (LCA) network.

[0114] In this embodiment, The network structure is set as follows: an encoder section consisting of a ResNet module, an LCA module, another ResNet module, and an LCA module in sequence; and a decoder section consisting of a ResNet module, an LCA module, and another ResNet module, and an LCA module in sequence. The input of each LCA module also includes spatial layout embedding. and semantic text embedding .

[0115] like Figure 3 As shown in (c), the LCA module calculates as follows:

[0116]

[0117] in, This indicates the layout cross-attention feature. for Output features of the ResNet module The mapping features (corresponding to the query), i.e. , , Embedded to the transformed layout, , Embedded in the transformed text, i.e. , , , ,in, This is a query mapping matrix for residual features. , These are the value mapping matrix and key mapping matrix for layout embedding, respectively. , These are the value mapping matrix and key mapping matrix for text embedding, respectively. This is the scaling factor set. In this embodiment, the LCA module is based on the input features. Embedded spatial layout and semantic text embedding Through two multi-head attention respectively , ,as well as , After processing, the results are summed and standardized to produce the output. .

[0118] Subsequently, the Orientation and Size Aware Attention (OSA) module performs feature resampling. Figure 3 As shown in (b) above. This step utilizes the predicted geometric parameters to construct a rotated sampling mesh and performs dynamic convolution to extract aligned features. First, it involves three independent networks... Predict the convolution kernel parameters and rotation angle parameters:

[0119]

[0120] in, The parameters of the predicted convolution kernel are given, with the subscript i indicating the height. ,width and direction angle , Represents network Network parameters. Three networks. The network structure is the same, consisting of: convolutional layer 1, activation function 1, dropout layer 1, convolutional layer 2, activation function 2, dropout layer 2, convolutional layer 3, and activation function, wherein the activation function is preferably the tanh function.

[0121] Then scale the parameters to a preset range:

[0122]

[0123] in, This represents a dynamically variable parameter. , These are represented by preset weights and intercepts, respectively.

[0124] To make it easier to distinguish, let , , .

[0125] Subsequently according to , Construct the current feature sampling point Dynamic sampling grid

[0126]

[0127] in, , It has dynamic height and width, ranging from [3, 15], which can be adjusted... and The value is used to achieve this.

[0128] Then, according to Rotating sampling grid:

[0129]

[0130] Finally, in the dynamic sampling grid Perform convolution operations on it.

[0131] To strictly limit the above operations to layout tags Within the defined rotated bounding box (OBB) of the object, this step introduces a layout mask attention mechanism. Each rotated bounding box is converted into a binary space mask. The mask for pixels located within the rotated bounding box is set to 1, and the rest to 0. The following operation restricts the dynamic feature transformation performed above to the target region:

[0132]

[0133] Among them, spatial mask Source: layout tag ; The scaling factor is set. Each is composed of a dynamic sampling grid The feature map obtained after performing a convolution operation is then mapped accordingly to obtain the final feature map. , , , To sample the grid in a dynamic manner The feature map obtained after performing the convolution operation. , , These are mapping matrices for queries, keys, and values, respectively. It is the output of the direction and scale attention module, also represented as After several steps of noise reduction, the final step... After decoding by the image decoder, the desired synthesized image is obtained. .

[0134] The training details for this phase are as follows:

[0135] The orientation and size-aware diffusion model (OSA-Diff) is trained using the standard objective function of the diffusion model:

[0136]

[0137] in, Denotes the mathematical expectation, and represents the expectation with respect to time steps. Original noise-free image (i.e., the synthesized image output by the model) and noise Find the average of the joint distributions. Indicates noise scheduling parameters, This represents a noise prediction network.

[0138] During training, the residual blocks, text encoder, and LLM parameters of the pre-trained model are frozen, and only the sine and cosine positional codes are updated. Parameters of the LCA and OSA modules. For example... Figure 4 As shown, compared with existing methods, the method proposed in this invention can better maintain consistency between the scale and orientation of objects and their corresponding layout labels when generating remote sensing images.

[0139] The third stage is quality screening and dataset construction. The purpose of this stage is to systematically evaluate and screen the quality of the generated images and integrate them into a synthetic remote sensing dataset that can be directly used for downstream tasks. This specifically includes steps S5 and S6:

[0140] Step S5: Quality assessment and quantification.

[0141] For each generated image Conduct a dual quality assessment, including:

[0142] Semantic-layout alignment score:

[0143]

[0144] in, A textual description of the layout. This represents the image encoder in the CLIP model. This represents the text encoder in the CLIP model. () indicates the calculation of cosine similarity.

[0145] The target classification confidence score is calculated using a pre-trained ResNet classifier:

[0146]

[0147] in, This indicates the number of targets in a single image. Indicates from the generated In the middle, according to the layout tags The Middle The bounding box location information of each target is used to crop out the corresponding sub-image region. This represents the sub-image region predicted by the classifier. Category The probability of.

[0148] Step S6: Threshold Filtering and Final Dataset Construction

[0149] Establish filtering criteria:

[0150]

[0151] The selected samples are merged with the original dataset to form the final augmented dataset. .

[0152] like Figure 5 As shown in Table 1, after training with the dataset synthesized in this embodiment, the accuracy of the object detector is improved compared with the model trained with the original dataset, and this improvement is consistent across different models.

[0153] Table 1. Performance comparison of raw data and generated data on different detectors.

[0154]

[0155] Table 1 compares the detection accuracy of multiple detector models on the synthesized dataset in this embodiment with the original dataset. The experimental results validate the effectiveness of the synthesized data: after training on the synthesized dataset in this embodiment, the detection accuracy of multiple mainstream detector models is superior to that trained on the original dataset. In Table 1, YOLO is an object detection algorithm (You Only Look Once), ATSS is Adaptive Training Sample Selection, Faster R-CNN is a faster region-based convolutional neural network, FCOS is Fully Convolutional One-Stage Object Detection, and RetinaNet is a retina network. Wherein, aug represents the dataset enhanced by the method of this invention, AP represents an airplane, AR represents an airport, BF represents a baseball field, BC represents a basketball court, BR represents a bridge; CH represents a chimney, DM represents a dam, ESA represents a highway service area, ETS represents a highway toll station, GF represents a golf course, GTF represents a ground track field, HB represents a port, OP represents an overpass or interchange, SH represents a ship, ST represents a stadium, STT represents an oil storage tank, TC represents a tennis court, TS represents a train station, VE represents a vehicle; and WM represents a windmill.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0157] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. An image generation method based on a diffusion model, characterized in that, Includes the following steps: Step 1: Perform distribution statistics on the layout labels of the original remote sensing dataset to extract the category distribution, target location distribution, target size distribution, and target orientation angle distribution; The layout labels are adjusted to obtain a balanced layout label distribution: for the category distribution, the category sampling probability is adjusted using inverse frequency weighting; for the target location distribution, the spatial location is sampled using an approximately uniform distribution. The target size distribution remains unchanged; the target orientation angle distribution is sampled based on a uniform distribution. Step 2: Randomly determine the number of targets for each synthetic image to be generated. Based on the obtained balanced layout label distribution, sample the category, position coordinates, target size and target orientation angle of each target in sequence to obtain the layout label of each target in the synthetic image to be generated. Step 3: Convert the layout tags of each target obtained in Step 2 into spatial layout embeddings and semantic text embeddings, and use them as multimodal conditional embeddings. Step 4: Based on multimodal conditional embedding, Gaussian noise is gradually converted into latent variables of the remote sensing image through the constructed direction and angle sensitive diffusion model. Then, the latent variables of the final remote sensing image are converted into the remote sensing image to obtain the generated synthetic image. The orientation and angle-sensitive diffusion model is a U-shaped network based on residual modules and layout cross-attention modules. It includes an encoder part composed of alternating stacks of residual modules and layout cross-attention modules, and a decoder part composed of alternating stacks of residual modules and layout cross-attention modules. The output of the decoder part is passed through the orientation and scale attention module to output the latent variables obtained by denoising at each step. The input of the orientation and angle-sensitive diffusion model is Gaussian noise or the latent variables obtained by denoising in the previous step. Step 5: The quality of the synthetic images generated in Step 4 is evaluated based on semantic-layout alignment and target classification confidence. Synthetic images that simultaneously meet the semantic-layout alignment threshold and target classification confidence threshold are considered high-quality synthetic images. An enhanced remote sensing dataset is obtained based on the original remote sensing dataset and all high-quality synthetic images. In step 3, the spatial layout is embedded. for: ； in, These are position parameters used to characterize the coordinates of the four corner points of the target's rotated bounding box, determined based on the target's position coordinates, target size, and target orientation angle. Indicates the positional encoding of sine and cosine. This represents the text encoder of the contrastive language-image pre-trained CLIP model. This represents a multilayer perceptron. For category; In step 3, semantic text embedding for: ； in, This indicates the selected large language model. This represents the layout label of the target obtained in step 2. This represents the prompt template set for the large language model; In step 4, the cross-attention module is laid out as follows: ； in, This indicates the layout cross-attention feature. Output characteristics of the residual module The mapping features, i.e. , , The layout embeds the mapped values ​​and key features. , ; , The values ​​and key features are embedded in the text. Embedded in the spatial layout, , ,in, This is a query mapping matrix for residual features. , These are the value mapping matrix and key mapping matrix for layout embedding, respectively. , These are the value mapping matrix and key mapping matrix for text embedding, respectively. This indicates the scaling factor that is set; In step 4, the direction and scale attention module is set as follows: Three prediction networks based on multi-layer convolutional neural networks were used to obtain information about height. ,width and direction angle convolution kernel parameters The input to the multi-layer convolutional neural network includes the output of a direction- and angle-sensitive diffusion model. and spatial layout embedding ; Subscript Used to indicate height ,width and direction angle ; convolution kernel parameters Scale to the preset value range to obtain the scaled parameters. ; and according to height and width Corresponding parameters Construct a sampling grid for the current feature sampling points; then based on the orientation angle Corresponding parameters Rotate the sampling grid and perform a convolution operation on the rotated sampling grid to obtain the corresponding output feature map. ; The layout labels of the target obtained in step 2 The rotated bounding box of the defined target is converted into a binary space mask. The mask of pixels located within the rotated bounding box is set to 1, and the mask of the rest is set to 0; based on Obtain the output of the direction and scale attention module ,in, The scaling factor is set to limit the scaling factor. The calculation results are within the normal range. Indicates the target number. Query the target number of a single image. ,key ,value They are respectively: , , ,in, , , These are mapping matrices for queries, keys, and values, respectively.

2. The image generation method based on a diffusion model as described in claim 1, characterized in that, Step 1, the specific process of adjusting the distribution of layout tags, includes: By using inverse frequency weighting to adjust the category sampling probabilities, we obtain the category distribution after balance. : ; in, This represents the extracted category distribution. and The sizes of the original dataset and the synthetic dataset are respectively. This represents the total number of categories; By sampling spatial locations using an approximately uniform distribution, the equilibrium position distribution is obtained. : ； in, Represents the total area of ​​the image. These are the width and height of a single image, respectively. Represents pixel coordinates; Based on uniformly distributed sampling of direction angles, the balanced direction angle distribution is obtained. ,in, Indicates the direction angle.

3. The image generation method based on a diffusion model as described in claim 1, characterized in that, The network structure of the prediction network consists of the following layers in sequence: convolutional layer 1, activation function 1, dropout layer 1, convolutional layer 2, activation function 2, dropout layer 2, convolutional layer 3, and activation function.

4. The image generation method based on a diffusion model as described in claim 1, characterized in that, The sampling grid for the current feature sampling points is constructed as follows: ； in, Represents the coordinates of the sampling grid. , They are respectively height and width Corresponding parameters .

5. The image generation method based on a diffusion model as described in claim 1, characterized in that, In step 5, the semantic-layout alignment is set to: ； in, The layout labels for the target obtained in step 2 The text description, This represents the image encoder in the CLIP model. For each image generated, This represents the text encoder in the CLIP model. () indicates the calculation of cosine similarity.

Images