Aerial photograph controllable multi-view data generation method based on graph-born 3D large model
By using large-scale 3D models to transform planar perspective data into drone aerial photography data, the problem of lack of diversity and perspective in drone aerial photography datasets is solved, and efficient aerial photography sample augmentation and content consistency control are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NAT INNOVATION INST OF DEFENSE TECH PLA ACAD OF MILITARY SCI
- Filing Date
- 2025-08-04
- Publication Date
- 2026-07-14
Smart Images
Figure CN122391774A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data augmentation, specifically relating to a method for generating controllable multi-view aerial photography data based on a large-scale 3D model derived from a graph. Background Technology
[0002] With the development of the "low-altitude economy," drones are increasingly being used in various smart city applications, such as logistics delivery, traffic monitoring, and law enforcement. In these tasks, drones need to accurately understand urban scenarios. Deep learning-based computer vision technology, due to its significantly superior generalization performance compared to traditional vision algorithms, has gradually become one of the key supporting technologies for drone environmental perception. However, deep learning-based computer vision algorithms often require a large amount of data for training, placing high demands on the scale and diversity of the training data. However, with the implementation of drone "no-fly" orders in various cities, collecting aerial photography data from the drone's perspective has become increasingly difficult. Existing data suffers from problems such as lack of perspective, limited target categories, and insufficient scene diversity, failing to effectively improve the generalization performance of perception models trained on such data and making it difficult to meet the needs of practical applications.
[0003] To address the shortage of drone aerial photography data, numerous data augmentation algorithms have been proposed. While the Stable Diffusion model can guide image generation using text, the lack of information in the text descriptions and insufficient constraints on the generated content prevent precise control over the generated images. Subsequent methods like T2I-Adapter and ControlNet use graph-to-graph approaches, adding image constraints to textual constraints. They guide the Stable Diffusion model through edge, depth, and pose information, improving the controllability of the generation process. However, they struggle to guarantee consistency in the generated images and cannot generate image samples with specific targets and poses. Furthermore, they cannot augment the shooting perspective or scene content of the generated images, failing to address the lack of perspective and diversity in drone aerial photography datasets. Summary of the Invention
[0004] The purpose of this invention is to provide a method for generating controllable multi-view aerial photography data based on a large 3D model of image-generated images. This method can convert planar view shooting data into UAV aerial photography data while maintaining the consistency of the generated layout content. It can also control the shooting angle and altitude of the generated data, ultimately achieving efficient aerial photography sample augmentation.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] A method for generating controllable multi-view aerial photography data based on a large-scale 3D model derived from images, the method comprising:
[0007] Acquire planar viewpoint data and use an image segmentation model to segment the foreground and background portions of the planar viewpoint data to form multiple two-dimensional instances;
[0008] Based on a multi-view diffusion model, N two-dimensional images are generated from the multiple two-dimensional instances;
[0009] Based on a multi-view 3D reconstruction model, 3D entities corresponding to the foreground and background parts are generated according to the 2D image;
[0010] Design different scene scripts and reconstruct the scene based on the three-dimensional entity;
[0011] Based on the reconstructed scene, obtain image sample data from different perspectives and altitudes captured by drone aerial photography;
[0012] The image sample data is used as a layout constraint and input into the multi-view diffusion model to generate the final aerial image dataset.
[0013] Preferably, the multi-view diffusion model is a fine-tuned SDXL large model; the SDXL large model includes a variational autoencoder and Unet, and the variational autoencoder includes an encoder and a decoder;
[0014] The multi-view diffusion model encodes the input two-dimensional example and the transformation matrix of the proposed new viewpoint through a pre-trained CLIP model, and then concatenates them into multi-view generation constraints; subsequently,
[0015] The generated multi-view generation constraints are embedded into the SDXL large model using a cross-attention mechanism to generate two-dimensional images of two-dimensional instances from N viewpoints.
[0016] Preferably, the step of "embedding the generated multi-view generation constraints into the SDXL large model using a cross-attention mechanism" includes:
[0017] The multi-view generation constraints are embedded into the SDXL large model, the multi-view diffusion model, using the cross-attention mechanism as shown in the following formula:
[0018]
[0019] Where Q, K, and V represent the three key values of the hybrid attention mechanism, and W... Q W K W VThese represent the preset mapping matrices, U(·) represents the Unet of the SDXL large model, c = clip(X,R,T) represents the multi-view generation constraints, X represents a two-dimensional instance, (R,T) represents the change matrix to generate a new viewpoint, d represents the dimension of the K key value matrix, Attention(·) represents the hybrid attention mechanism, softmax(·) represents the softmax function, and T represents the transpose symbol.
[0020] Preferably, the SDXL large model generates two-dimensional images of the two-dimensional instance from N viewpoints using the method shown in the following formula.
[0021]
[0022] in, This represents the 2D image of a 2D instance from the x-th viewpoint. This represents the SDXL large model, Z. t This represents the noise feature map generated by the encoder at step t of the SDXL large model, and... Z0 represents the noise amplitude hyperparameter, Z0 represents the latent space intermediate vector obtained by downsampling X by the encoder of the SDXL large model module, and ε represents the noise amplitude hyperparameter. t This represents the noise graph at step t.
[0023] Preferably, the fine-tuning training loss function L of the multi-view diffusion model is:
[0024]
[0025] in, Let ε represent the average denoising optimization error across the entire training data distribution. θ This represents the noise predicted by Unet in the SDXL large model at step t.
[0026] Preferably, the optimization formula for the output image of the multi-view 3D reconstruction model is as follows:
[0027]
[0028] Where, x π Represents the generated 3D entity, and represents the PAAS reconstruction score of the multi-view 3D reconstruction model. For gradient operators, The parameter is The probability distribution.
[0029] Preferably, the image segmentation model is a text-guided instance segmentation model;
[0030] The instance segmentation model consists of the zero-shot target detection model Grounding DINO and the zero-shot segmentation model SAM.
[0031] The foreground portion includes pedestrians, cyclists, and vehicles; the background portion includes trees, buildings, and infrastructure.
[0032] Preferably, the step of "designing different scene scripts and reconstructing the scene based on the 3D entity" is as follows:
[0033] Based on the layout in the two-dimensional image, the three-dimensional entity is embedded into the multi-view three-dimensional reconstruction model, and the three-dimensional entity is repaired and optimized; different scene scripts are designed to reconstruct the scene: for the overall scene, different time periods of lighting and weather conditions are designed.
[0034] Preferably, the planar perspective data includes image data generated from publicly available datasets and raw image models; the raw image models include text-based raw image models and image-based raw image models.
[0035] Preferably, the aerial image dataset is used to generate video data using a large image-to-video model.
[0036] The advantages of this invention are:
[0037] The present invention provides a method for generating controllable multi-view aerial photography data based on a large 3D model, which can convert planar view shooting data into UAV aerial photography data, maintain the layout consistency of the generated content during the view conversion process, and control the shooting angle and altitude of the generated data, ultimately achieving efficient aerial photography sample augmentation.
[0038] Furthermore, this invention designs a method for converting planar perspective image data into 3D data and reconstructing scenes. First, a text-guided image segmentation model is used to segment the foreground and background portions of the 2D image. Then, a multi-view diffusion model is used to sequentially convert various 2D instances into 3D entities. Next, various 3D entities are embedded into the multi-view 3D reconstruction model. Different scene scripts are designed, and different pedestrian poses, behavioral trajectories, vehicle flow lines, scene lighting, and weather conditions are added to complete the 3D scene reconstruction, achieving precise control over the generated data content.
[0039] Furthermore, this method allows for the setting of fine-grained shooting schemes based on 3D scenes, thereby collecting drone aerial image data covering different shooting angles and altitudes.
[0040] Furthermore, this method can also use a graph-based video model to generate aerial video data and simultaneously acquire video data with consistent content. Attached Figure Description
[0041] Figure 1 This is a schematic diagram of the main steps of a method for generating controllable multi-view aerial photography data based on a large 3D model based on a graph in this invention.
[0042] Figure 2 This is a schematic diagram of the model structure of an SDXL large model in this invention;
[0043] Figure 3 This is a schematic diagram of the structure of a multi-view diffusion model in this invention. Detailed Implementation
[0044] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0045] See Figure 1 , Figure 1 This is a schematic diagram illustrating the steps of a method for generating controllable multi-view aerial data based on a large-scale graph-generated 3D model. Figure 1 As shown in the figure, the method for generating controllable multi-view aerial data based on a large-scale 3D model provided in this embodiment includes:
[0046] Step S1: Obtain planar view data and use an image segmentation model to segment the foreground and background parts of the planar view data to form multiple two-dimensional instances.
[0047] Planar perspective data includes image data from publicly available datasets and image-generated models. Image-generated models include text-to-image and image-to-image large-scale models. Specifically, a portion of the planar perspective data can be collected from publicly available datasets, including autonomous driving datasets and city street datasets; another portion can be generated using text-to-image or image-to-image large-scale models. Specifically, text or image input can form a generation instruction (Prompt), guiding the Stable Diffusion XL (SDXL) large-scale model to generate images. This model is pre-trained on the large dataset LAION-5B, containing 5 billion text-image pairs. By fitting this dataset, the SDXL large-scale model aligns the control instructions (text and images) with the generated content, giving it powerful text-to-image and image-to-image generation capabilities. The model structure of the SDXL large-scale model is as follows: Figure 2As shown, the system consists of three parts: a pre-trained conditional embedding model, a variational autoencoder (VAE), and a denoising UNet. The conditional embedding model receives a text description and a constraint image to guide image generation. The text description is embedded into the denoising process using a pre-trained CLIP model via a cross-attention mechanism to guide image generation. The constraint image is encoded and embedded into the denoising process using a pre-trained T2I-Adapter or ControlNet to guide image generation. The variational autoencoder comprises an encoder and a decoder. The encoder downsamples a random noisy image to a low-dimensional latent space, then uses UNet for step-by-step denoising based on embedding conditions. Finally, the decoder upsamples the denoised latent space vector to generate planar viewpoint data.
[0048] After obtaining a large amount of planar viewpoint data, an image segmentation model is used to segment the foreground and background portions of the planar viewpoint data to form multiple 2D instances. The image segmentation model is a text-guided instance segmentation model. It segments the foreground and background portions of the planar viewpoint image using text guidance to form different 2D instances. This image segmentation model is a grounded-SAM instance segmentation model, composed of the zero-shot object detection model Grounding DINO and the zero-shot segmentation model SAM. First, the category label to be segmented is input as the text guidance word, and Grounding DINO is used to obtain bounding box information containing the text prompt. Then, the image and bounding box information are input into the SAM model to obtain the segmentation mask of the specified portion, segmenting different target entities. The foreground portion includes pedestrians, cyclists, vehicles, etc.; the background portion includes trees, buildings, infrastructure, etc. The segmented foreground and background portions are numbered according to category, and a 2D instance table is formed for subsequent 3D reconstruction.
[0049] Step S2: Generate two-dimensional images from N perspectives based on a multi-view diffusion model and multiple two-dimensional instances.
[0050] First, the 2D instances extracted in the previous steps are preprocessed to optimize boundary information and remove background information. The optimized 2D instances are then input into a multi-view diffusion generation model to predict image samples of the 2D instances from different viewpoints. For example... Figure 3As shown, the multi-view diffusion model is a fine-tuned SDXL large model. The SDXL large model includes a variational autoencoder and Unet, with the variational autoencoder comprising an encoder and a decoder. The multi-view diffusion model encodes the input 2D example X and the transformation matrix (R,T) of the proposed new viewpoints using a pre-trained CLIP model, and concatenates them into multi-view generation constraints. Then, a cross-attention mechanism is used to embed the generated multi-view generation constraints into the SDXL large model to generate 2D images of the 2D instance from N viewpoints.
[0051] Let c represent the multi-view generation constraint:
[0052] c = clip(X, R, T)
[0053] A cross-attention mechanism is used to establish the association between the multi-view generation constraints and the generated image modality, thus completing the constraint embedding, as shown in the following equation:
[0054]
[0055] Q, K, and V represent the three key values of the hybrid attention mechanism, respectively, and W... Q W K W V These represent the preset mapping matrices, U(·) represents the Unet of the SDXL large model, c = clip(X,R,T) represents the multi-view generation constraints, X represents a two-dimensional instance, (R,T) represents the change matrix to generate a new viewpoint, d represents the dimension of the K key value matrix, Attention(·) represents the hybrid attention mechanism, softmax(·) represents the softmax function, and T represents the transpose symbol.
[0056] SDXL large model generates two-dimensional instances of images under N new viewpoints. The generation method is as follows:
[0057]
[0058] in, This represents the 2D image of a 2D instance from the x-th viewpoint. This represents the SDXL large model, Z. t This represents the noise feature map generated by the encoder at step t of the SDXL large model, and... Z0 represents the noise amplitude hyperparameter, Z0 represents the latent space intermediate vector obtained by the encoder of the SDXL large model by downsampling X, and ε represents the noise amplitude hyperparameter. t This represents the noise graph at step t.
[0059] Fine-tuning training loss function L for the multi-view diffusion model:
[0060]
[0061] in, Let ε represent the average denoising optimization error across the entire training data distribution. θ This represents the noise predicted by Unet in the SDXL large model at step t. During fine-tuning training, this formula is first used to add noise to Z0 to obtain the ground truth noise value at each step. During denoising, the initial noise feature map Z is first sampled from an arbitrary Gaussian distribution. T Then, in each denoising step, Unet is used to generate predicted noise with c as the constraint, and then Z is used... T Subtract ε θ Proceed to the next denoising step. After T rounds of denoising, a low-dimensional feature map free of noise is obtained. The image is sent to the decoder module and output as the final generated image. The weights of the encoder / decoder and Unet components in the SDXL large model will be updated according to the above loss function during the denoising process.
[0062] Step S4: Based on the multi-view 3D reconstruction model, generate 3D entities corresponding to the foreground and background parts according to the 2D image.
[0063] After fine-tuning the SDXL large model, 2D images of 2D instances from N viewpoints are generated. Based on this, a pre-trained multi-view 3D reconstruction model, Score Jacobian Chaining (SJC), is used to transform the 2D instances from 2D images into 3D entities. This method uses a neural network to establish a latent function representation of 3D objects, uses multi-view images as input to the neural network for volumetric rendering, and finally generates 3D entities from the 2D images of the objects. During the training process, rendered images of the 3D entities from different viewpoints are randomly sampled and generated. Gaussian noise is then added to the images, and denoising is performed based on the input images and multi-view generation constraints. The optimization formula for the output rendered image is as follows:
[0064]
[0065] Where, x π Represents the generated 3D entity, and represents the PAAS reconstruction score of the multi-view 3D reconstruction model. For gradient operators, The parameter is The probability distribution.
[0066] Step S4: Design different scene scripts and reconstruct the scene based on the 3D entity.
[0067] After completing the 3D entity reconstruction, based on the layout in the 2D image, the 3D entities are sequentially embedded into multi-view 3D reconstruction models such as Blender. In the multi-view 3D reconstruction models, imperfections in the 3D entity reconstruction are repaired and optimized, while target textures and other elements are also optimized. Then, different scene scripts are designed to reconstruct the scene: for movable foreground elements, such as pedestrians and vehicles, skeletons are added using Blender software, and specific character poses, movement trajectories, and vehicle streamlines are designed; for the overall scene, different lighting conditions and weather conditions at different times are designed.
[0068] Step S5: Based on the reconstructed scene, obtain image sample data from different perspectives and altitudes captured by the drone.
[0069] Based on the preceding scenario, a drone aerial photography trajectory was set in the multi-view 3D reconstruction model. Specifically, along the central axis of the 3D scene, shooting was conducted starting at a height of 5 meters above the horizontal plane, gradually increasing in increments of 10 meters to 1000 meters. At each height interval, starting from a 0-degree overhead viewpoint, the angle of attack gradually increased in increments of 30 degrees to 90 degrees. For each shooting viewpoint, a 360-degree surround shot was performed starting from 0 degrees and increasing in increments of 30 degrees, collecting image data from different viewpoints and surround angles in the 3D scene to obtain multi-view drone aerial image sample data.
[0070] Step S6: Input the image sample data as layout constraints into the multi-view diffusion model to generate the final aerial image dataset.
[0071] After generating multi-view sample image data from drone aerial photography based on a specified shooting angle and altitude, the captured image data is used as a layout constraint input to the multi-view diffusion model for further data augmentation. Simultaneously, a large image-to-video model is used to generate video data with consistent content, ultimately completing the construction of the aerial image / video dataset.
[0072] In summary, the aerial photography controllable multi-view data generation method based on image-generated 3D large models provided by this invention can convert planar view shooting data into UAV aerial photography data while maintaining the consistency of the generated content. It can also control the shooting angle and altitude of the generated data, ultimately achieving efficient aerial photography sample augmentation.
[0073] Furthermore, this invention designs a method for converting planar perspective image data into 3D data and reconstructing scenes. First, a text-guided image segmentation model is used to segment the foreground and background portions of the 2D image. Then, a multi-view diffusion model is used to sequentially convert various 2D instances into 3D entities. Next, various 3D entities are embedded into the multi-view 3D reconstruction model. Different scene scripts are designed, and different pedestrian poses, behavioral trajectories, vehicle flow lines, scene lighting, and weather conditions are added to complete the 3D scene reconstruction, achieving precise control over the layout content of the generated data.
[0074] Furthermore, this method allows for the setting of fine-grained shooting schemes based on 3D scenes, thereby collecting drone aerial image data covering different shooting angles and altitudes.
[0075] Furthermore, this method can also use a graph-based video model to generate aerial video data and simultaneously acquire video data with consistent content.
[0076] Those skilled in the art will recognize that the method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.
[0077] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a parameter, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to those elements, method, or apparatus.
[0078] The above description describes the preferred embodiments of the present invention and the technical principles applied thereto. For those skilled in the art, any obvious changes such as equivalent transformations or simple substitutions based on the technical solutions of the present invention, without departing from the spirit and scope of the present invention, shall fall within the protection scope of the present invention.
Claims
1. A method for generating controllable multi-view aerial photography data based on a large-scale 3D model derived from images, characterized in that, The method includes: Acquire planar viewpoint data and use an image segmentation model to segment the foreground and background portions of the planar viewpoint data to form multiple two-dimensional instances; Based on a multi-view diffusion model, N two-dimensional images are generated from the multiple two-dimensional instances; Based on a multi-view 3D reconstruction model, 3D entities corresponding to the foreground and background parts are generated according to the 2D image; Design different scene scripts and reconstruct the scene based on the three-dimensional entity; Based on the reconstructed scene, obtain image sample data from different perspectives and altitudes captured by drone aerial photography; The image sample data is used as a layout constraint and input into the multi-view diffusion model to generate the final aerial image dataset.
2. The method for generating controllable multi-view aerial data based on a large-scale 3D model as described in claim 1, characterized in that, The multi-view diffusion model is a fine-tuned SDXL large model; the SDXL large model includes a variational autoencoder and Unet, and the variational autoencoder includes an encoder and a decoder. The multi-view diffusion model encodes the input two-dimensional example and the change matrix of the proposed new view through a pre-trained CLIP model, and then concatenates them into multi-view generation constraints. after, The generated multi-view generation constraints are embedded into the SDXL large model using a cross-attention mechanism to generate two-dimensional images of two-dimensional instances from N viewpoints.
3. The method for generating controllable multi-view aerial data based on a large-scale 3D model as described in claim 2, characterized in that, The steps of "embedding the generated multi-view constraints into the SDXL large model using a cross-attention mechanism" include: The generated multi-view constraints are embedded into the SDXL large model using the cross-attention mechanism as shown in the following formula: Where Q, K, and V represent the three key values of the hybrid attention mechanism, and W... Q W K W V These represent the preset mapping matrices, U(·) represents the Unet of the SDXL large model, c = clip(X,R,T) represents the multi-view generation constraints, X represents a two-dimensional instance, (R,T) represents the change matrix to generate a new viewpoint, d represents the dimension of the K key value matrix, Attention(·) represents the hybrid attention mechanism, softmax(·) represents the softmax function, and T represents the transpose symbol.
4. The method for generating controllable multi-view aerial data based on a large-scale 3D model as described in claim 3, characterized in that, The SDXL large model generates two-dimensional images of a two-dimensional instance from N perspectives using the method shown in the formula. in, This represents the 2D image of a 2D instance from the x-th viewpoint. This refers to the SDXL large model, Z. t This represents the noise feature map generated by the encoder at step t of the SDXL large model, and... Z0 represents the noise amplitude hyperparameter, Z0 represents the latent space intermediate vector obtained by downsampling X by the encoder of the SDXL large model module, and ε represents the noise amplitude hyperparameter. t This represents the noise graph at step t.
5. The method for generating controllable multi-view aerial data based on a large-scale 3D model as described in claim 4, characterized in that, The fine-tuning training loss function L of the multi-view diffusion model: in, Let ε represent the average denoising optimization error across the entire training data distribution. θ This represents the noise predicted by Unet in the SDXL large model at step t.
6. The method for generating controllable multi-view aerial data based on a graph-generated 3D large model as described in claim 1, characterized in that, The optimization formula for the output image of the multi-view 3D reconstruction model is as follows: Where, x π Represents the generated 3D entity. The PAAS reconstruction score represents the multi-view 3D reconstruction model. For gradient operators, The parameter is The probability distribution.
7. The method for generating controllable multi-view aerial data based on a large-scale 3D model as described in claim 1, characterized in that, The image segmentation model is a text-guided instance segmentation model; The instance segmentation model consists of the zero-shot target detection model Grounding DINO and the zero-shot segmentation model SAM; The foreground portion includes pedestrians, cyclists, and vehicles; The background includes trees, buildings, and infrastructure.
8. The method for generating controllable multi-view aerial data based on a graph-generated 3D large model as described in claim 1, characterized in that, The steps of "designing different scene scripts and reconstructing the scene based on the 3D entity" are as follows: Based on the layout in the two-dimensional image, the three-dimensional entity is embedded into the multi-view three-dimensional reconstruction model, and the three-dimensional entity is repaired and optimized; different scene scripts are designed to reconstruct the scene: for the overall scene, different time periods of lighting and weather conditions are designed.
9. The method for generating controllable multi-view aerial data based on a large-scale 3D model as described in claim 1, characterized in that, The planar perspective data includes image data generated from publicly available datasets and raw image models; the raw image models include text-based raw image models and image-based raw image models.
10. The method for generating controllable multi-view aerial data based on a large-scale 3D model as described in claim 1, characterized in that, The aerial image dataset is used to generate video data using a large image-to-video model.