A multimodal collaboratively driven DiT urban layout generation method

Abstract

This paper lies at the intersection of computer vision and urban design. It proposes a multimodal collaboratively driven DiT urban layout generation method to address the shortcomings of existing generation models in terms of spatial rationality and planning controllability. This method innovatively constructs a transformer-based diffusion generation model and integrates two additional control signals into the DiT model to achieve controlled generation. In terms of model architecture design, a DiT framework with multimodal conditional fusion is constructed, integrating a dual guidance mechanism of road network sketches (image control) and planning semantics (text control). In terms of text modality, a dynamic semantic fusion module based on a pre-trained language model is designed to deeply embed textual information into the generation process. In terms of image control, a hybrid attention regulation mechanism is proposed, combining cross-attention with AdaLN technology to achieve multi-scale fusion of road network structural features and address the problem of spatial layout distortion. Experiments show that this method can effectively integrate text and image information, and the generated urban layout images are superior to existing methods.

Description

Technical Field

[0001] This invention belongs to the intersection of computer vision and urban design technology, and in particular relates to a multimodal collaboratively driven DiT urban layout generation method, which aims to use advanced model architecture to achieve diversified, style-controllable and realistic urban layout generation. Background Art

[0002] Urban layout generation has become a key research frontier in computer vision and computational design, and has made significant contributions to the advancement of urban morphological pattern learning and urban layout planning. Currently, research on urban layout morphological patterns mainly focuses on three technical paths, each of which addresses different challenges encountered in urban layout generation research. (1) Using satellite and aerial imagery for geospatial reconstruction and deep segmentation networks, but facing difficulties such as insufficient image resolution, content occlusion and segmentation accuracy. (2) Procedural modeling frameworks generate hierarchical urban elements through shape grammars and L-systems to achieve reasonable road networks and vegetation distribution. However, they rely on specially formulated rules, such as space grammar parameters, which makes it too complicated to generate coordinated layouts. (3) Inverse procedural modeling attempts to use data-driven modeling of urban layouts, but often oversimplifies urban complexity into geometric priors, and is therefore unable to generate stylized urban layouts.

[0003] The increasing abundance of geospatial data and the development of generative artificial intelligence (AI) have provided new approaches for learning diverse urban morphological patterns and generating layouts. However, existing models often struggle to capture the inherent diversity of layout styles found in global urban structures. This diversity requires that next-generation generative models be able to learn the layout characteristics of different cities.

[0004] In addition, in terms of technical path, three basic challenges hinder the application of new generative models to the task of urban layout generation. First, due to the high cost of professional measurement, high-quality urban layout datasets are still scarce. Open source platforms such as OpenStreetMap (OSM) have geospatial asymmetry: developed areas provide high accuracy, while emerging cities are often represented using incomplete or schematic data, which limits the ability of existing models to capture real urban morphological patterns. Second, the mainstream model architecture has a convolutional inductive bias, which prioritizes local texture consistency rather than global spatial consistency, resulting in discontinuous roads and color drift in the generated plots. Third, the urban layout styles learned by existing model frameworks tend to be homogenized, failing to capture the diversity between different urban layouts and not supporting the generation of layout data guided by multimodal information. Summary of the Invention

[0005] To address the problems of insufficient dataset resources, inability of model architectures to effectively capture the diversity of urban styles, and poor spatial coherence of generated results in existing urban layout generation technologies, this paper proposes a multimodal collaboratively driven DiT urban layout generation method. This method aims to learn cross-city layout patterns from limited real-world datasets through innovative data processing methods and model architecture design, generating diverse, style-controlled, and structurally sound urban layouts. This method improves the quality and efficiency of urban layout generation, filling the gaps in the application of stable diffusion models and Transformer models in urban planning, design, and related research fields.

[0006] This paper treats urban layout styles as learnable multimodal annotations and embeds them into a Diffusion Transformer (DiT) model. This enables our framework to capture the structural elements and stylistic nuances of diverse urban layouts. To train our model, we curate a large-scale dataset of real urban layouts containing images of urban layouts and their stylistic descriptions. We propose a comprehensive multi-scale data processing workflow and corresponding model training framework, leveraging the diffusion model with the Transformer architecture to generate diverse urban layouts.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A multimodal collaboratively driven DiT city layout generation method includes the following steps:

[0009] Step S101: Acquisition and preprocessing of training data: acquiring training sample data from the open source data platform OSM, performing data preprocessing and enhancement, and constructing a spatial database;

[0010] Step S102: construct a three-level spatial pyramid to perform multi-scale visualization of the data obtained in step S101;

[0011] Step S103, performing city layout classification and semantic information annotation on the visualization result in step S102;

[0012] Step S104: introducing a VAE (Variational Autoencoder) perceptual compression model trained on a large-scale dataset as a pre-training model;

[0013] Step S105, using the pre-trained model introduced in step S104 to compress the sample instance obtained in step S102 in the feature space, transferring the original sample instance from the image space to the latent space, and obtaining a training sample in the latent space;

[0014] Step S106, inputting the sample semantic annotation information in step S103 into a semantic information fusion module to extract semantic features;

[0015] Step S107: Introduce the large-scale visual model DINOv2 trained on a large-scale dataset as an image feature encoding module to obtain the encoded image guidance information features as a strong conditional signal to guide the model generation process;

[0016] Step S108, adding Gaussian noise to the training sample in step S105 through a forward diffusion process to obtain a noisy training sample;

[0017] Step S109: Fusing the semantic feature information, image guidance information features, noisy training samples, and the temporal dynamic characteristics of the diffusion process, using the Diffusion Transformer-based DiT generative model to train the diffusion model and output the predicted noise;

[0018] Step S110 , inverse diffusion is performed to generate a predicted noise image, and a final generated layout image is obtained. The mean square error between the predicted noise and the actually added noise is calculated, and the losses of multiple samples are weighted averaged.

[0019] This technical solution is further optimized. In step S102, a three-level spatial pyramid is constructed, and the layout data obtained in step S101 is layered and saved as image sample data according to a macro-scale radius of 600m, a meso-scale radius of 400m, and a micro-scale radius of 200m.

[0020] This technical solution is further optimized, and the VAE architecture is used in step S105 to achieve eight times space compression.

[0021] This technical solution is further optimized. In terms of optimizing the semantic information fusion mechanism, step S106 adopts the Flan-T5-base model as the semantic encoding core, maps the text model semantic information into a 1024-dimensional text feature vector, and performs linear mapping and GeLU activation processing.

[0022] This technical solution is further optimized. In step S107, the self-supervised pre-trained DINOv2 visual large model is introduced into the urban layout generation task for the first time, and strong semantic guidance is constructed through multi-scale feature extraction and hierarchical fusion strategy.

[0023] The technical solution is further optimized, and the step S108 is specifically as follows: perform a forward diffusion process on N training samples in the latent space. By mixing linear interpolation with random noise, the noisy latent variable sequence is generated as The maximum number of diffusion steps; specifically, the time step is sampled independently for each training sample , add noise according to the following formula:

[0024]

[0025] in, , diffusion noise scheduling coefficient , . Loop variable Indicates from 1 to Each step is used to calculate the cumulative coefficient The intermediate time step index of . Represents the time from time step 1 to the current step every intermediate step. represents a random Gaussian noise, Indicates that the mean is 0 and the covariance is the unit matrix The standard normal distribution of represents the random Gaussian noise added Characteristics of data that follow a standard normal distribution. represents a decreasing noise schedule sequence, represent The amount of noise increased step by step.

[0026] This technical solution is further optimized. The DiT generation model in step S109 forms a deep network architecture by stacking 28 Transformer modules. Each Transformer module gradually refines the multi-level features of the image through the self-attention mechanism and the feedforward network, and performs adaptive instance normalization AdaLN-Zero on the semantic feature information in step S106.

[0027] This technical solution is further optimized. During the training process of the multimodal DiT city layout generation model, the probability Randomly discard conditional information and use the classifier's free bootstrap loss The constrained multimodal DiT urban layout generation model simultaneously learns conditional and unconditional generation capabilities. The specific calculation method is as follows:

[0028]

[0029] in, is a binary indicator function, Indicates conditional information. Indicates that there is no condition information. When there is condition information hour, =1, when the condition information is discarded, is the noisy output predicted by the DiT model, is random noise sampled from a standard Gaussian distribution, Represents a variable Take expectations. Indicates that the model is under given conditions In the case of generate The noise added when the conditional generation loss The smaller it is, the better the model is at recovering the target image from noise under conditional guidance. Represents the prediction noise of the model without any conditions, unconditional generation loss The smaller the loss, the better the model is at restoring the image even without any hints.

[0030] This technical solution is further optimized. In step S110, a multi-step iterative denoising mechanism is used to generate the final layout image and calculate the mean square error loss. as follows:

[0031]

[0032] Among them, the expected symbol Represents the time step is a uniform distribution on the interval [1, T] Randomly sampled from Represents the original sample From the data distribution In the Indicates that the noise is obtained from a standard Gaussian distribution. It is the first The noisy samples of the step, , is the diffusion noise scheduling coefficient, It is the Flan-T5 semantic fusion vector and the DINOv2 visual guidance vector. represents standard Gaussian noise, is the model prediction output.

[0033] Different from the existing technology, the above technical solution has the following beneficial effects:

[0034] Data Advantages: The CityStyle-OSM dataset is large in scale, covers a wide range of cities, and is richly annotated. This provides sufficient and diverse data support for the urban layout generation model, helping the model learn more comprehensive and realistic urban layout patterns and improving the accuracy and diversity of the generated results.

[0035] Previous similar methods were all based on the GAN framework. This invention, for the first time, introduced the Diffusion model and Transformer architecture into the urban layout image generation work, and integrated the text information fusion module and the visual information attention guidance module into the DiT generation model. The generated urban layout is more structurally reasonable and visually coherent, and performs better than existing methods in terms of structural and visual consistency, demonstrating for the first time the potential of the DiT model in urban layout generation tasks.

[0036] Application value: The diverse and style-controlled urban layouts generated by this invention can provide urban planners and designers with rich design inspiration and reference solutions, helping them to carry out urban planning and design work more efficiently. At the same time, it provides a powerful tool for urban-related research, helping to promote research progress in areas such as urban development and urban morphology analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Flowchart of the DiT city layout generation method driven by multimodal collaboration. DETAILED DESCRIPTION

[0038] In order to explain the technical content, structural features, achieved objectives and effects of the technical solution in detail, the following is a detailed description in conjunction with specific embodiments and accompanying drawings.

[0039] like Figure 1 The figure shows a flow chart of a multi-modal collaboratively driven DiT city layout generation method. The city layout generation method of this embodiment specifically includes the following steps:

[0040] Step S101: Acquisition and preprocessing of training data.

[0041] Gets geographic information spatial features within a specified set of longitude and latitude from the OpenStreetMap (OSM) platform, including road data nodes, building shapes, longitude and latitude coordinates, and geographic names.

[0042] Step S102: Construct a three-level spatial pyramid to visualize the data acquired in step S101 at multiple scales. Visualize the geographic information features acquired within the latitude and longitude ranges at the macroscale (600m radius), mesoscale (400m radius), and microscale (200m radius) and save them as a set of N image data (1024×1024 resolution).

[0043] Step S103: Semantic information annotation is performed on the sample data, and the visualization results in step S102 are classified and annotated with semantic information. Based on the information such as the number of buildings, road length, building type, and geographical location obtained in step S101, the image samples in step S102 are classified and annotated with semantic information.

[0044] Step S104: introduce the VAE perceptual compression model trained on a large-scale dataset as a pre-training model.

[0045] Data-driven deep learning techniques require extensive training data to build large-scale models with strong generalization and robustness. This paper introduces a pre-trained variational autoencoder, which learns basic feature representations on the ImageNet-21K general vision dataset, as a plug-and-play perceptual compression module. Trained on the LAION-5B ultra-large-scale multimodal dataset (containing 5.85 billion image-text pairs), the model achieves 8x compression from the original image to the latent space through an 8-layer convolutional downsampling architecture (with a stride of 2 per layer). The residual block architecture and self-attention mechanism in the encoder and decoder accurately capture key geometric features of urban layouts, such as road topology and building outlines.

[0046] In step S105 , the pre-trained model introduced in step S104 is used to compress the sample instance obtained in step S102 in the feature space, and the original sample instance is transferred from the image space to the latent space to obtain a training sample in the latent space.

[0047] A VAE architecture is used to achieve eightfold spatial compression. Taking an original sample image of 512 pixels as an example, the encoder uses four convolutional layers with a stride of 2 to achieve spatial downsampling, compressing the original 512×512 pixels to a 64×64 latent space representation. The number of channels is increased to 512 dimensions. Residual connections and convolutional layers are introduced in the decoder to maintain reconstruction accuracy.

[0048] Based on the introduced pre-trained variational autoencoder, the original N city layout images (resolution 1024×1024 pixels, RGB three channels) are compressed into the latent space through four-level convolution downsampling operations with a stride of 2, generating N 128×128 dimensional four-channel latent variable representations. Specifically, the input image Via encoder The latent variable is obtained by nonlinear mapping At the same time, the multi-scale feature extraction capability obtained through pre-training is utilized to retain key spatial semantic information such as road connectivity, land shape, and building orientation, making it possible to efficiently train the subsequent DiT model in the latent space.

[0049] Step S106 , extracting features from the semantic information of the training samples, and inputting the sample semantic annotation information in step S103 into a semantic information fusion module for semantic feature extraction.

[0050] A semantic encoder is constructed based on the Flan-T5-base pre-trained language model. This encoder converts the N textual semantic information annotated in step S103 (including unstructured descriptions such as building type, geographic name, and road grade) into a fusible dense vector representation. To optimize the semantic information fusion mechanism, the Flan-T5-base model is used as the semantic encoding core. This maps the textual semantic information into a 1024-dimensional text feature vector, which is then linearly mapped and activated with GeLU.

[0051] In step S107, the large visual model DINOv2 trained on a large-scale dataset is introduced as an image feature encoding module to obtain the encoded image guidance information features as a strong conditional signal to guide the model generation process.

[0052] For the first time, the self-supervised pre-trained DINOv2 visual model was introduced to the urban layout generation task. Strong semantic guidance was constructed through multi-scale feature extraction and hierarchical fusion strategies. Low-level texture features (256 channels) at 128×128 resolution and high-level semantic features (1536 channels) at 32×32 resolution were extracted from the guidance information image. After feature compression using global average pooling and max pooling, a 1024-dimensional guidance vector was generated through a linear transformation.

[0053] Input road network image as visual guidance condition. Use DINOv2-giant pre-trained visual model as image guidance encoder and input reference image After 14×14 block processing using the ViT architecture, image information features are extracted:

[0054]

[0055] Global average pooling preserves spatial distribution. Global maximum pooling to highlight salient areas. It is a channel splicing operation. Represents low-level detail features such as road network texture and building edge details, Represents high-level semantic features such as spatial topological relationships and layout area distribution. Finally, the encoded visual guidance feature vector is obtained , To initialize the learnable weight matrix, the input features are linearly transformed. It is the bias vector, which adds a constant offset to the result after linear transformation.

[0056] Step S108 , adding noise to the training samples through a forward diffusion process, that is, adding Gaussian noise to the training samples in step S105 through a forward diffusion process to obtain noisy training samples.

[0057] Perform forward diffusion on N training samples in the latent space. Generate a noisy latent variable sequence by mixing linear interpolation with random noise ( =1000 is the maximum number of diffusion steps). Specifically, each training sample is independently sampled with time steps , add noise according to the following formula:

[0058]

[0059] in, , diffusion noise scheduling coefficient . Loop variable Indicates from 1 to Each step is used to calculate the cumulative coefficient The intermediate time step index of . Represents the time from time step 1 to the current step every intermediate step. represents a random Gaussian noise, Indicates that the mean is 0 and the covariance is the unit matrix The standard normal distribution of represents the random Gaussian noise added Characteristics of data that follow a standard normal distribution. represents a decreasing noise schedule sequence, represent The amount of noise increased step by step.

[0060] Step S109: Construct a Diffusion Transformer (DIT) generative model for training. The noisy samples, semantic feature information, and image guidance information are integrated with the temporal dynamic characteristics of the diffusion process. The Diffusion Transformer-based Diffusion Transformer generative model is used for diffusion model training and outputs the predicted noise.

[0061] The DiT generative model in step S109 is constructed by stacking 28 Transformer modules to form a deep network architecture. Each Transformer module gradually refines the multi-level features of the image through a self-attention mechanism and a feedforward network. Adaptive instance normalization (AdaLN-Zero) is performed on the semantic feature information in step S106.

[0062] A cross-attention layer information fusion layer is introduced to fuse the visual guidance vector generated in step S108, and multimodal conditional injection is achieved through the key-value projection matrix.

[0063] During the training process of the multimodal DiT city layout generation model, the probability Randomly discard conditional information and use the classifier's free bootstrap loss The constrained multimodal DiT urban layout generation model simultaneously learns conditional and unconditional generation capabilities. The specific calculation method is as follows:

[0064]

[0065] in, is a binary indicator function, Indicates conditional information. Indicates that there is no condition information. When there is condition information hour, =1, when the condition information is discarded, is the noisy output predicted by the DiT model, is random noise sampled from a standard Gaussian distribution, Represents a variable Take expectations. Indicates that the model is under given conditions In the case of generate The noise added when the conditional generation loss The smaller it is, the better the model is at recovering the target image from noise under conditional guidance. Represents the prediction noise of the model without any conditions, unconditional generation loss The smaller the loss, the better the model is at restoring the image even without any hints.

[0066] By stacking 28 Transformer modules, a DiT deep network architecture is formed. During the training phase, the noise latent variable in step S108 is , the semantic information vector in step S106 With time step Input DiT network to predict target noise and real noise Calculating MSE loss , using the AdamW optimizer ( ) to update the parameters, and the learning rate is kept at 10 -4 .

[0067] In the In the layer Transformer module, the latent space tensor after noise addition obtained in step S108 is input , the Patchify operation is divided into 16 × 16 block sequences (a total of 128 / 16 × 128 / 16 = 4096), each block is mapped to a 1152-dimensional feature through linear embedding, and the initial sequence is obtained , and superimposes learnable positional encodings. Each Transformer module gradually refines the multi-level features of the image through multi-head self-attention and feedforward networks, and performs adaptive instance normalization and cross-attention fusion operations.

[0068] In the In the Transformer module of the layer, adaptive instance normalization AdaLN-Zero is performed on the semantic feature information in step S106, and the fusion guidance semantic information is entered into the training framework. The specific operations are as follows:

[0069]

[0070] Then perform conditional normalization and self-attention:

[0071]

[0072]

[0073] in, It is 8-head self-attention.

[0074] In the In the layer Transformer module, the visual guidance vector generated in step 108 is Using the cross-attention layer information fusion layer, the visual guidance vector , after the key-value projection matrix , mapped to, , visual condition injection DiT generation model is realized by interacting with the current feature through cross attention. Calculate cross attention and superimpose residuals:

[0075]

[0076] Among them, the cross attention calculation method is:

[0077]

[0078] Key-value projection matrix Map the visual conditions to a 128-dimensional subspace, query matrix Process the current feature.

[0079] In the In the Transformer module of the layer, the cross-attention output is subjected to secondary normalization and MLP operation.

[0080] Step S110: Generate the layout image and calculate the mean square error loss The predicted noise image is generated by reverse diffusion, and the final generated layout image is obtained. The mean square error between the predicted noise and the actual added noise is calculated, and the loss of multiple samples is weighted averaged.

[0081] A multi-step iterative denoising mechanism is used to generate the final layout image and calculate the mean square error loss as follows:

[0082]

[0083] Among them, the expected symbol Represents the time step is a uniform distribution on the interval [1, T] Randomly sampled from Represents the original sample From the data distribution In the Indicates that the noise is obtained from a standard Gaussian distribution. It is the first The noisy samples of the step, , is the diffusion noise scheduling coefficient, It is the Flan-T5 semantic fusion vector and the DINOv2 visual guidance vector. represents standard Gaussian noise, is the model prediction output.

[0084] Output at layer 28 After linear projection, it is restored to , the generated city layout image is obtained through the VAE decoder:

[0085]

[0086] And calculate the prediction noise With real noise The mean square error :

[0087]

[0088] Under conditions of generating images at different resolutions, the proposed algorithm achieved superior overall results across seven common evaluation metrics compared to the current mainstream generative model frameworks based on generative adversarial networks (GANs) and stable diffusion (SDXL). These metrics are: FID (Fréchet Inception Distance), KID (Kernel Inception Distance), MMD (Maximum Mean Discrepancy), MSE (Mean Square Error), LPIPS (Learned Perceptual Image Patch Similarity), PSNR (Peak Signal-to-Noise Ratio), and SSIM (Structural Similarity Index Measure). Detailed results are shown in the table below:

[0089]

[0090] Among them, FID measures the distance between the generated image and the real image in feature space; lower values ​​indicate closer proximity to the real image. KID measures distributional discrepancy; lower values ​​indicate better results. MMD also measures distributional discrepancy; smaller values ​​indicate a closer distribution between the generated image and the real image. MSE calculates the mean squared error of the pixel errors in an image, which directly reflects the degree of image distortion; smaller values ​​indicate closer proximity to the original image. LPIPS measures image discrepancy from a perceptual perspective and is more consistent with human visual perception; lower values ​​indicate a more perceptually similar generated image to the real image. PSNR evaluates image quality based on the peak signal-to-noise ratio; higher values ​​indicate better image quality. SSIM evaluates images from the perspective of structural similarity; closer to 1 indicates a more similar structure between the generated image and the real image, indicating higher quality. Taking these evaluation metrics into consideration, the proposed algorithm demonstrates significant advantages over mainstream GAN and SDXL frameworks.

[0091] Among the currently known solutions to the urban layout generation task, this invention introduces the concept of embedding multimodal guidance conditions for the first time, supporting the guidance of urban layout generation by both visual and textual information. It integrates the text information fusion module and the visual information attention guidance module to form a Diffusion-based Time-of-Take (DiT) generation model framework that supports multimodal information guidance. This is the first time that the Diffusion model and the Transformer architecture have been introduced into urban layout image generation, providing a new approach to regenerating urban layouts from geospatial data and demonstrating the potential of the DiT model for urban layout generation tasks. Furthermore, this invention supports the generation of layout image data of varying resolutions and exhibits good generalization.

[0092] This invention has broad application prospects and high practical value. For example, the generated urban geospatial layout data can be used for spatial analysis, not just for providing layout images. The model can generate realistic urban forms in areas lacking architectural data. It can also assist planners and designers in their initial modeling needs for complex processes, such as simulating urban expansion or new regional development.

[0093] It should be noted that, in this document, relational terms such as first and second, etc., are used solely to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Furthermore, the terms "include," "comprise," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. Without further limitation, elements defined by the phrase "include..." or "comprising..." do not exclude the presence of additional elements in the process, method, article, or terminal device comprising the elements. Furthermore, in this document, "greater than," "less than," "exceeding," etc., are understood to exclude the number itself; "above," "below," "within," etc., are understood to include the number itself.

[0094] Although the above embodiments have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concepts. Therefore, the above descriptions are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of the present invention.

Claims

1. A multi-modal collaboratively driven DiT city layout generation method, characterized by: The following steps are involved: Step S101: Acquisition and preprocessing of training data: acquiring training sample data from the open source data platform OSM, performing data preprocessing and enhancement, and constructing a spatial database; Step S102: construct a three-level spatial pyramid to perform multi-scale visualization of the data obtained in step S101; Step S103, performing city layout classification and semantic information annotation on the visualization result in step S102; Step S104, introducing a variational autoencoder perceptual compression model trained on a large-scale dataset as a pre-training model; Step S105, using the pre-trained model introduced in step S104 to compress the sample instance obtained in step S102 in the feature space, transferring the original sample instance from the image space to the latent space, and obtaining a training sample in the latent space; Step S106, inputting the sample semantic information annotation in step S103 into the semantic information fusion module for semantic feature extraction; Step S107: Introduce the large-scale visual model DINOv2 trained on a large-scale dataset as an image feature encoding module to obtain the encoded image guidance information features as a strong conditional signal to guide the model generation process; Step S108, adding Gaussian noise to the training sample in step S105 through a forward diffusion process to obtain a noisy training sample; Step S109: Fusing the semantic feature information, image guidance information features, noisy training samples, and the temporal dynamic characteristics of the diffusion process, using the Diffusion Transformer-based DiT generative model to train the diffusion model and output the predicted noise; Step S110 , inverse diffusion is performed to generate a predicted noise image, and a final generated layout image is obtained. The mean square error between the predicted noise and the actually added noise is calculated, and the losses of multiple samples are weighted averaged.

2. The multimodal collaboratively driven DiT city layout generation method according to claim 1, characterized in that: In step S102, a three-level spatial pyramid is constructed, and the layout data obtained in step S101 is layered and saved as image sample data according to a macro-scale radius of 600m, a meso-scale radius of 400m, and a micro-scale radius of 200m.

3. The multimodal collaboratively driven DiT city layout generation method according to claim 1, characterized in that: In step S105, VAE architecture is used to achieve eight times space compression.

4. The multimodal collaboratively driven DiT city layout generation method according to claim 1, characterized in that: In terms of optimizing the semantic information fusion mechanism, step S106 adopts the Flan-T5-base model as the semantic encoding core, maps the text model semantic information into a 1024-dimensional text feature vector, and performs linear mapping and GeLU activation processing.

5. The multimodal collaboratively driven DiT city layout generation method according to claim 1, characterized in that: In step S107, the self-supervised pre-trained DINOv2 visual large model is introduced into the urban layout generation task for the first time, and strong semantic guidance is constructed through multi-scale feature extraction and hierarchical fusion strategy.

6. The multimodal collaboratively driven DiT city layout generation method according to claim 1, characterized in that: The step S108 is specifically as follows: perform a forward diffusion process on N training samples in the latent space, and transform the original latent variables Generate a noisy latent variable sequence by mixing linear interpolation with random noise , =1000 is the maximum number of diffusion steps; specifically, each training sample is sampled independently for the time step , add noise according to the following formula: in, , diffusion noise scheduling coefficient , loop variable Indicates from 1 to Each step is used to calculate the cumulative coefficient The intermediate time step index of Represents the time from time step 1 to the current step Each intermediate step, represents a random Gaussian noise, Indicates that the mean is 0 and the covariance is the unit matrix The standard normal distribution of represents the random Gaussian noise added The data characteristics obey the standard normal distribution. represents a decreasing noise schedule sequence, represent The amount of noise increased step by step.

7. The multimodal collaboratively driven DiT city layout generation method according to claim 1, characterized in that: The DiT generation model in step S109 forms a deep network architecture by stacking 28 Transformer modules. Each Transformer module gradually refines the multi-level features of the image through the self-attention mechanism and the feedforward network, and performs adaptive instance normalization AdaLN-Zero on the semantic feature information in step S106.

8. The multimodal collaboratively driven DiT city layout generation method according to claim 1, characterized in that: In the step S109, the Diffusion Transformer-based DiT generation model is used to train the diffusion model. Randomly discard conditional information and use the classifier's free bootstrap loss The constrained multimodal DiT urban layout generation model simultaneously learns conditional and unconditional generation capabilities. The specific calculation method is as follows: in, is a binary indicator function, Indicates conditional information. Indicates that there is no conditional information. When there is conditional information hour, , when the condition information is discarded, is the noisy output predicted by the DiT model, is random noise sampled from a standard Gaussian distribution, Represents a variable Take expectations, Indicates that the model is under given conditions In the case of generate The noise added when the conditional generation loss The smaller it is, the better the model is at recovering the target image from noise under conditional guidance. Represents the prediction noise of the model without any conditions, unconditional generation loss The smaller the loss, the better the model is at restoring the image even without any hints.

9. The multimodal collaboratively driven DiT city layout generation method according to claim 1, characterized in that: In step S110, a multi-step iterative denoising mechanism is used to generate the final layout image and calculate the mean square error loss as follows: Among them, the expected symbol Represents the time step is a uniform distribution on the interval [1, T] Randomly sampled from Represents the original sample From the data distribution In the Indicates that the noise is obtained from a standard Gaussian distribution, It is the first The noisy samples of the step, , is the diffusion noise scheduling coefficient, It is the Flan-T5 semantic fusion vector and the DINOv2 visual guidance vector. represents standard Gaussian noise, is the model prediction output.

Images