Method and system for generating building planning class plane based on potential diffusion model

Abstract

The invention discloses a potential diffusion model-based building planning type plane generation method and system, and the method comprises the steps: collecting a land use planning map, and the total plan layout data and plane layout data of a building / building group, and constructing a corresponding data set; a pre-trained potential diffusion model is used as a base model, low-rank adaptation fine tuning training is carried out, and a low-rank adaptation model is obtained; and based on the pre-trained potential diffusion model, loading a low-rank adaptation model, inputting the sample data into the model, and generating a corresponding general plan layout / plane layout scheme of the building / building group. The method provided by the invention is small in data demand, flexible in data processing and high in generation speed, so that a reasonable planar graph or general graph is generated according to a newly input contour or partial content during reasoning; the problems of low design efficiency, dependence on personal experience, low flexibility, high cost, limited design result and the like of the traditional land planning, building and building group total plane layout and building plane generation method are solved.

Description

TECHNICAL FIELD

[0002] The present application relates to the technical field of the intersection of artificial intelligence generated content (AIGC) and computer aided design (CAD), and in particular to a method and system for generating architectural planning type planes based on a latent diffusion model. BACKGROUND

[0004] In the field of architecture and planning design, internal plane layout according to a given area boundary, land red line or building contour (i.e. the external boundary of a building) is the core and basic work in the scheme design stage, which involves comprehensive consideration of land use nature, building function partition, streamline organization, space utilization and environment creation, etc., to form a specific building arrangement scheme, such as land use planning map, building (group) general plane layout, building plan, etc.

[0005] The existing land use planning, building (group) general plane layout and building plan design methods mainly include the following:

[0006] 1) Traditional manual design and drawing method, which mainly relies on manual drawing or parametric modeling, and the whole process is time-consuming and laborious. This method mainly has the following problems: 1. Manual method is low in efficiency and depends on experience, and repeated modification; 2. The flexibility of parametric tool is insufficient, and it is difficult to meet the needs of complex building design, especially in handling large-scale building groups or layout design under special terrain conditions, the effect is not ideal; 3. The design result is limited, and it is difficult to produce a large number of alternative schemes in a short time for evaluation and selection.

[0007] 2) Parametric automatic generation method, which encodes design rules into algorithms to automatically generate schemes by adjusting parameters. The core is "rule driven". This method mainly has the following problems: 1. Rule making is difficult and rigid: it is extremely challenging to convert complex and ambiguous architectural design knowledge into precise and complete computer rules. 2. The quality of the scheme completely depends on the quality of the preset rules: if the rules are not considered, the generated scheme may meet the rules but be actually unreasonable or unusable. 3. Cannot learn implicit knowledge: parametric generation cannot learn the implicit experience in excellent cases, such as "design sense", aesthetic principles and "design sense", which are difficult to express. The generated results are often mechanical and rigid, lacking in humanistic atmosphere. 4. High upfront cost: developing a reliable parametric generation system for each new project or building type requires a large amount of upfront time and expert knowledge investment, and has poor universality.

[0008] 3) Generative Adversarial Network (GAN) based method, which usually contains a generator and a discriminator. The generator takes the building outline map as the conditional input and tries to generate a realistic building plan; the discriminator tries to distinguish the generated plan from the plan in the real data set. Both are trained through adversarial game, and the ultimate goal is to make the generator produce high-quality plans. The main problems of this method are as follows: 1. The training process is unstable and prone to mode collapse: the training of GAN relies on the dynamic game between the generator and the discriminator, which is difficult to achieve the ideal balance state (Nash equilibrium). 2. The quality of generated images is difficult to control, and artifacts and unreasonable structures are prone to appear: the learning goal of GAN is to indirectly optimize the generation quality by cheating the discriminator, rather than directly minimizing the difference with the real data. 3. It is difficult to evaluate the training state and convergence: since the loss function cannot directly reflect the generation quality (the decrease of generator loss may only mean that it successfully deceived the discriminator, rather than quality improvement), it is difficult for developers to judge whether the model is well trained and when to stop training.

[0009] 4) Variational Autoencoder (VAE) based method, which encodes the plan through the encoder to the latent space, and then decodes and reconstructs through the decoder combined with the outline condition to generate new plan. The main problems of this method are as follows: 1. The generated image is blurred and lacks details: the loss function of VAE contains a reconstruction loss and a regularization term, and the reconstruction loss usually uses mean square error, which tends to punish the pixel-level difference with the target image, which encourages the model to output the "average" of all possible outputs, resulting in too smooth results. 2. The generation diversity is limited and too conservative: due to strong regularization constraints, VAE tends to generate conservative results similar to the training data. 3. Challenges in generating discrete structures: some elements in building plan (such as the exact position and orientation of doors and windows) have discrete nature. VAE as a continuous model in nature may not be accurate enough when dealing with such discrete structures, resulting in "ambiguous" door and window positions.

[0010] 5) Direct fine-tuning of large pre-trained diffusion models, which directly obtains a large pre-trained diffusion model (such as Stable Diffusion) on general images (such as natural images), and fine-tunes all parameters of the entire model using an architectural contour-planar map paired dataset to adapt the model to the architectural generation field. This method mainly has the following problems: 1. Huge consumption of computing resources: large diffusion models have a huge number of parameters (usually up to billions), and fine-tuning all parameters requires extremely high GPU memory and a long training time, which is difficult to bear. 2. High risk of overfitting: the scale of high-quality datasets in the architectural field is usually limited, and fine-tuning a large-scale model with full parameters is prone to overfitting on small datasets, resulting in poor generalization ability of the model and poor generation effect when facing new contours outside the training set. High storage and deployment costs: fine-tuning a complete model for each specific architectural type or style will generate a huge model file (usually several GB), making it extremely inconvenient to store and deploy multiple such models.

[0011] To solve the problems in the prior art, the present application provides a building planning type planar generation method and system based on a latent diffusion model.

[0012] The information disclosed in this Background section is intended only to enhance understanding of the general background of the present application, and should not be considered to be an acknowledgement or any form of suggestion that this information forms the prior art that is already known to those skilled in the art. SUMMARY

[0014] The present application aims to provide a building planning type planar generation method and system based on a latent diffusion model. The present application performs low-rank adaption (LoRA, low-rank adaption) fine-tuning on a pre-trained diffusion model by performing custom "color-semantic mapping" functional labeling and data augmentation on a small number of architectural images, so that the model can quickly learn the relationship between "contours" and "layout" ("content"), and the model can efficiently generate and complete land use planning maps, architectural (group) overall layout, architectural plan, and architectural local plan. This method has small data requirements, flexible data processing, and fast generation, so that during inference, a reasonable plan or overall map can be generated according to new input contours or partial content, and can be applied to land use planning maps, overall layout design of buildings or building groups, architectural plan design, and local plan design and completion, to solve the technical problems of low design efficiency, reliance on personal experience, low flexibility, high cost, and limited design results in traditional land use planning, architectural and building group overall layout, and architectural plan generation methods.

[0015] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0016] The application provides a method for generating a building planning type plane based on a latent diffusion model, comprising the following steps:

[0017] S1, collecting land use planning map data, general plan layout data and plane layout data of a building / building group, and constructing a corresponding data set;

[0018] S2, using a pre-trained latent diffusion model as a base model, performing low-rank adaptive fine-tuning training to obtain a low-rank adaptive model;

[0019] S3, based on the pre-trained latent diffusion model, loading the low-rank adaptive model, inputting sample data into the model, and generating a corresponding general plan layout / plane layout scheme of the building / building group.

[0020] Further, the step S1 of constructing the corresponding data set specifically comprises the following steps:

[0021] S11, data standardization: standardizing the collected data, unifying the scale of the original pictures, so that all pictures are under a unified scale, and unifying the size and resolution of the original pictures to adapt to the latent diffusion model;

[0022] S12, establishing a function and color mapping rule: without relying on an existing general semantic segmentation system, but defining the corresponding relationship between color and function by experts according to the professional requirements of the field of architectural design / planning, customizing the "color-semantic" mapping rule according to the design requirements, and this customized rule directly encodes professional knowledge;

[0023] Through this "color-semantic" mapping, the architectural design and planning task is converted into an image-semantic alignment task; in the conditional input stage of the diffusion model, the model can identify and learn the functional attributes represented by different color regions, thereby ensuring the generated results to have explainability and functional consistency;

[0024] During training, the color-semantic mapping is directly converted into an additional regularization term to improve the functional consistency;

[0025] Let the data set input image be , the semantic set be , and the color set be , wherein each corresponds to a functional semantic ;

[0026] Define a mapping function:

[0027] ;

[0028] During data preprocessing, the pixel color of the input image is Convert to functional semantic labels: ;

[0029] Final label matrix As conditional input or regular constraints;

[0030] S13, data labeling: according to the self-defined rules, the functional areas in each image are accurately labeled, so that they match the color labels at the pixel level;

[0031] S14, data augmentation: apply horizontal flip, vertical flip, and random rotation geometric transformation to the labeled images to enrich the training samples and reduce model overfitting;

[0032] S15, generate text description: based on the labeling results, generate a corresponding text file for each image, describing the functional partition and its position, area, and area ratio information, thereby constructing a picture-text pair dataset.

[0033] Further, in the specific operation of step S12, the correspondence between each functional area and color is set as follows: for land use planning maps, yellow (255, 255, 0) represents residential land, red (255, 0, 0) represents commercial land, and green (0, 77, 0) represents sports land; for the building / complex general layout plan in the implementation case, green (0, 255, 0) represents playground, red (255, 0, 0) represents administrative building, and blue (0, 0, 255) represents teaching building; for building plan, yellow (255, 255, 0) represents guest elevator area, olive green (177, 176, 70) represents fire escape, blue (0, 0, 255) represents stairwell, and gray (115, 127, 123) represents equipment room, and naming specifications are developed; these are just examples, and the principle is to customize the functional area and color correspondence.

[0034] Further, step S2 specifically includes the following steps:

[0035] S21, load and freeze the pre-trained latent diffusion model:

[0036] Select a diffusion model pre-trained on a large-scale general image dataset as the base model, load the variational autoencoder (VAE), text encoder (CLIP), and UNet from the pre-trained latent diffusion model, and fix their weights to ensure they do not update during fine-tuning;

[0037] S22, construct the latent representation of the training data:

[0038] Encode the pictures of the dataset prepared in step S1 into latent vectors through VAE ; Textual description corresponding to each image, converted to conditional vector by text encoder ; Since the latent vector of each image is reused during the training process, it is selected to pre-cache these vectors to speed up subsequent iterations;

[0039] S23, inject LoRA module:

[0040] Insert the LoRA module into the cross-attention layers of the U-Net down-sampling block, the middle block and the up-sampling block, and the attention layers of the text encoder Transformer; these attention modules all contain four linear projection matrices: query ( ), key ( ), value ( ) and output ( ); according to the needs, choose to insert only linear layers and 1x1 convolution layers; or extend to 3x3 convolution layers on this basis to increase the capture ability of local patterns of the model; set the dimension of LoRA and the scaling coefficient ;

[0041] S24, initialize the low-rank matrix:

[0042] For each inserted layer, split the original weight into a fixed base weight W and a trainable low-rank update , where , , r is the low-rank dimension, which satisfies ; after initialization, the output weight of each layer rewritten by LoRA is updated to ;

[0043] S25, training process:

[0044] Follow the training process of the diffusion model, in the forward process, gradually add noise to the target planar graph; in the reverse denoising process, require the model to predict the added noise according to the corresponding data set; by calculating the difference between the predicted noise and the real noise as the loss function, to optimize the parameters of the LoRA matrix.

[0045] Further, the specific operation steps of step S25 are as follows:

[0046] S251, sample noise step, randomly select a diffusion step t from a uniform distribution, calculate the noise scheduling parameter , the specific calculation method is:

[0047] Set the beta scheduling table: define a set of variance parameters β1, β2, …, β T , which increases with time step t = 1, …, T;

[0048] Calculate the single-step α value: Calculate for each step t:

[0049] ;

[0050] here This represents the proportion of the original image information retained in step t;

[0051] Calculate cumulative noise scheduling parameters To directly represent the sample with added noise at any time step, calculate:

[0052] ;

[0053] S252. Generate a noisy latent vector and input it into the original latent vector. Add noise to it, according to the formula:

[0054] ;

[0055] Obtain the noisy latent vector , where ϵ∼N(0,I) is standard Gaussian noise;

[0056] S253, Denoising Prediction:

[0057] Will The input is injected into a LoRA-based U-Net, resulting in predicted noise. ;

[0058] This represents "noise estimation of the network output with parameter θ";

[0059] θ represents all trainable parameters of the neural network;

[0060] S254, Loss Calculation:

[0061] Using mean squared error (MSE) as the primary training objective:

[0062] ;

[0063] To ensure consistency in the generated functional areas, a conditional constraint regularization term is added to the basic loss to obtain the total loss:

[0064] ;

[0065] Lambda: This is a weight coefficient used to balance the importance between the basic loss and the conditional loss; if lambda is set to a small value, the training of the model is mainly driven by the MSE (Mean Squared Error) loss; if lambda is set to a large value, the conditional constraint is emphasized, making the model pay more attention to meeting the consistency of functional areas or color labels; in actual operation, the value of lambda is determined through validation set tuning or cross-validation;

[0066] Conditional constraint loss or regularization term: This term is used to further constrain the generated results beyond the basic denoising task, so that they meet the functional division described by the input image or text; in actual operation, the input provides geometric constraints, MSE (Mean Squared Error) loss is used to measure the difference between the generated results and the target constraints;

[0067] S254, optimization update

[0068] Calculate the gradient of the loss with respect to A and B, and update these two low-rank matrices using an appropriate optimizer, while all other parameters are not updated, and use the cosine annealing restart learning rate scheduling strategy;

[0069] S255, loop iteration:

[0070] Repeat steps S252-S254 until a predetermined number of steps is reached, and the training is complete;

[0071] S26, model output:

[0072] Save the low-rank adaptation weight file periodically during training.

[0073] Further, step S3 specifically includes:

[0074] After training, load the low-rank adaptation weight and the base model dynamically, and generate the architectural layout through model inference;

[0075] Load the model: load the pre-trained base diffusion model and the trained special LoRA weight file;

[0076] Input: the user inputs a new target contour map; after the same preprocessing as during training, the contour map is input into the model as a generation condition;

[0077] Generation: the model starts the reverse denoising sampling of the diffusion process; this process is an iterative process, starting from a pure noise map, gradually removing noise under the guidance of the contour condition, and after a certain number of steps, generating a brand new, high-definition, contour-constrained and reasonable layout land use planning map, building / building group general plan or architectural plan.

[0078] A system for implementing a building planning class plane generation method of a potential diffusion model, the system comprising:

[0079] A data preprocessing module: responsible for standardization and enhancement of images;

[0080] A model training module: responsible for loading a basic model, injecting LoRA parameters, and executing a training cycle;

[0081] An inference generation module: responsible for loading a trained model, receiving user input, executing a generation process, and outputting results.

[0082] With the above technical solution, the present application has the following beneficial effects:

[0083] High generation quality and good diversity: thanks to the advanced generation mechanism of the diffusion model, the floor plan generated by the present application is significantly superior to the GAN / VAE-based method in terms of clarity, detail richness, and structural rationality, and can produce more diversified schemes.

[0084] High efficiency and low cost: using LoRA fine-tuning, the computational resource demand is greatly reduced (saving video memory and shortening training time), making it possible to train on consumer-grade GPUs. At the same time, the overfitting problem commonly encountered in direct fine-tuning of large models is avoided, and the generalization ability is stronger; compared with the parameterized design automatic generation program, a separate system for a corresponding project is avoided, and the investment is reduced.

[0085] High flexibility and convenient deployment: compared with artificial design Figure One A small LoRA file can represent a professional model for a specific style or type, and it is very convenient to manage and switch different models, with extremely low storage and deployment cost.

[0086] Improving design efficiency: compared with traditional artificial drawing or parameterized automatic generation, a large number of high-quality and high-creative alternative schemes can be provided for planning designers and architectural designers at the initial stage of the scheme, greatly stimulating design inspiration and shortening the design cycle. BRIEF DESCRIPTION OF DRAWINGS

[0088] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0089] Figure 1 The figure is a schematic diagram of the overall process of the present application.

[0090] Figure 2 A corresponding data set flowchart is constructed for the present application;

[0091] Figure 3 A general flowchart for the present application based on pre-training latent diffusion model for low-rank adaptation fine-tuning training;

[0092] Figure 4 A specific process chart for the present application based on pre-training latent diffusion model for low-rank adaptation fine-tuning training. DETAILED DESCRIPTION

[0094] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0095] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0096] Before further explaining the present application, the technical terms involved in the present application are explained as follows:

[0097] Diffusion Model: A new generation of generative model that generates data by simulating a forward process of gradually adding noise and a backward process of learning to gradually denoise. It has high generation quality and stable training.

[0098] LoRA (Low-Rank Adaptation): A technology for efficient fine-tuning of large models. It injects a low-rank decomposition bypass matrix into the original linear transformation layer in the model to simulate parameter updates, thereby greatly reducing the parameter quantity and computational complexity required for fine-tuning.

[0099] Latent Diffusion Model (LDM): A model that performs diffusion process in latent space rather than pixel space, significantly reducing computational complexity. Stable Diffusion is a well-known LDM.

[0100] Cross-Attention Layer: A kind of attention mechanism in Transformer model, which allows the model to focus on the information of another conditional sequence (such as the encoding of the outline graph) when processing a sequence (such as the image latent representation). In conditional diffusion model, it is the key layer to integrate conditional information.

[0101] Denoising Sampling: The process of generating images from diffusion models is a process of starting from random noise, gradually iterating and removing noise, and finally generating clear images.

[0102] DyLoRA: The dimensions of LoRA are split into multiple subunits for simultaneous training, automatically exploring the optimal dimensions.

[0103] Mean-square error (MSE) is a statistical measure of the difference between the estimated value and the true value or the predicted value and the true value. It is defined as the mathematical expectation of the square of the deviation or the mean of the square of the error. Its calculation formula is MSE = 1 / n∑(yi−ŷi)², which amplifies the influence of large errors by squaring and reflects the overall error level.

[0104] ControlNet (Conditional Control Network for Diffusion Models): A controllable image generation framework based on diffusion models, its core idea is to introduce an additional conditional control network based on the pre-trained latent diffusion model (such as Stable Diffusion), so that the model can accurately follow the input constraints (such as edges, depth, etc.) to generate target images while maintaining the original generation ability.

[0105] DreamBooth: A diffusion model fine-tuning method for personalized customization generation. Its core idea is to fine-tune the pre-trained diffusion model using a small amount of samples (usually 3-5 images), so that the model can accurately learn the characteristics of specific objects, characters or styles while maintaining the original generation ability.

[0106] In combination Figures 1-4 As shown in the embodiment, the embodiment provides a building planning type plane generation method and system based on a latent diffusion model. The detailed steps of the method are as follows:

[0107] Step 1: Collect data and build corresponding data sets;

[0108] The data of land use planning map, building (group) general layout, and building plane includes city land parameter, building design parameter, constraint condition, and related other data, such as land nature, building type, building function, room function, area, and area ratio.

[0109] S11, data standardization: standardize the collected data, unify the scale of the original picture, make all pictures under the same scale, unify the size and resolution of the original picture, ensure that the width and height are integer multiples of 64 to adapt to the potential diffusion model (such as 512x512 pixels).

[0110] S12, establish function and color mapping rules: not dependent on existing general semantic segmentation systems (such as Cityscapes, COCO Stuff general categories), but for the professional needs of architectural design / planning field, the corresponding relationship between color and function is defined by experts, and the "color-semantic" mapping rules are customized according to the design requirements. This custom rule can directly encode professional knowledge. In specific operation, the corresponding relationship between each functional area and color, for example, for land use planning map, yellow (255, 255, 0) represents residential land, red (255, 0, 0) represents commercial land, and green (0, 77, 0) represents sports land; for the building (group) layout plan in the implementation case, green (0, 255, 0) represents playground, red (255, 0, 0) represents administrative building, and blue (0, 0, 255) represents teaching building; for building plan, yellow (255, 255, 0) represents guest elevator area, olive green (177, 176, 70) represents fire escape, blue (0, 0, 255) represents stairwell, and gray (115, 127, 123) represents equipment room, etc., and naming specification is made; these are just examples, and "color-semantic" pairs can be customized according to specific implementation.

[0111] Through this "color-semantic" mapping, the architectural design and planning task is transformed into an image-semantic alignment task. In the conditional input stage of the diffusion model, the model can identify and learn the functional attributes represented by different color regions, so as to ensure the explainability and functional consistency of the generated results. Color mapping provides an explicit, low-dimensional functional label expression form, which not only preserves spatial information, but also serves as a lightweight conditional constraint.

[0112] During training, color-semantic mapping can be directly converted into additional regularization terms (such as area proportion, adjacency relationship regularization) to improve functional area consistency.

[0113] Let the data set input image be , the semantic set be , and the color set be , where each corresponds to a functional semantic .

[0114] Define the mapping function:

[0115] ;

[0116] Data preprocessing, the pixel color of the input image is converted into a functional semantic label:

[0117] The final label matrix Can be used as conditional input or regular constraints.

[0118] The advantage is that the professional semantics are better expressed, and the method is more efficient than general segmentation methods in small sample scenarios, avoiding dependence on large-scale semantic annotation systems.

[0119] S13, data labeling: according to the self-defined rules, the functional areas in each image are accurately labeled, so that they match the color label at the pixel level.

[0120] S14, data augmentation: apply geometric transformations such as horizontal flipping, vertical flipping, random rotation (e.g. 90°, 180°, 270°) to the labeled images to enrich the training samples and reduce model overfitting.

[0121] S15, generate text description: based on the labeling results, generate a corresponding text file for each image, describing the functional partition and its position, area, area ratio, etc. to build a picture-text data set. The text description in the same class of data set should be similar in structure, which is convenient for the model to learn the corresponding relationship between the image and the text.

[0122] Step two: use the pre-trained latent diffusion model as the base model for low-rank adaptation fine-tuning to obtain the low-rank adaptation model;

[0123] S21, load and freeze the pre-trained latent diffusion model;

[0124] Select a diffusion model pre-trained on a large-scale general image dataset as the base model, such as Stable Diffusion, Latent Diffusion Model (LDM, latent diffusion model), which has powerful image generation and understanding capabilities. Load the variational autoencoder (VAE), text encoder (CLIP) and UNet from the pre-trained latent diffusion model, and fix (freeze) their weights to ensure they do not update during fine-tuning.

[0125] S22, construct the latent representation of the training data;

[0126] Encode the pictures of the dataset prepared in S1 into latent vectors through VAE ; and the text description corresponding to the image is converted into a conditional vector through the text encoder ​​Since the latent vectors of each image are reused during training, these vectors can be pre-cached to accelerate subsequent iterations.

[0127] S23, Inject the LoRA module;

[0128] The LoRA module is inserted into the cross-attention layers of the U-Net downsampling block, intermediate block, and upsampling block, as well as the attention layer of the Transformer text encoder. These attention modules all contain four linear projection matrices: query( ),key( ),value( ) and output ( Depending on the needs, you can choose to insert only linear layers and 1×1 convolutional layers; or you can extend this to 3×3 convolutional layers to enhance the model's ability to capture local patterns. Set the dimensions of LoRA. and scaling factor For example, setting =32, =1 indicates the use of a 32-dimensional low-rank matrix.

[0129] S24. Initialize the low-rank matrix;

[0130] For each insertion layer, the original weights Decomposed into fixed base weights W and trainable low-rank updates. ,in , Let r be a low-rank dimension, satisfying After initialization, the output weights of each layer modified by LoRA are updated to... .

[0131] S25, Training Process;

[0132] Training Process: The training flow follows that of a diffusion model. During the forward pass, noise is progressively added to the target plane image; during the backward pass, the model is required to predict the added noise based on the corresponding dataset. The parameters of the LoRA matrix are optimized by calculating the difference between the predicted noise and the actual noise (e.g., mean squared error, MSE Loss) as the loss function.

[0133] Detailed operation steps:

[0134] S251. Sample the noise step: Randomly select a diffusion step t (e.g., between 0 and 1000) from a uniform distribution, and calculate the noise scheduling parameters. ;

[0135] The specific calculation method is as follows:

[0136] Set the beta schedule: Define a set of variance parameters β1, β2, …, βT that increase over time steps t = 1, …, T T .

[0137] (e.g. βt= β0+ (β1− β0) tT). T From a small value such as 1 × 10− 4 −4 to a larger value such as 0.02) uniformly.

[0138] Compute the single-step value: For each step t, compute:

[0139] ;

[0140] Here denotes the proportion of the original image information that is preserved in the t-th step.

[0141] Compute the cumulative noise schedule parameter : To directly represent the noisy sample at any time step, one can compute:

[0142] ;

[0143] S252, Generate a noisy latent vector by adding noise to the original latent vector generated in S22, according to:

[0144] ;

[0145] Obtain the noisy latent vector , where ϵ ~ N(0, I) is standard Gaussian noise.

[0146] S253, Denoising prediction

[0147] Input the into the LoRA-injected U-Net to obtain the predicted noise ;

[0148] denotes the “noise estimate of the network output with parameters θ”;

[0149] θ denotes all the trainable parameters of the neural network.

[0150] S254, Loss computation

[0151] Use mean squared error (MSE) as the main training objective:

[0152] ;

[0153] If it is necessary to ensure the consistency of the generated functional areas, a conditional constraint regular term can be added to the basic loss to obtain the total loss:

[0154] ;

[0155] λ: This is a weight coefficient to balance the importance between the basic loss and the conditional loss. If λ is set to a small value, the training of the model is mainly driven by the MSE loss; if λ is set to a large value, the conditional constraint is emphasized, making the model pay more attention to meeting the consistency of functional areas or color labels. In practical operation, the value of λ can be determined by validation set tuning or cross-validation.

[0156] : This is called conditional constraint loss or regularization term. Its role is to further constrain the generated results beyond the basic denoising task, so that they meet the functional division described by the input image or text.

[0157] In practical operation, the input provides geometric constraints such as area proportion, The difference between the generated result and the target constraint can be measured by the MSE loss.

[0158] For example, for the functional color block proportion constraint in land use planning maps, building (group) general layout, building plan, or building local plan, different colors represent different functional land properties (such as residential land, industrial land, etc.), buildings (such as administrative building, teaching building, dormitory, etc.), sites (such as parking lot, green land, sports ground, etc.), or spatial functions (such as office area, core tube, stairwell, etc.), usually want the generated color block area proportion to be consistent with the planning requirements. By defining an area proportion regularization term:

[0159] Calculate the pixel proportion p of each color in the generated image i .

[0160] Set the target proportion ,

[0161] Add the difference between the two to the loss, for example, squared error:

[0162] ;

[0163] This way you can punish the results that deviate from the target area ratio during training, so that the model gradually adjusts the relative area of each functional color block in the generated image while optimizing noise prediction.

[0164] For the total loss of the target to generate the building (group) general layout:

[0165] ;

[0166] In addition, multiple regularization terms can be added to ensure that the generated result composite design logic. For example, for building floor plan, adjacency relationship constraints can be introduced, using distance matrix to determine whether different functional areas in the generated result are adjacent as required, for example, elevators must be close to the hall, stairs and elevator room must maintain a certain distance. The pixel overlap or distance error deviating from the target adjacency pattern can be weighted and summed as a regularization term; shape consistency constraints can also be introduced, such as controlling the aspect ratio of elevator shaft or toilet, and the difference between the generated area aspect ratio and the target value can be calculated as a loss. These regularization terms can be calculated by a custom function, and their contribution to the total loss can be adjusted by a weight λ. The total loss is usually written as:

[0167] ;

[0168] is the basic denoising loss; and control the influence of area proportion constraint and space layout constraint on the total loss, their size should be set according to the importance of each constraint to the result; and correspond to the regularization terms of area proportion and layout constraint. By reasonably designing these regularization terms, the specific requirements of the planning site plan, building general plan layout, building floor plan, building local plan, etc. on the proportion of functional areas and the spatial relationship can be met while maintaining the diversity of model generation.

[0169] S254, optimization update;

[0170] Calculate the gradient of the loss with respect to A and B, and update these two low-rank matrices using an appropriate optimizer, while all other parameters are not updated, and learning rate scheduling strategies such as cosine annealing restart are used.

[0171] If multiple ranks or different levels of learning rates are needed, dynamic low-rank and hierarchical learning rate strategies can be enabled.

[0172] The low-rank dimension (r) is not the higher the better, but needs to be found according to the model, dataset, task, etc. Using dynamic low-rank, multiple dimensions of LoRA can be learned at the same time under the specified dimension, thus saving the trouble of finding the best dimension.

[0173] First, select a maximum rank rmax, and then split it into several smaller sub-ranks (such as 4 for each sub-rank). During training, these sub-ranks will participate in updating at the same time, as if multiple LoRA models are trained in parallel. After training is complete, multiple LoRA weight combinations of different ranks can be extracted from this set of joint models, such as r=4,8,12,16, etc. The optimal rank can be selected according to the performance on the validation set, without the need to retrain.

[0174] Extension to convolutional layers: Dynamic LoRA also applies to 3x3 convolutional layers, where the rank of the convolutional kernel is consistent with linear layers. Avoid manual tuning, efficiently obtain LoRA adaptation of multiple ranks; especially suitable for scenarios with limited samples but need to find the best model capacity.

[0175] Different learning rate weights are set at different layers of U-Net, making the model more flexible when adjusting shallow or deep features.

[0176] Hierarchical learning rate: The structure of U-Net usually includes multiple scale down blocks, intermediate blocks and scale up blocks, each block representing different levels of features. In order to make the adaptation of LoRA more accurate, different learning rate weights can be set for different blocks:

[0177] Specify weights: Assign weight vectors to the down-sampling part (12 layers), the middle part (1 layer) and the up-sampling part (12 layers) respectively, for example, the down-sampling part weight vector is [0.5, 0.5,…, 1.5], the middle sampling part weight vector is 2.0, and the up-sampling part weight vector is [1.5, 1.5,…, 0.5].

[0178] Mechanism of action: Blocks with high weight values update more significantly in training, while blocks with low or zero weights do not generate LoRA modules or have minimal update amplitude. This allows the model to fully learn new layout patterns at certain scales while maintaining stability at other scales.

[0179] Threshold control: A threshold can be set, and learning rate weights below the threshold will not create LoRA modules, which can be used to automatically filter layers that do not need to be adapted.

[0180] Dynamic low rank focuses on adaptively selecting the rank of LoRA, while hierarchical learning rate focuses on applying differential updates at different depths of the network. These two strategies can be combined to quickly find the optimal rank and adaptation depth in the case of limited training samples.

[0181] S255, loop iteration;

[0182] Repeat steps S252-S254 until a predetermined number of steps is reached to complete training.

[0183] S26, model output;

[0184] Periodically save the low-rank adaptation weight file (such as.safetensors format) during training.

[0185] Step three: based on the pre-trained latent diffusion model, load the low-rank adaptation model, input the sample data into the model, and generate the corresponding building (group) general plan layout / plan layout scheme.

[0186] After training, load the low-rank adaptation weights and the base model dynamically, generate the building layout through model inference.

[0187] Load model: load the pre-trained base diffusion model and the trained special LoRA weight file.

[0188] Input: the user inputs a new target contour map. After the same preprocessing as during training, the contour map is input into the model as a generation condition.

[0189] Generation: the model starts the reverse denoising sampling of the diffusion process (such as using the DDIM sampler). This process is an iterative process that starts with a pure noise image and gradually removes noise under the guidance of the contour condition. After a certain number of steps (such as 50 steps), a completely new, high-definition, contour-constrained, and reasonable layout land use planning map, building (group) general layout map, or building plan is generated.

[0190] Inference can be performed for different scenarios:

[0191] Land use planning map generation: input the design area boundary, and the model generates a city land use plan layout in potential space according to land property requirements. The output image represents different land properties with different colors, which can be adjusted by the designer.

[0192] General layout map generation: input the site boundary, and the model generates a building general layout in potential space according to functional requirements. The output image represents different functional buildings or sites with different colors, which can be adjusted by the designer.

[0193] Plan generation: input the floor contour, and the model generates a complete indoor functional layout. The designer can adjust it accordingly.

[0194] Local area plan generation: input the corresponding contour, and the model generates the plan layout of the corresponding area, such as core tube plan, hotel room plan, conference room plan, office area plan, etc.

[0195] Local completion: when there is already part of the layout, mark the known area in the input mask, and the model uses the denoising ability of the diffusion model to complete the missing area, so that the newly generated part is consistent with the original layout in function and style.

[0196] The system structure for implementing this method can include:

[0197] Data preprocessing module: responsible for image standardization and enhancement.

[0198] Model training module: responsible for loading the base model, injecting LoRA parameters, and executing the training loop.

[0199] Inference generation module: responsible for loading the trained model, receiving user input, executing the generation process and outputting the results.

[0200] Compared with the prior art, the innovations of the present application include:

[0201] 1) Collect building plan and general plan samples, label different building functions with colors, and generate a text file describing the correspondence between color and function;

[0202] 2) Horizontally flip, vertically flip and optionally randomly rotate the labeled images;

[0203] 3) Insert the LoRA low-rank adaptation module into the pre-trained latent diffusion model, set the dimensions and scaling coefficients, adjust the weights with the formula ( ), and only train the low-rank matrix (A, B);

[0204] 4) Fine-tune the model using mean square error loss; it can be extended to conditional regularization, dynamic low-rank adaptation, and hierarchical learning rate for fine-tuning;

[0205] The conditional regularization can be one or more.

[0206] 5) Load the trained LoRA weights back into the base model, and generate or complete planning schemes, building (group) general plan layouts or building plans according to the input site boundary, building outline or partial layout.

[0207] The present application innovatively proposes a specific data processing method: including constructing a custom "function-color" mapping, unifying the same set of data scale, and performing data labeling based on the custom "function-color" mapping.

[0208] The present application establishes a color-semantic mapping relationship, converts the functional attributes of buildings and planning into learnable color coding, and thus provides efficient conditional input for diffusion models. Existing diffusion models are mostly based on natural image data or semantic segmentation labeling, while the present application constructs a special data set for the field of architecture through "color-semantic" mapping, realizing cross-modal alignment. This mapping method does not rely on expensive pixel-level semantic segmentation tools and only needs color filling to complete data labeling, greatly reducing the cost of data set construction. And this data processing method can be applied to other model training.

[0209] "Color-semantic" mapping is not only applicable to architecture and urban planning, but also can be transferred to other image generation or understanding tasks:

[0210] Medical images: different colors represent different tissues, organs or lesion areas, which serve as conditional input to guide the model to generate or repair medical images;

[0211] Industrial inspection: Different color marks product defect types or part functions, used to assist in automatically generating inspection reports or training detection models;

[0212] Remote sensing image: Color distinguishes different ground object categories (farmland, water body, building, etc.), used for remote sensing image generation or change detection;

[0213] Education visualization: Quickly define teaching cases by customizing color labels to reduce the cost of complex data labeling.

[0214] Preferably, in order to achieve the purpose of the invention, the alternative solutions that can be adopted include:

[0215] Alternative of LoRA injection position: In addition to injecting into the cross-attention layer, the LoRA module can also be injected into other parts of the diffusion model UNet, such as the feedforward network layer or other linear transformations of the attention layer.

[0216] Alternative of conditional guidance method: In addition to directly inputting the contour map as a condition, the contour map can also be encoded into a semantic segmentation map, an edge map, or another form of latent representation before being input into the model as a condition.

[0217] Alternative of base model: The invention is not limited to Stable Diffusion or LDM, but can also be applied to other diffusion models with different structures (such as diffusion models based on U-ViT architecture and diffusion models based on DiT architecture) or other pre-trained image generation models (such as Transformer-based autoregressive models), and LoRA is also used for fine-tuning.

[0218] Alternative of training strategy:

[0219] For similar data forms, training strategies for different data sizes can be replaced:

[0220] Small sample customization: When there are only a few pictures of specific themes (such as CBD core area land planning, point tower residential area general layout, and triangular core tube plan) in the training set, DreamBooth can be used for personalized fine-tuning. DreamBooth binds a unique identifier for a specific subject in the pre-trained diffusion model and fine-tunes it with a small number of images (usually 3-5), achieving customization for specific subjects.

[0221] Large-scale data: ControlNet conditional guidance;

[0222] If there are a large number of contour-content pairs of data (for example, a large number of land edge maps-content planning map pairs, core tube plane outer contour-core tube function color filling picture pairs, etc.), ControlNet can be used. ControlNet copies a trainable network without changing the weights of the base model and gradually learns the spatial conditions through a "zero convolution layer". Its training usually relies on a large dataset of contour-content paired data for training.

[0223] Massive data: full parameter fine-tuning;

[0224] When the data size is huge and the overall model performance is expected to be improved, full parameter fine-tuning of the pre-trained diffusion model can be considered. This method re-trains all the weights of the model, which usually requires expensive computing resources and a large amount of data. Therefore, full parameter fine-tuning can be used as an alternative solution in extreme cases.

[0225] Expansion of application fields: the core idea of this scheme (pre-trained diffusion model + LoRA fine-tuning + conditional dataset) can be extended to other design generation fields, such as given contour content generation within the contour in the fields of interior design, urban planning, industrial design, and art, by replacing the corresponding training dataset.

[0226] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for building planning class floor plan generation based on latent diffusion model, characterized in that, Comprise the following steps: S1, collect land use planning map data, building / building group general layout data and layout data, and construct the corresponding data set; S2, using a pre-trained latent diffusion model as a base model, low-rank adaptive fine-tuning training is performed to obtain a low-rank adaptive model; S3, based on the pre-trained latent diffusion model, load the low-rank adaptive model, input the sample data into the model, and generate the corresponding building / building group general layout / layout scheme.

2. The method of generating a building planning class plan of a potential diffusion model according to claim 1, wherein, The corresponding data set constructed in step S1 specifically comprises the following steps: S11, data standardization: standardize the collected data, unify the scale of the original picture, so that all pictures are under the same scale, and unify the size and resolution of the original picture to adapt to the latent diffusion model; S12, establish the function and color mapping rule: without relying on the existing general semantic segmentation system, but according to the professional needs of architectural design / planning field, the expert defines the corresponding relationship between color and function, and according to the design requirements, the "color-semantic" mapping rule is self-defined. This self-defined rule directly encodes professional knowledge; Through this "color-semantic" mapping, the architectural design and planning task is converted into an image-semantic alignment task; In the conditional input stage of the diffusion model, the model can identify and learn the functional attributes represented by different color regions, so as to ensure the generated results have explainability and functional consistency; During training, the color-semantic mapping is directly converted into an additional regularization term to improve functional consistency; Let the dataset input image be , the semantic set be , and the color set be , wherein each corresponds to a functional semantic . Define the mapping function: ； During data pre-processing, the pixel colors of the input image are converted into functional semantic labels: ;​ The resulting label matrix As a conditional input or regular constraint; S13, data annotation: according to the self-defined rule, accurately annotate each functional area in each image so that it matches the color label at the pixel level; S14, data enhancement: apply horizontal flip, vertical flip and random rotation geometric transformation to the annotated image to enrich the training samples and reduce model overfitting; S15, generate text description: based on the annotation results, generate a corresponding text file for each image to describe the functional partition, its position, area and area ratio information, thereby constructing a picture-text data set.

3. The method of generating building planning class planes of a potential diffusion model according to claim 2, characterized in that, In the specific operation of step S12, the corresponding relationship between each functional area and color is set as follows: for the land use planning map, yellow (255, 255, 0) represents residential land, red (255, 0, 0) represents commercial land, and green (0, 77, 0) represents sports land; For the building / building group general layout map in the implementation case, green (0, 255, 0) represents playground, red (255, 0, 0) represents administrative building, and blue (0, 0, 255) represents teaching building; For the building plan, yellow (255, 255, 0) represents guest elevator area, olive green (177, 176, 70) represents fire escape, blue (0, 0, 255) represents stairwell, and gray (115, 127, 123) represents equipment room, and a naming specification is established.

4. The method of claim 1, wherein the building planning class plane of the potential diffusion model is generated by, Step S2 specifically comprises the following steps: S21, load and freeze the pre-trained latent diffusion model: A diffusion model pre-trained on a large-scale general image dataset is selected as the base model, and the variational autoencoder (VAE), text encoder (CLIP), and UNet are loaded from the pre-trained latent diffusion model, with their weights fixed to ensure they are not updated during the fine-tuning process; S22, construct the latent representation of the training data: For the pictures of the dataset prepared in step S1, encode into latent vectors by VAE ; the text description corresponding to each picture is converted into a conditional vector by a text encoder ; since the latent vectors of each picture are reused during the training process, it is selected to pre-cache these vectors to speed up subsequent iterations; S23, inject the LoRA module: LoRA modules are inserted into cross-attention layers of U-Net down-sampling blocks, middle blocks and up-sampling blocks, and attention layers of text encoder Transformer; these attention modules all contain four linear projection matrices: query ( ), key ( ), value ( ) and output ( ); as needed, only linear layers and 1x1 convolution layers are selected to be inserted; or extended to 3x3 convolution layers on this basis to increase the capturing ability of local patterns of the model; the dimensions of LoRA and scaling coefficients are set; S24, initialize the low-rank matrix: For each inserted layer, the original weights are split into a fixed base weight W and a trainable low-rank update where , r is the low-rank dimension, satisfying After initialization, each layer output weight that is overwritten by LoRA is updated as ; S25, training process: Follow the training process of the diffusion model, in the forward process, gradually add noise to the target plan; in the reverse denoising process, require the model to predict the added noise according to the corresponding data set; calculate the difference between the predicted noise and the real noise as the loss function to optimize the parameters of the LoRA matrix.

5. The method of generating a building planning class plan of a potential diffusion model according to claim 4, wherein, The specific operation steps of step S25 are as follows: S251. Sampling noise step: Randomly select a diffusion step t from the uniform distribution and calculate the noise scheduling parameters. The specific calculation method is as follows: Set the beta schedule: define a set of variance parameters β1, β2,..., βT increasing in time steps t = 1,..., T T ; Calculate the single-step alpha value: for each step t, calculate: ； Here represents the proportion of the original image information preserved in the t-th step; Computing cumulative noise schedule parameters : To directly represent the sample after adding noise at any time step, compute: ； S252, generating a noisy latent vector, to the original latent vector with noise, according to the formula: ； obtaining a noisy latent vector where e ~ N(0, I) is standard Gaussian noise; S253, denoising prediction: Will Input the LoRA-injected U-Net to get the predicted noise ; denotes "noise estimate of the network output with parameter Θ"; θ represents all trainable parameters of the neural network; S254, loss calculation: Use mean square error (MSE) as the main training target: ； If you need to ensure the consistency of the generated functional areas, add a conditional constraint regular term to the basic loss to get the total loss: ； λ: This is a weight coefficient used to balance the importance between the basic loss and the conditional loss; if λ is set to a small value, the model's training is mainly driven by the MSE (mean square error) loss; if λ is set to a larger value, the conditional constraint is emphasized, making the model pay more attention to meeting the consistency of the functional areas or color labels; in actual operation, the value of λ is determined through validation set tuning or cross-validation; : This is called conditional constraint loss or regular term; its role is to further constrain the generated results beyond the basic denoising task, so that they conform to the functional partition described by the input image or text; in actual operation, the input provides geometric constraints, The difference between the generated results and the target constraints is measured by the MSE (Mean Square Error) loss. S254, optimization update: Calculate the gradient of the loss with respect to A and B, and use an appropriate optimizer to update these two low-rank matrices, while all other parameters are not updated, and a cosine annealing restart learning rate scheduling strategy is adopted; S255, loop iteration: Repeat steps S252-S254 until a predetermined number of steps is reached to complete the training; S26, model output: Save the low-rank adaptation weight file periodically during the training process.

6. The method of generating a building planning class plan of a potential diffusion model according to claim 1, wherein, Step S3 specifically includes: After training, load the low-rank adaptation weight with the base model dynamically, and generate the building plan layout through model inference; Load model: load the pre-trained base diffusion model and the trained special LoRA weight file; Input: the user inputs a new target contour map; after the same preprocessing as during training, the contour map is input into the model as a generation condition; Generation: the model starts the reverse denoising sampling of the diffusion process; this process is an iterative process that starts with a pure noise map and gradually removes noise under the guidance of the contour condition; after a certain number of steps, a brand new, high-definition, contour-constrained, and layout-reasonable land use planning map, building / building group general plan, or building plan is generated.

7. A system for implementing a method for generating a building planning type plan according to any one of the latent diffusion models of claims 1-6, characterized in that, The system includes: Data preprocessing module: responsible for image standardization and enhancement; Model training module: responsible for loading the base model, injecting LoRA parameters, and executing the training loop; Inference generation module: responsible for loading the trained model, receiving user input, executing the generation process, and outputting the results.

Images