Urban plan generation method based on remote sensing prediction and style migration

The integration of remote sensing prediction and style migration with LoRA and ControlNet enhances generative urban planning models, addressing data scarcity and context limitations, resulting in accurate and versatile urban planning outputs.

CN119090990APending Publication Date: 2024-12-06TONGJI UNIV
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411212859.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing generative urban planning methods lack generalization capabilities and rely on scarce and poorly formatted data sets, leading to inaccurate and context-limited outputs, especially when applied to regions with different geographical environments or social conditions.

Method used

A method integrating remote sensing prediction and style migration, using a potential diffusion model fine-tuned with LoRA and enhanced by ControlNet convolutional neural networks to extract and embed geographic information, enabling accurate and versatile urban planning generation.

Benefits of technology

The method generates urban planning outputs with high accuracy and versatility, effectively incorporating geographic information and reducing the need for extensive retraining, while maintaining model efficiency and stability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090990A_ABST
    Figure CN119090990A_ABST
Patent Text Reader

Abstract

The invention relates to a city planning generation method based on remote sensing prediction and style migration, and the method comprises the steps: carrying out the feature extraction of an original image, and taking the spatial information of a to-be-planned region as a first generation condition; regulating and controlling an image generation network trained by a remote sensing data set according to a first generation condition, and obtaining a remote sensing prediction image under the first generation condition; performing feature extraction on the remote sensing prediction image to obtain a second generation condition, combining the second generation condition and the first generation condition to serve as a third generation condition, regulating and controlling an image generation network trained by a city planning data set according to the third generation condition, and migrating the style of the remote sensing prediction image to the city planning image, and outputting the target city planning image. Compared with the prior art, the method has the advantages of being high in accuracy, good in universality, efficient, stable and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of urban planning, and in particular to an urban planning generation method based on remote sensing prediction and style migration. Background Art

[0002] Generative urban planning is the study of using generative models to assist urban planning. It aims to help address some of the limitations of traditional design planning and significantly improve planning efficiency, such as replacing repetitive work in the design process and calculating and balancing various constraints that often conflict with each other in urban planning.

[0003] However, most existing generative urban planning methods rely on imitating training set data, which results in the generated content being reasonable only within the scope of the training set and lacking generalization capabilities. For example, if an excellent generative urban planning model is trained using data from a certain city, then the model can only generate content similar to that of the training city in any other task. When faced with other regions with different geographical environments or social conditions, it must be retrained because these models do not actually have the ability to consider the above conditions.

[0004] At the same time, generative urban planning also has the problem of scarce effective data sets. Relevant data such as planning maps used for model training lack formatted large-scale data sets. At the same time, the artistic elements in the planning maps will seriously affect the model learning performance, which further increases the difficulty of training generative urban planning models. Even if the optimal model architecture can be obtained, it is still difficult to obtain a pre-trained model that can generate the required data without a suitable data set. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a city planning generation method based on remote sensing prediction and style migration with high accuracy, good versatility, high efficiency and stability.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for generating urban planning based on remote sensing prediction and style migration, the method comprising:

[0008] Extracting features from the original image to obtain spatial information of the area to be planned as the first generation condition;

[0009] According to the first generation condition, an image generation network trained with a remote sensing data set is regulated to obtain a remote sensing predicted image under the first generation condition;

[0010] Feature extraction is performed on the remote sensing prediction image to obtain a second generation condition, the second generation condition and the first generation condition are combined as a third generation condition, and the image generation network trained with the urban planning data set is regulated according to the third generation condition to transfer the style of the remote sensing prediction image to the urban planning image, and output the target urban planning image.

[0011] Preferably, the original image of the area to be planned includes a remote sensing image, a road network screenshot and a hand-drawn draft;

[0012] The remote sensing image is preprocessed by edge detection method before feature extraction;

[0013] The road network screenshots are pre-processed by using a straight line detection method before feature extraction.

[0014] Preferably, the first generation condition includes a text feature condition and an image feature condition, and the image feature condition is extracted by using a convolutional neural network; the second generation condition is obtained by extracting remote sensing prediction image features by using a convolutional neural network.

[0015] Preferably, the text feature conditions are extracted using a Transformer cross-attention network.

[0016] Preferably, the convolutional neural network is a ControlNet convolutional neural network.

[0017] Preferably, the image generation network uses a potential diffusion model as a base model and uses a low-rank adaptive LoRA fine-tuning method for fine-tuning training.

[0018] Preferably, the output expression of the potential diffusion model under the control of the ControlNet convolutional neural network is:

[0019]

[0020] Where: is the output of the initial latent diffusion model, x is the input of the initial latent diffusion model, and Φ is the weight; is the output of ControlNet convolutional neural network, Φ cl With Φ c2 represents the weights of the additional blocks C1 and C2 of the ControlNet convolutional neural network, Φ c represents the weight updated after training of the trainable replica block N0; by cl , Φ c2 With Φ c The training,featured condition c processed by the ControlNet convolutional neural network block is injected into the diffusion behavior of the potential diffusion model through the convolutional layer output;

[0021] For the image generation network trained with remote sensing datasets, the input x of its initial latent diffusion model is the original input image; for the image generation network trained with urban planning datasets, the input x of its initial latent diffusion model is the remote sensing predicted image.

[0022] Preferably, the step of regulating the image generation network trained with the remote sensing data set according to the first generation condition to obtain the remote sensing predicted image under the first generation condition is specifically as follows:

[0023] The spatial information C(c) of the area to be planned is input into the convolutional neural network, and the semantic label and planning target information y corresponding to the remote sensing dataset are input into the potential diffusion model corresponding to the remote sensing prediction link to generate the remote sensing prediction image of the area to be planned under the specified planning target conditions.

[0024] Preferably, the feature extraction of the remote sensing prediction image is performed to obtain the second generation condition, the second generation condition and the first generation condition are used as the third generation condition, the image generation network trained with the urban planning data set is regulated according to the third generation condition, the style of the remote sensing prediction image is transferred to the urban planning image, and the output target urban planning image is specifically:

[0025] Combine the spatial information C(c) of the planned area with the remote sensing prediction image Input into the convolutional neural network to extract the geographic feature information and graphic information in the remote sensing prediction image as the second generation condition;

[0026] Input the planning target information y into the feature encoder τ of the potential diffusion model corresponding to the style transfer link θ In this paper, the semantic labels introduced by the urban planning dataset are used to transfer the style of the remote sensing prediction image to the urban planning image, and the target urban planning image is output.

[0027] Preferably, for the image generation network trained with the urban planning dataset, the urban planning style is used as the first-level semantic label, the spatial information is used as the second-level semantic label and is aligned with the remote sensing prediction link.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) The generative urban planning method based on remote sensing prediction and style transfer designed by the present invention introduces additional geographic information into generative urban planning through remote sensing images, thereby generating more accurate urban planning results.

[0030] (2) The latent diffusion model is used as the generative basis model for text-to-image translation. The latent diffusion model is efficiently fine-tuned to achieve learning of urban planning and remote sensing data. It has better diversity capture and cross-domain generalization capabilities, and transforms the general generative model into a model specific to the field of urban planning with precisely controllable generative behavior, which is more universal.

[0031] (3) The present invention can introduce remote sensing and urban planning related knowledge and semantics into the pre-trained image generation model using only a small amount of data set samples, and use the model fine-tuning method to fine-tune the potential diffusion model instead of retraining it. This can retain the generation capability of the fully trained model while introducing the required knowledge and semantic content at a lower training cost.

[0032] (4) By introducing an additional convolutional neural network architecture, we can embed and regulate detailed geospatial information based on remote sensing images, robustly extract spatial information from image inputs and regulate the generation behavior of the diffusion model. We also embed the ability to consider the surrounding environment based on geographic information into the generation process, which is lacking in most generative urban planning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a logical architecture diagram of the method model of the present invention;

[0034] Figure 2 This is the actual architecture diagram of this system model;

[0035] Figure 3 It is the architecture diagram of the remote sensing prediction pipeline model;

[0036] Figure 4 This is the style transfer pipeline model architecture diagram;

[0037] Figure 5 It is the embedded structure diagram of the convolutional neural network in the generated network;

[0038] Figure 6 is a schematic diagram of a method for fine-tuning a potential diffusion model in an embodiment;

[0039] Figure 7 This is a schematic diagram of LDM behavior under ControlNet control;

[0040] Figure 8 Schematic diagram for comparing preprocessing results; (a) is a schematic diagram for comparing edge detection results for remote sensing images, and (b) is a schematic diagram for comparing straight line detection results for road network screenshots. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0042] Example 1

[0043] This embodiment provides a method for generating urban planning based on remote sensing prediction and style migration. Figure 1 As shown in the figure, it mainly consists of two processing pipelines: remote sensing prediction pipeline and style migration pipeline. The former is responsible for processing geographic information input and generating corresponding remote sensing prediction images, and the latter is responsible for redrawing remote sensing prediction images to urban planning maps. The method includes:

[0044] Step S1: Use a remote sensing dataset and an urban planning dataset with semantic labels to train a text-to-image generative network, respectively, to obtain a model network that can generate remote sensing data and urban planning data respectively;

[0045] In this embodiment, a potential diffusion model is used as a base model, and its training refers to fine-tuning training of a pre-trained base model. The base model and the fine-tuning method have a certain range of selection, such as model version, etc. Here, the base model uses the Stable-Diffusion-v1.5 pre-trained model, which is widely used in the Stable-Diffusion series of models provided by runwayml training, and the fine-tuning method uses the low-rank adaptive fine-tuning method LoRA (Low-Rank Adaptation). The reason for the introduction of LoRA is that the complete retraining of large pre-trained models has extremely high costs, and it is difficult to obtain the desired model when the data set is limited (especially in the field of urban planning). Therefore, it is necessary to fine-tune the excellent pre-trained model and introduce additional knowledge or semantic content to it while retaining the generation ability of the original pre-trained model. Figure 6 The basic structure and fine-tuning principle of LoRA are demonstrated.

[0046] Assume the pre-trained model weight matrix is Then its fine-tuning weight update ΔW can be represented by the following low-rank decomposition:

[0047] W0+ΔW=W0+BA

[0048] in And the low rank r<<min(d,k).

[0049] When training is initialized, A is initialized to a Gaussian random number and B is initialized to 0, so that ΔW = BA = 0. During the training process, W0 and ΔW both accept the input x and are included in the forward transfer process. The forward transfer content after adding low-rank decomposition is:

[0050] h=W0x+ΔWx=W0x+BAx

[0051] For LoRA, the fine-tuning optimization objective expression is:

[0052]

[0053] Through fine-tuning, two types of image data are introduced into LDM: geographic information data based on remote sensing and planning master plan data. Remote sensing geographic information data is used to introduce the generation capability based on the real urban spatial layout into the model, and then the planning master plan data is used for style transfer and the generated spatial layout is converted into a planning master plan. This is achieved by adding training set type labels to the training set in addition to the category semantic labels within the training set during training. That is, for the same type of planning data, the category labels in the two training sets are kept the same, and an additional label is used to mark whether the data belongs to remote sensing data or planning data.

[0054] LoRA training uses the following training configuration: 50 images from the training set are selected each time to enter the network, and the training is repeated 10 times per image per epoch. The AdamW8bit optimizer is selected, the learning rate is set to 0.00001, and cosine warmup is used for training images with a resolution of 512X512.

[0055] Step S2: Use a deep neural network to extract generation conditions represented by text, images, etc., and convert them into feature vectors in the generation network;

[0056] The semantic content of LDM can only be controlled by text prompts, and the image input will lose pixel spatial conditions after being compressed by the featurizer. It is difficult to accurately express complex spatial structures by adding descriptions only through text prompts, and too many prompts will cause the subsequent prompts to have insufficient influence or important prompts to be covered due to the prompt weight reduction mechanism of LDM. Generating an ideal image with spatial layout requires frequent trial and error and prompt fine-tuning.

[0057] The present invention extracts geographic and spatial layout information from images through the ControlNet convolutional neural network, and converts it into features to be embedded in the LDM diffusion process, so that the model can use the image to control the generation logic and effectively embed geographic and spatial layout information. This is achieved by connecting an additional end-to-end neural network structure to the pre-trained model, while retaining the original model function and adding additional conditional control learning capabilities to it, providing it with precise spatial layout control capabilities in the generation application of urban planning, and allowing this paper to provide additional geographic information input to the model through additional remote sensing data. Figure 5 The embedded structure of the ControlNet convolutional network used in the LDM network is shown.

[0058] For the original LDM network, the following formula is used to briefly summarize its generation behavior:

[0059]

[0060] Among them, the weight of the original LDM is expressed as Φ, which means that the LDM network can accept input x (that is, the basic conditions such as text prompt accepted by the basic LDM) and output image y.

[0061] like Figure 6 As shown, the LDM behavior under ControlNet control can be described as:

[0062]

[0063] Among them, Φ c1 With Φ c2 represents the weights of the additional blocks C1 and C2 of ControlNet, Φ c represents the weight updated by the trainable replica block N0 after training, and y c It represents the LDM output after being regulated by ControlNet, that is, the output with the additional condition c embedded. cl , Φ c2 With Φ c The characterized conditions c processed by the ControlNet block (for example, the training ControlNet block establishes a mapping from the spatial layout condition input to the output containing the layout image) are injected into the LDM diffusion behavior through the convolutional layer output, and the LDM diffusion process is controlled by the same conditions.

[0064] When the system is running, it first needs to extract spatial information from the input image through preprocessing. The input image can be a remote sensing image, a road network screenshot, a hand-drawn draft, etc. Selecting the appropriate preprocessing method for different types of input images can achieve multi-modal spatial condition input. For remote sensing images, the edge detection (Canny Edge) method is selected; for road network screenshots, the straight line detection (M-LSD) method is selected. Figure 7 Shows the spatially conditioned samples after preprocessing.

[0065] Step S3: using the extracted conditional features to regulate the generative network trained with remote sensing data to obtain the remote sensing predicted image output under the condition;

[0066] This step is the workflow of the remote sensing prediction pipeline. Figure 3 The basic architecture of the remote sensing prediction pipeline is shown. The extracted spatial information C(c) is input into the ControlNet network as an additional condition, and the semantic label corresponding to the remote sensing dataset and the planning target semantic label y are input into the LDM to generate the remote sensing prediction image of the area after using the specified planning target. Taking the two inputs used above as an example, both inputs are labeled as "Residential" and their remote sensing images are predicted after being planned as "residential areas" (assuming that all residential areas are labeled as "Residential" during the training process).

[0067] The remote sensing prediction output obtained is the remote sensing prediction output after embedding geographic information. The essence of this output is the remote sensing-based prediction image generated after planning based on the input spatial information and planning goals, and will be further processed as the input of the urban planning master plan stylization pipeline. The remote sensing prediction pipeline will output the following information to the stylization pipeline:

[0068] (1) Spatial information C(c) of the area to be planned: This information is obtained by extracting spatial information from the preliminary spatial input c such as the sketch of the area to be planned, and generating the content space layout through ControlNet control. The extracted spatial information will be input into the stylization pipeline for spatial information control.

[0069] (2) Planning target information y: This information is used as the LDM input of the remote sensing prediction link to control the type of spatial layout generated for the planned area during the remote sensing prediction process. For example, if you want to draw a planning master plan with residential areas as the target, you must first generate a remote sensing prediction with residential areas as the content. The planning target information will be input into the stylization pipeline to specify the planning target (style).

[0070] (3) Remote sensing prediction images The generated content contains possible remote sensing data predicted after training based on real remote sensing data, as remote sensing input when there is no original remote sensing image. The remote sensing geographic information will be input into the stylization pipeline to provide it with additional geographic information guidance.

[0071] Step S4: using the remote sensing prediction image output and the extracted generation conditions as new generation conditions, regulating the generation network trained with the urban planning dataset, converting the remote sensing image with the urban planning image style, and outputting the target urban planning image;

[0072] Figure 4 Shows the basic architecture of the style transfer pipeline.

[0073] Step S4 mainly includes the following steps:

[0074] S41, uses a convolutional neural network to extract features again on the remote sensing prediction image output by S3, obtains the geographic feature information and graphic information contained in the remote sensing image, which is used to introduce geographic information guidance for the urban planning map and optimize the output graphic stability.

[0075] In this step, the stylization pipeline obtains the following outputs from the remote sensing prediction pipeline: spatial information C(c), which is extracted by reprocessing the remote sensing prediction image of the planned area; planning target information y, which is the semantic label input by the text prompt of LDM; geographic information Provided by the output of the remote sensing prediction module. C(c) and Input the ControlNet convolutional neural network and input the planning target information y into the LDM feature encoder τ θ , which realizes the style transfer of remote sensing prediction input through the semantic labels introduced by L2

[0076] S42, the extracted geographic feature information and graphic information are sent to the model network trained by the urban planning map, and the generation conditions extracted in S2 are also input to realize the generation of the urban planning map based on the remote sensing prediction image.

[0077] The stylized pipeline uses LoRAL2, which is trained on the master plan dataset and is dedicated to style transfer. The master plan style is the first-level semantic label, the spatial layout is the second-level semantic label, and it is aligned with the remote sensing prediction module. Since it is very difficult to collect data that can be paired with geographic information and planning maps in urban planning datasets, unpaired data is used for training, and semantic labels are used to distinguish geographic information from the master plan.

[0078] Experiments have found that embedding geographic feature information conditions after a certain number of iterations and unloading them after a certain number of steps can obtain the best generation effect and the most stable graphics. Experiments have shown that the optimal value of this number of steps is not exactly the same under different pre-training models, but after experiments, the geographic information conditions are embedded at 50% of the iteration steps and unloaded at 80% of the iteration steps, while the generation conditions are embedded at 0% and unloaded at 100%. When the embedding weight strength of the two is set to 1, they often have better effects. On this basis, the above hyperparameters are further fine-tuned to obtain the optimal values.

[0079] In summary, the present invention provides a generative urban planning solution based on remote sensing prediction and style transfer, and proposes a generative model architecture that generates content that accurately meets certain knowledge and standards. Through the remote sensing field, high-quality resources based on data sets are provided to a certain extent for urban planning research, and remote sensing is integrated to introduce additional information resources for generative urban planning, and the ability to consider inputs such as geographic information is introduced into generative urban planning.

[0080] An urban planning dataset is constructed in the field of integrated remote sensing, and the model architecture is adjusted to adapt to input information from multiple fields. The model is regulated to generate the output of the integrated two. With the support of the constructed urban planning and remote sensing datasets, an updated model architecture is provided for the cutting-edge generative model, enabling it to provide robust, general, and precisely controlled urban planning generation capabilities.

[0081] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for generating urban planning based on remote sensing prediction and style migration, It is characterized in that The method includes: Extracting features from the original image to obtain spatial information of the area to be planned as the first generation condition; According to the first generation condition, an image generation network trained with a remote sensing data set is regulated to obtain a remote sensing predicted image under the first generation condition; Feature extraction is performed on the remote sensing prediction image to obtain a second generation condition, the second generation condition and the first generation condition are combined as a third generation condition, and the image generation network trained with the urban planning data set is regulated according to the third generation condition to transfer the style of the remote sensing prediction image to the urban planning image, and output the target urban planning image.

2. According to claim 1, a method for generating urban planning based on remote sensing prediction and style migration, It is characterized in that The original images of the area to be planned include remote sensing images, road network screenshots and hand-drawn drafts; The remote sensing image is preprocessed by edge detection method before feature extraction; The road network screenshots are pre-processed by using a straight line detection method before feature extraction.

3. According to claim 1, a method for generating urban planning based on remote sensing prediction and style migration, It is characterized in that The first generation condition includes text feature conditions and image feature conditions, and the image feature conditions are extracted using a convolutional neural network; the second generation condition is obtained by extracting remote sensing prediction image features using a convolutional neural network.

4. According to claim 3, a method for generating urban planning based on remote sensing prediction and style migration, It is characterized in that The text feature conditions are extracted using a Transformer cross-attention network.

5. According to claim 3, a method for generating urban planning based on remote sensing prediction and style migration, It is characterized in that The convolutional neural network is a ControlNet convolutional neural network.

6. The urban planning generation method based on remote sensing prediction and style migration according to claim 5, It is characterized in that The image generation network uses the potential diffusion model as the base model and uses the low-rank adaptive LoRA fine-tuning method for fine-tuning training.

7. The urban planning generation method based on remote sensing prediction and style migration according to claim 6, It is characterized in that The output expression of the potential diffusion model under the control of the ControlNet convolutional neural network is: Where: is the output of the initial latent diffusion model, x is the input of the initial latent diffusion model, and Φ is the weight; is the output of ControlNet convolutional neural network, Φ cl With Φ c2 Represents the extra block C of the ControlNet convolutional neural network 1 and C 2 The weight of c Represents the trainable replica block N 0 The weights updated after training; by cl , Φ c2 With Φ c The training,featured condition c processed by the ControlNet convolutional neural network block is injected into the diffusion behavior of the potential diffusion model through the convolutional layer output; For the image generation network trained with remote sensing datasets, the input x of its initial latent diffusion model is the original input image; for the image generation network trained with urban planning datasets, the input x of its initial latent diffusion model is the remote sensing predicted image.

8. The urban planning generation method based on remote sensing prediction and style migration according to claim 3, It is characterized in that The step of regulating the image generation network trained with the remote sensing data set according to the first generation condition to obtain the remote sensing predicted image under the first generation condition is specifically: The spatial information C(c) of the area to be planned is input into the convolutional neural network, and the semantic label and planning target information y corresponding to the remote sensing dataset are input into the potential diffusion model corresponding to the remote sensing prediction link to generate the remote sensing prediction image of the area to be planned under the specified planning target conditions.

9. The urban planning generation method based on remote sensing prediction and style migration according to claim 8, It is characterized in that The feature extraction of the remote sensing prediction image is performed to obtain the second generation condition, the second generation condition and the first generation condition are used as the third generation condition, the image generation network trained with the urban planning data set is regulated according to the third generation condition, the style of the remote sensing prediction image is transferred to the urban planning image, and the target urban planning image is output as follows: Combine the spatial information C(c) of the planned area with the remote sensing prediction image Input into the convolutional neural network to extract the geographic feature information and graphic information in the remote sensing prediction image as the second generation condition; Input the planning target information y into the feature encoder τ of the potential diffusion model corresponding to the style transfer link θ In this paper, the semantic labels introduced by the urban planning dataset are used to transfer the style of the remote sensing prediction image to the urban planning image, and the target urban planning image is output.

10. The urban planning generation method based on remote sensing prediction and style migration according to claim 1, It is characterized in that For the image generation network trained with the urban planning dataset, the urban planning style is used as the first-level semantic label, and the spatial information is used as the second-level semantic label and is aligned with the remote sensing prediction link.

Citation Information

Cited By

  • Sonar image generation method fusing stable diffusion and style migration

    CN119832113A

  • Multi-conditional constraint fusion generation method and system for planning intention graph generation

    CN120745021A

  • A multi-condition constraint fusion generation method and system for planning intention graph generation

    CN120745021B