Method and equipment for generating ultrahigh-resolution city layout by using design drawing, and medium

Abstract

The invention provides a method and equipment for generating an ultrahigh-resolution city layout by using a design drawing, and a medium, relates to the field of image generation, and is used for solving the problem that a traditional automatic city generation program is based on rules and lacks flexibility. The method comprises the steps of obtaining a target city layout design drawing drawn by a user; according to a preset step length and a preset sequence, sequentially extracting design sub-drawings with preset window sizes from the target city layout design drawing; coding the design sub-graph into a condition vector feature graph; inputting the character cue words and the condition vector feature graph into a generative model to obtain a partial city layout map corresponding to the design sub-graph; the character prompt words are used for describing task information of image generation; and filling the partial city layout map into corresponding positions in the blank map until the target city layout design drawing is traversed, and obtaining a target city layout map. The method can adapt to design drawings with different complexities and flexibly and automatically generate the city layout map.

Description

Technical Field

[0001] The present application relates to the field of image generation, and provides a method, device and medium for generating ultra-high-resolution urban layout using design drawings. Background Art

[0002] Traditional automated city generation programs are tools that use algorithms and data models to simulate and create urban environments. Such programs are often used in game development, urban planning, architectural design, simulation, and artificial intelligence research. They can automatically generate large-scale urban layouts and structures, including roads, buildings, green spaces, etc.

[0003] Traditional automated city generation programs generally rely on some basic algorithms and rules and lack flexibility. Summary of the invention

[0004] The present application provides a method, device and medium for generating ultra-high-resolution city layout using design drawings, which are used to solve the problem that traditional automated city generation programs are rule-based and lack flexibility.

[0005] In a first aspect, the present application provides a method for generating an ultra-high-resolution urban layout using a design drawing, comprising:

[0006] Obtain the target city layout design map drawn by the user;

[0007] Extracting design sub-graphs of preset window sizes from the target city layout design graph in sequence according to a preset step length and a preset order;

[0008] encoding the designed subgraph into a conditional vector feature graph;

[0009] Inputting the text prompt words and the conditional vector feature map into the generation model to obtain a partial city layout map corresponding to the design sub-map; wherein the text prompt words are used to describe the task information of image generation; the generation model is obtained by training based on multiple groups of data pairs; each group of data pairs includes the text prompt words, each real city layout map and the city layout design map corresponding to each real city layout map;

[0010] The partial city layout map is filled into the corresponding position in the blank map until the target city layout design map is traversed to obtain the target city layout map; the blank map has the same size as the target city layout design map.

[0011] Optionally, each set of data pairs is obtained as follows:

[0012] A plurality of real city layout maps are obtained from a map database; various elements in each real city layout map are represented by preset RGB values;

[0013] By manually drawing, the urban layout design map corresponding to each urban layout real map is obtained; various elements in each urban layout design map are represented by preset RGB values;

[0014] Each group of data pairs is obtained according to the text prompt words, each real city layout picture, and the city layout design picture corresponding to each real city layout picture.

[0015] Optionally, obtaining each set of data pairs according to the text prompt words, each real city layout map, and the city layout design map corresponding to each real city layout map includes:

[0016] Randomly selecting some real city layout pictures from the plurality of real city layout pictures;

[0017] Covering the upper half of the urban layout design drawing corresponding to the partial urban layout real picture with the upper half of the partial urban layout real picture, and / or covering the left half of the urban layout design drawing corresponding to the partial urban layout real picture with the left half of the partial urban layout real picture;

[0018] Each group of data pairs is obtained according to the text prompt words, each real city layout picture, and the city layout design picture corresponding to each real city layout picture.

[0019] Optionally, encoding the designed sub-graph into a conditional vector feature graph includes:

[0020] The designed subgraph is input into the encoding network, and a conditional vector feature graph of a 64*64 feature space is output; wherein the encoding network includes four convolutional layers connected in sequence; the number of channels of the four convolutional layers are 16, 32, 96 and 256 respectively; and each convolutional layer is connected using an activation function.

[0021] Optionally, the generation model includes a stable diffusion model and a control network model, and the stable diffusion model includes a 64*64 SD encoding block, a 32*32 SD encoding block, a 16*16 SD encoding block, an 8*8 SD encoding block, an 8*8 SD intermediate block, an 8*8 SD decoding block, a 16*16 SD decoding block, a 32*32 SD decoding block, and a 64*64 SD decoding block connected in sequence;

[0022] The control network model includes a zero convolution layer, a 64*64 SD coding block, a 32*32 SD coding block, a 16*16 SD coding block, an 8*8 SD coding block and an 8*8 SD intermediate block connected in sequence;

[0023] The 64*64 SD coding block in the control network model is jump-connected with the 64*64 SD decoding block in the stable diffusion model through a zero convolution layer, the 32*32 SD coding block in the control network model is jump-connected with the 32*32 SD decoding block in the stable diffusion model through a zero convolution layer, the 16*16 SD coding block in the control network model is jump-connected with the 16*16 SD decoding block in the stable diffusion model through a zero convolution layer, the 8*8 SD coding block in the control network model is jump-connected with the 8*8 SD decoding block in the stable diffusion model through a zero convolution layer, and the 8*8 SD intermediate block in the control network model is jump-connected with the 8*8 SD intermediate block in the stable diffusion model through a zero convolution layer.

[0024] Optionally, the step of inputting the text prompt word and the conditional vector feature map into a generation model to obtain a partial city layout map corresponding to the design sub-map includes:

[0025] After inputting the conditional vector feature map into the zero convolution layer, it is added to the image potential vector with noise added to obtain a target feature map;

[0026] The text prompt word, the time step and the image potential vector are input into the stable diffusion model, and the text prompt word, the time step and the target feature map are input into the control network model, and a partial city layout map corresponding to the design sub-map is output.

[0027] Optionally, the generative model is obtained by training in the following manner:

[0028] Inputting each city layout design drawing in the plurality of data pairs into the encoding network to obtain a conditional vector feature map of each city layout design drawing;

[0029] Inputting the text prompt words and the conditional vector feature map of each urban layout design map into the original generation model;

[0030] According to the loss function, the weights of the original generation model except the frozen module are adjusted by the back propagation mechanism until the preset conditions are met to obtain the generation model; the frozen module includes the 64*64 SD coding block, the 32*32 SD coding block, the 16*16 SD coding block and the 8*8 SD coding block in the stable diffusion model.

[0031] Optionally, filling the part of the city layout map into the corresponding position in the blank map until the target city layout design map is traversed to obtain the target city layout map includes:

[0032] Calculating the similarity between the RGB value of each pixel in the partial city layout map and all preset RGB values;

[0033] Replace the RGB value of the corresponding pixel point in the partial city layout map with the preset RGB value with the greatest similarity to obtain a corrected partial city layout map;

[0034] The corrected partial city layout map is filled into the corresponding position in the blank map until the target city layout design map is traversed to obtain the target city layout map.

[0035] In a second aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of generating an ultra-high-resolution urban layout using a design drawing as described in the first aspect.

[0036] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and a processor executes the computer program to implement the method of generating an ultra-high-resolution urban layout using a design drawing as described in the first aspect.

[0037] Compared with the prior art, the beneficial effects of this application are:

[0038] The present application proposes a method for generating ultra-high-resolution urban layout using a design drawing, the method comprising: obtaining a target urban layout design drawing drawn by a user; extracting design sub-drawings of preset window sizes from the target urban layout design drawing in sequence according to a preset step size and a preset order; encoding the design sub-drawings into conditional vector feature maps; inputting text prompt words and the conditional vector feature maps into a generation model to obtain a partial urban layout map corresponding to the design sub-drawing; wherein the text prompt words are used to describe the task information of image generation; the generation model is obtained by training based on multiple groups of data pairs; each group of data pairs includes text prompt words, each real urban layout map and a city layout design map corresponding to each real urban layout map; filling the partial urban layout map into the corresponding position in the blank map until the target urban layout design map is traversed to obtain the target urban layout map; the blank map has the same size as the target urban layout design map.

[0039] The present application provides an automated and highly flexible urban layout generation method, which can automatically generate a detailed urban layout map based on a design drawing drawn by a user, ensure that the generated urban layout map is structurally consistent with the design drawing, and can adapt to design drawings of different complexities, thereby further improving the diversity and innovation of urban layout map generation, and overcoming the problem that traditional automated urban generation programs are rule-based and lack flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of a process for generating an ultra-high resolution urban layout using a design drawing provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of a target city layout design provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of the structure of the coding network provided in the embodiment of the present application;

[0043] Figure 4 A schematic diagram of an image processing method provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the structure of the generation model provided in the embodiment of the present application;

[0045] Figure 6 A schematic diagram of a process for generating a city layout diagram based on a design diagram provided in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of generating a city layout map based on a sliding window method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] The method of the embodiment of the present application can be executed in the form of a program by running a program on an existing computer device. The computer device can be a mobile phone, a tablet, a desktop computer, etc. The computer device may include a processor, a storage medium, etc. The storage medium is used to store the program for executing the method of the present embodiment, and the processor is used to load and execute the computer program so that the computer device executes the method for generating an ultra-high-resolution urban layout using a design drawing as provided in the embodiment of the present application.

[0049] In order to solve the problem of lack of flexibility in traditional automated city generation programs, the present application embodiment provides a method for generating ultra-high resolution city layout using design drawings, which can be executed by a computer device. Figure 1 As shown, it is a schematic diagram of the process of the method for generating ultra-high-resolution urban layout using design drawings provided in an embodiment of the present application. The process of the method is introduced as follows.

[0050] S101, obtaining a target city layout design drawing drawn by a user;

[0051] In the specific implementation process, an open source 2D drawing tool, such as Konva, excalidraw, etc., is embedded in the web page. Users can use the drawing tool on the web page to draw the target city layout design containing multiple elements with preset colors and brushes. The multiple elements include urban road trunk roads, road branches, water bodies, vegetation, grassland bare soil, buildings, construction sites, etc.

[0052] Please refer to Figure 2 , is a schematic diagram of a target city layout design diagram provided in an embodiment of the present application, Figure 2 It includes three elements of different colors, namely the main roads in the city, water bodies and vegetation.

[0053] S102: extracting design sub-drawings of preset window sizes from the target city layout design drawing in sequence according to a preset step length and a preset order.

[0054] In the specific implementation process, when drawing the target city layout design map, the user can set the canvas size to Size_Window×Size_Window, (Size_Window-Step)*N (N is a positive integer). Among them, Size_Window is the preset window size, that is, the image size of the input generation model, for example, it can be set to 1024. Step is the size of the generated image filled into the design sub-map when generating an ultra-high-resolution city layout map, for example, it can be set to 128. Size_Window-Step represents the preset step size. The preset order is from left to right and from top to bottom.

[0055] A sliding window method is adopted, with the preset window size as Size_Window and the preset step size as Size_Window-Step. The contents of the window in the target city layout design map are extracted in sequence from left to right and from top to bottom, namely, the design sub-map, recorded as map1.

[0056] S103: Encode the designed sub-graph into a conditional vector feature graph.

[0057] Since the input size of the generative model is 512*512, the design subgraph needs to be encoded as a conditional vector of a 64*64 feature space. Therefore, in a possible embodiment, the design subgraph is input into the encoding network, and a conditional vector feature map of a 64*64 feature space is output; wherein the encoding network includes four convolutional layers connected in sequence; the number of channels of the four convolutional layers are 16, 32, 96 and 256 respectively; and each convolutional layer is connected using an activation function.

[0058] Please refer to Figure 3 , which is a schematic diagram of the structure of the encoding network provided in the embodiment of the present application. Wherein, Conv represents the convolution layer, kernel_size represents the convolution kernel size, step represents the step size, channel represents the number of channels, and Hard Swish represents the activation function. The encoding network includes four convolution layers and four activation layers connected alternately. The four convolution layers all use 3*3 convolution kernels, the step size is 2, and the number of channels is 16, 32, 96 and 256 respectively; the four activation layers all use the Hard Swish activation function.

[0059] In the specific implementation process, the designed subgraph is input into the encoding network, and passes through the first convolution layer (kernel_size=3*3, step=2, channel=16), the first activation layer (Hard Swish), the second convolution layer (kernel_size=3*3, step=2, channel=32), the second activation layer (Hard Swish), the third convolution layer (kernel_size=3*3, step=2, channel=96), the third activation layer (Hard Swish), the fourth convolution layer (kernel_size=3*3, step=2, channel=96) and the fourth activation layer (Hard Swish), and outputs the conditional vector Cf of the 64*64 feature space, that is, the conditional vector feature map.

[0060] In an embodiment of the present application, considering that during the generation process of the stable diffusion model of the generative model, the feature map resolution usually used internally is gradually reduced to 64x64 or even lower, the design sub-map is adjusted to 64x64 to make it consistent with the feature map resolution of the stable diffusion model, which helps to better integrate the conditional information and the generation process.

[0061] S104, inputting the text prompt words and the conditional vector feature map into the generation model to obtain a partial city layout map corresponding to the design sub-map.

[0062] The generative model is obtained by training based on multiple sets of data pairs; each set of data pairs includes text prompts, each real city layout map, and the city layout design map corresponding to each real city layout map. The text prompts are used to describe the task information of image generation to help the generative model generate a city layout map that is more in line with expectations. Text prompts include ada map and city map. City map means generating a city layout map, and ada map emphasizes that the generated city layout map is consistent with the style of the training data (i.e., multiple sets of data pairs).

[0063] In a possible embodiment, each set of data pairs is obtained in the following manner:

[0064] A plurality of real city layout maps are obtained from a map database; various elements in each real city layout map are represented by preset RGB values; a city layout design map corresponding to each real city layout map is obtained by manual drawing; various elements in each city layout design map are represented by preset RGB values; each group of data pairs is obtained according to text prompt words, each real city layout map, and the city layout design map corresponding to each real city layout map.

[0065] In the specific implementation process, OpenStreetMap (OSM for short) is an open source, free map project that aims to create a global geographic information database that allows users to jointly edit and maintain map data. Multiple real city layout maps can be obtained from OpenStreetMap, with an image size of 512*512. These images include elements such as the city's road network, building distribution, and green areas. These elements are divided into various types of elements such as urban road trunks, urban road branches, water bodies, vegetation, grassland bare soil, buildings, and construction sites, and are represented by preset RGB values. In order to maximize the distinction between different elements, the cosine similarity between different types of elements is less than the preset similarity, and the preset similarity is, for example, 0.6.

[0066] For example: Urban layout involves a total of 7 elements, which are represented by 7 preset RGB values. The RGB value of the main road of the city is (R1, G1, B1), the RGB value of the branch road of the city is (R2, G2, B2), the RGB value of the water body is (R3, G3, B3), the RGB value of vegetation is (R4, G4, B4), the RGB value of grassland bare soil is (R5, G5, B5), the RGB value of the building is (R6, G6, B6), and the RGB value of the construction site is (R7, G7, B7).

[0067] Furthermore, by manually drawing, the main roads, water bodies, vegetation and other areas of the city are drawn in a graffiti manner according to the real map of the city layout, and are represented by preset RGB values ​​respectively to obtain the city layout design map corresponding to each real map of the city layout, such as Figure 2 The image size of the city layout design map is also 512*512 pixels.

[0068] In the embodiment of the present application, each element in the real city layout map and the city layout design map is represented by a preset RGB value. In this way, the model can accurately identify and distinguish different elements. Based on this fine labeling, the model can train a feature representation with a high degree of differentiation for these elements, so that when generating a city layout map, higher quality automatic generation can be achieved. And by combining the text prompt words with the paired data of the city layout map, the model can further enhance its ability to generate text to image, greatly improving the automation and intelligence level of urban planning.

[0069] In a possible embodiment, the step of obtaining each set of data pairs according to the text prompt words, each real city layout map, and the city layout design map corresponding to each real city layout map includes:

[0070] Randomly select part of the real city layout pictures from multiple real city layout pictures; cover the upper half of the city layout design pictures corresponding to the part of the real city layout pictures with the upper half of the part of the real city layout pictures, and / or cover the left half of the city layout design pictures corresponding to the part of the real city layout pictures with the left half of the part of the real city layout pictures; obtain each group of data pairs according to the text prompt words, each real city layout picture, and the city layout design pictures corresponding to each real city layout picture.

[0071] In the specific implementation process, 30% of the multiple real city layout maps can be randomly selected as partial real city layout maps, and the urban layout design maps corresponding to the 30% partial real city layout maps can be overwritten in any of the following ways:

[0072] Method 1: Use the upper half of the partial city layout actual map to cover the upper half of the city layout design map corresponding to the partial city layout actual map.

[0073] For example: replace the 1st to 128th lines of the actual city layout map with the 1st to 128th lines of the corresponding city layout design map.

[0074] Method 2: Cover the left half of the urban layout design drawing corresponding to the partial urban layout actual map with the left half of the partial urban layout actual map.

[0075] For example: replace the 1st to 128th columns of a part of the real city layout map with the 1st to 128th columns of the corresponding city layout design map.

[0076] Method three: Use the upper half of the partial city layout real map to cover the upper half of the city layout design map corresponding to the partial city layout real map, and use the left half of the partial city layout real map to cover the left half of the city layout design map corresponding to the partial city layout real map.

[0077] For example: replace the 1st to 128th rows and the 1st to 128th columns of a part of the real city layout map with the 1st to 128th rows and the 1st to 128th columns of the corresponding city layout design map.

[0078] Please refer to Figure 4 , is a schematic diagram of an image processing method provided in an embodiment of the present application, wherein, Figure 4 (a) indicates that the upper half of the real urban layout map is used to cover the corresponding position of the real urban layout map; Figure 4 (b) indicates that the corresponding position of the real urban layout map is covered with the left half of the real urban layout map; Figure 4 (c) indicates that the corresponding positions of the actual urban layout map are covered with the upper and left halves of the actual urban layout map.

[0079] In an embodiment of the present application, an overlay operation is performed on a randomly selected urban layout design drawing to ensure that during the ultra-high resolution image generation process, the upper and left halves of the image remain consistent, thereby maintaining the natural transition and continuity of the image.

[0080] Please refer to Figure 5 , which is a schematic diagram of the structure of the generation model provided in an embodiment of the present application. The generation model includes a stable diffusion model and a control network model (ControlNet), and the stable diffusion model includes a 64*64 SD encoding block 1, a 32*32 SD encoding block 2, a 16*16 SD encoding block 3, an 8*8 SD encoding block 4, an 8*8 SD intermediate block, an 8*8 SD decoding block 4, a 16*16 SD decoding block 3, a 32*32 SD decoding block 2, and a 64*64 SD decoding block 1 connected in sequence.

[0081] The control network model (ControlNet) includes a zero convolution layer (zero convolution), a 64*64 SD coding block 1, a 32*32 SD coding block 2, a 16*16 SD coding block 3, an 8*8 SD coding block 4 and an 8*8 SD intermediate block connected in sequence;

[0082] The 64*64 SD encoding block 1 in the control network model has a skip connection with the 64*64 SD decoding block 1 in the stable diffusion model through a zero convolution layer. The 32*32 SD encoding block 2 in the control network model has a skip connection with the 32*32 SD decoding block 2 in the stable diffusion model through a zero convolution layer. The 16*16 SD encoding block 3 in the control network model has a skip connection with the 16*16 SD decoding block 3 in the stable diffusion model through a zero convolution layer. The 8*8 SD encoding block 4 in the control network model has a skip connection with the 8*8 SD decoding block 4 in the stable diffusion model through a zero convolution layer. The 8*8 SD intermediate block in the control network model has a skip connection with the 8*8 SD intermediate block in the stable diffusion model through a zero convolution layer.

[0083] In a possible embodiment, the generative model is obtained by training in the following manner:

[0084] Each urban layout design drawing in multiple data pairs is input into the encoding network to obtain the conditional vector feature map of each urban layout design drawing; the prompt words in this article and the conditional vector feature map of each urban layout design drawing are input into the original generation model; according to the loss function, the weights of the original generation model except the frozen module are adjusted by the back propagation mechanism until the preset conditions are met to obtain the generation model; the frozen module includes 64*64 SD coding blocks, 32*32 SD coding blocks, 16*16 SD coding blocks and 8*8 SD coding blocks in the stable diffusion model.

[0085] In the specific implementation process, the 64*64 SD coding block, 32*32 SD coding block, 16*16 SD coding block, and 8*8 SD coding block in the stable diffusion model use the pre-trained weights of stablediffusion 2.1 (this is the version number of the stable diffusion model), and the weights of the coding blocks of the stable diffusion model are frozen during the joint training process. The loss function uses the following formula:

[0086]

[0087] Among them, Loss is the loss function, t is the time step, Z t is the image potential vector with noise added at step t, C t For text prompts, C f is the conditional vector feature map, θ is the network parameter learned by the generative model, ∈ is the noise, ∈ θ is the noise predicted by the generative model, Express expectations.

[0088] The training is performed on multiple data pairs, and the back-propagation mechanism is used to update the network weights by calculating the gradient of the loss function Loss to the network parameters. During back-propagation, the frozen coding blocks do not perform weight updates, and only the unfrozen parts perform gradient calculations and updates. The AdamW optimizer is used, and the learning rate is set to the initial value of 1×10 -5 By monitoring the validation loss, save the weight file when the model performs best on the validation set for subsequent inference or fine-tuning.

[0089] In an embodiment of the present application, the weights of the coding blocks in the stable diffusion model are frozen, and only the weights of the unfrozen parts are updated during the joint training process. The frozen coding blocks will not perform gradient calculations and weight updates during back propagation, thereby significantly reducing the computational overhead during model training, and can optimize those layers that need to be updated more quickly, thereby accelerating the entire training process. Freezing the weights of some coding blocks helps to reduce the parameter space of the model so that overfitting will not occur when training on limited data. The pre-trained weights of the stable diffusion model are usually optimized on a large amount of data. Freezing the weights of these coding blocks can maintain the capabilities of the pre-trained model and avoid losing the knowledge that has been learned.

[0090] In a possible embodiment, the generative model generates images in the following manner:

[0091] After the conditional vector feature map is input into the zero convolution layer, it is added to the image latent vector with noise added to obtain the target feature map; the text prompt word, time step and image latent vector are input into the stable diffusion model, and the text prompt word, time step and target feature map are input into the control network model to output the partial city layout map corresponding to the designed sub-map.

[0092] During the specific implementation process, please continue to refer to Figure 5 , the conditional vector feature map C f After inputting the zero convolution layer, the image potential vector Z with noise added in the tth step in the stable diffusion model is t Add together to obtain the target feature map Z'. You can first add the text prompt word C t Input the text encoder, input the time step t into the step encoder, and then input each SD encoding block, each SD intermediate block, and each SD decoding block. After the target feature map passes through different encoding blocks and intermediate blocks in the control network model, feature maps of different resolutions are obtained, which are recorded as F1, F2, F3, F4, and F5 respectively. The feature maps F1, F2, F3, F4, and F5 are connected to SD decoding block 1, SD decoding block 2, SD decoding block 3, SD decoding block 4, and SD intermediate blocks in the stable diffusion model through zero convolution layers and jump connections in turn, and finally output the partial urban layout map corresponding to the design sub-map through SD decoding block 1.

[0093] In the embodiment of the present application, when the stable diffusion model (Stable Diffusion) generates a city layout diagram, the control network model (ControlNet) is used to extract the information of the design diagram, which is used as additional condition information in the generation process to guide the generated city layout diagram to follow the city layout design diagram, thereby ensuring that the structure of the generated image is consistent with the design diagram input by the user. By using the precise control capability of ControlNet, the stable diffusion model can be better guided to generate a city layout diagram that meets the needs of the user, and a highly personalized city layout design can be achieved.

[0094] S105, filling part of the city layout map into the corresponding position in the blank map, until the target city layout design map is traversed to obtain the target city layout map.

[0095] In a possible embodiment, the similarity between the RGB value of each pixel in the partial city layout map and all preset RGB values ​​is calculated; the RGB value of the corresponding pixel in the partial city layout map is replaced with the preset RGB value with the greatest similarity to obtain a corrected partial city layout map; the corrected partial city layout map is filled into the corresponding position in the blank map until the target city layout design map is traversed to obtain the target city layout map.

[0096] In the specific implementation process, a blank map is created, denoted as MapOut, and the RGB values ​​are set to (0, 0, 0). The blank map has the same size as the target city layout design map and is used to save the generated city layout map. All preset RGB values ​​refer to the preset RGB values ​​corresponding to all types of elements in the city layout.

[0097] Please refer to Figure 6 , which is a schematic diagram of a process of generating a city layout map based on a design drawing provided by an embodiment of the present application. The city layout sketch (city layout design drawing) drawn by the user and the prompt words (ada map, city map) are input into the generation model (stable diffusion + ControlNet) to obtain the city layout map, and then the city layout map is color corrected to obtain the corrected city layout map.

[0098] In the embodiment of the present application, considering that the color of the generated image is often affected by the model training data, randomness in the generation process and noise, the color value of the generated city layout map may be different from the preset RGB value. Color correction can be used to adjust the color of the city layout map to the preset value to ensure that the color value is relatively fixed, which is convenient for subsequent processing of the city layout map.

[0099] Please refer to Figure 7, is a schematic diagram of generating a city layout map based on a sliding window method provided in an embodiment of the present application. It can be seen that the preset window size is 1024, the preset step size is 1024-128, and the preset order is from left to right and from top to bottom.

[0100] from Figure 7 It can be seen that there are overlapping areas between adjacent sliding windows. Therefore, in a possible embodiment, after S102, if the design sub-image has an overlapping area with other design sub-images at the corresponding position of the blank image, the image of the overlapping area is copied to the design sub-image, and then S103-S105 are executed.

[0101] In summary, the embodiment of the present application provides a method for generating ultra-high-resolution urban layout using a design drawing. By utilizing the image generation capability of the stable diffusion model (Stable Diffusion) and the precise control capability of the control network model (ControlNet), a detailed urban layout map can be automatically generated based on the design drawing drawn by the user, ensuring that the generated urban layout map is structurally consistent with the design drawing. At the same time, the data set construction and generation method are optimized, and an ultra-high-resolution (greater than 1024x1024) urban layout map can be automatically generated to maintain the continuity of the image.

[0102] In addition, in one embodiment, the present application also provides a computer device, which includes a processor, a memory, and a computer program stored in the memory, and when the computer program is executed by the processor, it implements the aforementioned method of generating an ultra-high-resolution urban layout using a design drawing.

[0103] In addition, in one embodiment, the present application also provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the aforementioned method of generating an ultra-high-resolution urban layout using a design drawing is implemented.

[0104] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0105] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0106] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0107] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating ultra-high resolution urban layout using a design drawing, characterized in that: include: Obtain the target city layout design map drawn by the user; Extracting design sub-graphs of preset window sizes from the target city layout design graph in sequence according to a preset step length and a preset order; encoding the designed subgraph into a conditional vector feature graph; Inputting the text prompt words and the conditional vector feature map into the generation model to obtain a partial city layout map corresponding to the design sub-map; wherein the text prompt words are used to describe the task information of image generation; the generation model is obtained by training based on multiple groups of data pairs; each group of data pairs includes the text prompt words, each real city layout map and the city layout design map corresponding to each real city layout map; The partial city layout map is filled into the corresponding position in the blank map until the target city layout design map is traversed to obtain the target city layout map; the blank map has the same size as the target city layout design map.

2. The method for generating ultra-high resolution urban layout using design drawings as claimed in claim 1, characterized in that: Each data pair is obtained as follows: A plurality of real city layout maps are obtained from a map database; various elements in each real city layout map are represented by preset RGB values; By manually drawing, the urban layout design map corresponding to each urban layout real map is obtained; various elements in each urban layout design map are represented by preset RGB values; Each group of data pairs is obtained according to the text prompt words, each real city layout map, and the city layout design map corresponding to each real city layout map.

3. The method for generating ultra-high resolution urban layout using design drawings as claimed in claim 2, characterized in that: The step of obtaining each set of data pairs according to the text prompt words, each real city layout map, and the city layout design map corresponding to each real city layout map comprises: Randomly selecting some real city layout pictures from the plurality of real city layout pictures; Covering the upper half of the urban layout design drawing corresponding to the partial urban layout real picture with the upper half of the partial urban layout real picture, and / or covering the left half of the urban layout design drawing corresponding to the partial urban layout real picture with the left half of the partial urban layout real picture; Each group of data pairs is obtained according to the text prompt words, each real city layout map, and the city layout design map corresponding to each real city layout map.

4. The method for generating ultra-high resolution urban layout using design drawings as claimed in claim 1, characterized in that: The step of encoding the designed sub-graph into a conditional vector feature graph comprises: The designed subgraph is input into the encoding network, and a conditional vector feature graph of a 64*64 feature space is output; wherein the encoding network includes four convolutional layers connected in sequence; the number of channels of the four convolutional layers are 16, 32, 96 and 256 respectively; and each convolutional layer is connected using an activation function.

5. The method for generating ultra-high resolution urban layout using design drawings as claimed in claim 1, characterized in that: The generation model includes a stable diffusion model and a control network model, wherein the stable diffusion model includes a 64*64 SD encoding block, a 32*32 SD encoding block, a 16*16 SD encoding block, an 8*8 SD encoding block, an 8*8 SD intermediate block, an 8*8 SD decoding block, a 16*16 SD decoding block, a 32*32 SD decoding block, and a 64*64 SD decoding block connected in sequence; The control network model includes a zero convolution layer, a 64*64 SD coding block, a 32*32 SD coding block, a 16*16 SD coding block, an 8*8 SD coding block and an 8*8 SD intermediate block connected in sequence; The 64*64 SD coding block in the control network model is jump-connected with the 64*64 SD decoding block in the stable diffusion model through a zero convolution layer, the 32*32 SD coding block in the control network model is jump-connected with the 32*32 SD decoding block in the stable diffusion model through a zero convolution layer, the 16*16 SD coding block in the control network model is jump-connected with the 16*16 SD decoding block in the stable diffusion model through a zero convolution layer, the 8*8 SD coding block in the control network model is jump-connected with the 8*8 SD decoding block in the stable diffusion model through a zero convolution layer, and the 8*8 SD intermediate block in the control network model is jump-connected with the 8*8 SD intermediate block in the stable diffusion model through a zero convolution layer.

6. The method for generating ultra-high resolution urban layout using design drawings as claimed in claim 5, characterized in that: The step of inputting the text prompt word and the conditional vector feature map into a generation model to obtain a partial city layout map corresponding to the design sub-map includes: After inputting the conditional vector feature map into the zero convolution layer, it is added to the image potential vector with noise added to obtain a target feature map; The text prompt word, the time step and the image potential vector are input into the stable diffusion model, and the text prompt word, the time step and the target feature map are input into the control network model, and a partial city layout map corresponding to the design sub-map is output.

7. The method for generating ultra-high resolution urban layout using design drawings as claimed in claim 5, characterized in that: The generative model is trained in the following way: Inputting each city layout design drawing in the plurality of data pairs into the encoding network to obtain a conditional vector feature map of each city layout design drawing; Inputting the text prompt words and the conditional vector feature map of each urban layout design map into the original generation model; According to the loss function, the weights of the original generation model except the frozen module are adjusted by the back propagation mechanism until the preset conditions are met to obtain the generation model; the frozen module includes the 64*64 SD coding block, the 32*32 SD coding block, the 16*16 SD coding block and the 8*8 SD coding block in the stable diffusion model.

8. The method for generating ultra-high resolution urban layout using design drawings as claimed in claim 1, characterized in that: The step of filling the part of the city layout map into the corresponding position in the blank map until the target city layout design map is traversed to obtain the target city layout map includes: Calculating the similarity between the RGB value of each pixel in the partial city layout map and all preset RGB values; Replace the RGB value of the corresponding pixel point in the partial city layout map with the preset RGB value with the greatest similarity to obtain a corrected partial city layout map; The corrected partial city layout map is filled into the corresponding position in the blank map until the target city layout design map is traversed to obtain the target city layout map.

9. A computer device, characterized in that: The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method for generating an ultra-high-resolution urban layout using a design drawing as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the processor executes the computer program to implement the method for generating an ultra-high-resolution urban layout using a design drawing as described in any one of claims 1-8.

Images