A method, device and medium for generating a building image

Abstract

The application discloses a building image generation method and device and a medium, and relates to the technical field of building image generation. The method comprises the following steps: acquiring an original building design image and a mask image; performing preprocessing on the original building design image according to the mask image to obtain a local building design image and a spliced building design image, and performing synthesis on the mask image according to the local building design image to obtain a synthesized mask image; extracting text features and image features of the local building design image; inputting the text features and the image features into a pre-trained application style model to output a second guide condition; inputting the spliced building design image, the first guide condition and the second guide condition into a diffusion model to obtain an inference building design image; and performing post-processing on the inference building design image according to the mask image to obtain a target building design image. The application has the advantages of supporting multi-modal input and generating high-quality building images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of building image generation, in particular to a building image generation method, device and medium. BACKGROUND

[0002] Building image generation technology is mainly based on deep learning, computer vision and generative models, which can automatically generate or assist in designing building renderings according to input conditions such as text descriptions and sketches.

[0003] Diffusion models, as an important generative model, can support multi-modal inputs such as text, images and style references, thereby generating building renderings that meet user requirements. For example, when the input text description is a villa, the image is a hand-drawn sketch, and the style is modern, the diffusion model can generate the corresponding building rendering.

[0004] However, the current diffusion model still has problems such as style confusion (such as combining Chinese-style roofs with glass curtain walls) and image blurring (such as insufficient image detail levels) when generating building renderings, resulting in poor image quality. SUMMARY

[0005] The present application aims to solve one of the problems in the related art to some extent. To this end, the present application provides a building image generation method, device and medium, which has the advantages of supporting multi-modal input and generating high-quality building images.

[0006] In order to achieve the above-mentioned purpose, as a first aspect of the present application, a building image generation method is provided, wherein the method comprises:

[0007] Obtaining an original building design image and a mask image, the mask image comprising an occlusion region and a target region, the shape of the target region matching at least one local image block in the original building design image;

[0008] Preprocessing the original building design image according to the mask image to obtain a local building design image and a spliced building design image, and synthesizing the mask image according to the local building design image to obtain a synthesized mask image; wherein the synthesized mask image is a first guide condition;

[0009] Extracting text features and image features of the local building design image;

[0010] Inputting the text features and the image features into a pre-trained application style model to output a second guide condition, the second guide condition corresponding to a specific building style;

[0011] input the spliced building design image, the first guidance condition and the second guidance condition into the diffusion model to obtain an inference building design image;

[0012] perform post-processing on the inference building design image according to the mask image to obtain a target building design image.

[0013] Optionally, the inputting the text feature and the image feature into the pre-trained application style model to output the second guidance condition comprises:

[0014] fusing the text feature and the image feature to generate a fusion feature;

[0015] inputting the fusion feature into the application style model to output a specific building style.

[0016] Optionally, the specific building style is selected from style information of a building image, texture information of the building image, environment information of the building image, color and material information of the building image, and form and structure information of the building image.

[0017] Optionally, the inputting the spliced building design image, the first guidance condition and the second guidance condition into the diffusion model to obtain the inference building design image comprises:

[0018] inputting the spliced building design image into a VAE encoder to output an encoding feature of the spliced building design image in a latent space;

[0019] inputting the encoding feature of the latent space, the first guidance condition and the second guidance condition into a U-Net model to output an encoding feature of the latent space of the spliced building design image under the first guidance condition and the second guidance condition;

[0020] inputting the encoding feature of the latent space under the first guidance condition and the second guidance condition into a VAE decoder to obtain the inference building design image.

[0021] Optionally, the inputting the encoding feature of the latent space, the first guidance condition and the second guidance condition into the U-Net model to output the encoding feature of the latent space of the spliced building design image under the first guidance condition and the second guidance condition comprises:

[0022] inputting the encoding feature of the latent space as a basic carrier into the U-Net model;

[0023] inputting the first guidance condition and the second guidance condition as control conditions into the U-Net model to guide the U-Net model to perform inference on the basic carrier;

[0024] The output stitched architectural design image is based on the image features of the potential space under the first and second guiding conditions.

[0025] Optionally, the original architectural design image includes a reference architectural design image and a redrawn architectural design image; the mask image includes a reference mask and a redrawn mask; the reference mask and the reference architectural design image have the same size, and the redrawn mask and the redrawn architectural image have the same size; the partial architectural design image includes a partial reference architectural design image and a partial redrawn architectural design image.

[0026] The process of preprocessing the original architectural design image based on the mask image to obtain a partial architectural design image and a stitched architectural design image, and then synthesizing the mask image based on the partial architectural design image to obtain a composite mask image includes:

[0027] The reference architectural design image is cropped based on the target area of ​​the reference mask, and a partial reference architectural design image is output.

[0028] Based on the target area of ​​the redraw mask, the architectural design image is cropped and redrawn, and the partially redrawn architectural design image is output.

[0029] Based on the size of the partially redrawn architectural design image, the local reference architectural design image is scaled and output to align with the reference architectural design image;

[0030] The aligned reference architectural design image and the partially redrawn architectural design image are stitched together in a row or column direction to output a stitched architectural design image.

[0031] The target area of ​​the redrawn mask and the aligned reference architectural design image are stitched together in a row or column direction to output a composite mask image.

[0032] Optionally, the step of post-processing the inference architectural design image based on the mask image to obtain the target architectural design image includes:

[0033] Cropping the inference architectural design image based on the target area of ​​the redrawn mask, and outputting the cropped inference architectural design image;

[0034] The cropped inference architectural design image is filled into a local image block of the redrawn architectural design image that matches the target area of ​​the redrawn mask, and the target architectural design image is output.

[0035] Optionally, the extraction of text features and image features from the local architectural design image includes:

[0036] When the local reference architectural design image includes text content, the local reference architectural design image is input into the Clip model, and the text features of the local reference architectural design image are output.

[0037] Input a local reference architectural design image into the Clip visual model and output the image features of the local reference architectural design image;

[0038] When the partial reference architectural design image only includes the image content, the partial reference architectural design image is input into the language model, and the text features of the cropped reference architectural design image are output.

[0039] As a second aspect of the present invention, an electronic device is provided, comprising:

[0040] One or more processors;

[0041] A memory having stored one or more computer programs that, when executed by one or more processors, cause the one or more processors to implement the generation method provided in the first aspect of the invention.

[0042] Furthermore, as a third aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the generation method provided in the first aspect of the present invention.

[0043] The architectural image generation method provided by this invention first determines a local architectural design image and a stitched architectural design image using a mask image. Then, it synthesizes the mask image based on the local architectural design image to obtain the first guiding condition of the diffusion model, i.e., the synthesized mask image. Next, it extracts text and image features from the local architectural design image, fuses these features, and inputs them into a pre-trained application style model to obtain the second guiding condition of the diffusion model. Subsequently, it uses the stitched architectural design image as the main input of the diffusion model, and the first and second guiding conditions together serve as the control conditions for the diffusion model. The diffusion model is then used to infer from the stitched architectural design image to obtain an inferred architectural design image. Finally, post-processing operations such as cropping and filling are performed on the inferred architectural design image to generate the target architectural design image. The generation method provided by this invention, with its first guiding condition locking the target area and reducing the influence of non-target areas on the generation result, and the second guiding condition guiding the diffusion model to generate an accurate architectural design image, solves the problems of style inconsistencies and image blurring in the architectural image generated by the diffusion model.

[0044] These features and advantages of the present invention will be disclosed in detail in the following specific embodiments and accompanying drawings. The preferred embodiments or means of the present invention will be shown in detail in conjunction with the accompanying drawings, but are not intended to limit the technical solutions of the present invention. In addition, each of these features, elements and components appearing in the following text and drawings is a plurality of, and different symbols or numbers are used for convenience of representation, but all represent parts with the same or similar construction or function. Attached Figure Description

[0045] The present invention will be further described below with reference to the accompanying drawings:

[0046] Figure 1 A flowchart illustrating a method for generating architectural images provided by this invention;

[0047] Figure 2 A flowchart illustrating one embodiment of step S140 of the generation method provided by the present invention;

[0048] Figure 3 A flowchart illustrating one embodiment of step S150 of the generation method provided by the present invention;

[0049] Figure 4 A flowchart illustrating one embodiment of step S152 of the generation method provided by the present invention;

[0050] Figure 5 A flowchart illustrating one embodiment of step S120 of the generation method provided by the present invention;

[0051] Figure 6 A flowchart illustrating one embodiment of step S160 of the generation method provided by the present invention;

[0052] Figure 7 A flowchart illustrating one embodiment of step S130 of the generation method provided by the present invention;

[0053] Figure 8 A flowchart illustrating one implementation of the generation method provided by this invention;

[0054] Figure 9 A flowchart illustrating another implementation of the generation method provided by the present invention;

[0055] Figure 10 A module diagram of an electronic device provided by the present invention;

[0056] Figure 11 This is a schematic diagram of a computer-readable medium provided by the present invention.

[0057] Explanation of reference numerals in the attached figures

[0058] Among them, 101 is the processor; 102 is the memory; 103 is the I / O interface; and 104 is the bus. Detailed Implementation

[0059] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described are intended to explain the present invention and should not be construed as limiting the invention.

[0060] The terms "an embodiment," "example," or "trademark" used in this specification refer to a particular feature, structure, or characteristic described in connection with the embodiment itself that may be included in at least one embodiment disclosed in this invention. The phrase "in an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0061] Diffusion models, as a generative model, are widely used in the field of architectural image generation. While single-input diffusion models can generate architectural design images that meet the input conditions, they cannot satisfy users' demands for diversity and stylization in the generated images. Diffusion models that support multimodal inputs (such as text, sketches, style references, etc.) suffer from poor image quality due to difficulties in cross-modal semantic alignment (e.g., the ambiguity of sketches versus the precision of text descriptions prevents the model from accurately understanding the relationship between text and sketches; multiple styles cover different types, leading to the model's inability to accurately understand user intent) and imbalanced feature focus (e.g., the diffusion process may focus on irrelevant local features or ignore key structures). These issues result in architectural design images with stylistic inconsistencies, structural deviations, and blurriness.

[0062] In view of this, in order to meet the diverse requirements of users for generated architectural design images and to solve the problem of poor quality of architectural design images generated by diffusion models for multimodal inputs, as a first aspect of the present invention, a method for generating architectural images is provided, such as... Figure 1 As shown, the method includes:

[0063] In step S110, an original architectural design image and a mask image are obtained. The mask image includes an occlusion area and a target area. The shape of the target area matches at least one local image block in the original architectural design image.

[0064] In step S120, the original architectural design image is preprocessed according to the mask image to obtain a partial architectural design image and a stitched architectural design image. The mask image is then synthesized according to the partial architectural design image to obtain a composite mask image. The composite mask image serves as the first guiding condition.

[0065] In step S130, text features and image features of the local architectural design image are extracted;

[0066] In step S140, the text features and the image features are input into the pre-trained application style model, and a second guiding condition is output, which corresponds to a specific architectural style.

[0067] In step S150, the stitched architectural design image, the first guiding condition, and the second guiding condition are input into the diffusion model to obtain the inference architectural design image;

[0068] In step S160, the inference architectural design image is post-processed based on the mask image to obtain the target architectural design image.

[0069] The architectural image generation method provided by this invention first determines a local architectural design image and a stitched architectural design image using a mask image. Then, it synthesizes the mask image based on the local architectural design image to obtain the first guiding condition of the diffusion model, i.e., the synthesized mask image. Next, it extracts the text features and image features of the local architectural design image, fuses the text features and image features, and inputs them into a pre-trained application style model to obtain the second guiding condition of the diffusion model. Subsequently, it uses the stitched architectural design image as the main input of the diffusion model, and the first and second guiding conditions together serve as the control conditions of the diffusion model. It uses the diffusion model to infer the stitched architectural design image to obtain the inferred architectural design image. Finally, it performs post-processing operations such as cropping and filling on the inferred architectural design image to generate the target architectural design image. The generation method provided by this invention has two aspects. First, the guiding condition can lock the target region and reduce the influence of non-target regions on the generation result during the generation process, thus solving the problem of feature focus imbalance in the generation process of the diffusion model. Second, the guiding condition can output accurate style features, thus solving the problem of semantic alignment difficulty of multimodal input. This dual guiding condition, as a control condition, solves the problems of style disorder, image blur and other poor image quality in the architectural image generated by the diffusion model.

[0070] As can be seen from steps S110-S160, the architectural image generation method provided by the present invention mainly includes operations such as preprocessing the input architectural design image, feature extraction, feature fusion, using the first guiding condition and the second guiding condition together as the control conditions of the diffusion model to guide the diffusion model inference, and post-processing the inferred architectural design image.

[0071] The output of the second guiding condition in step S140 is described in detail. As an optional implementation of step S140, such as... Figure 2 As shown, the step of inputting the text features and the image features into a pre-trained application style model and outputting a second guiding condition includes:

[0072] In step S141, the text features and the image features are fused to generate fused features;

[0073] In step S142, the fused features are input into the application style model, and the corresponding specific architectural style is output.

[0074] It should be noted that the applied style model is a self-developed model trained by this invention based on a large amount of textual and image information related to architecture and accurate architectural styles. The specific architectural styles include, but are not limited to, the style information of the architectural images (such as Gothic style, modernist style, Huizhou style architecture, etc.), the texture information of the architectural images (such as the rough texture of stone, the natural texture of wood, the luster and smooth texture of metal, etc.), the environmental information of the architectural images (such as the architectural images being in a natural seaside environment, the architectural images being in a historical district environment, etc.), the color and material information of the architectural images (such as the overall or partial color matching of the architectural images, and the building materials, etc.), and the form and structure information of the architectural images (such as the building's outline gradually tapering from bottom to top, and the internal structure being an atrium design, etc.).

[0075] The architectural image generation method provided by this invention mainly relies on a diffusion model to infer the input architectural image. As an optional implementation of step S150, such as... Figure 3 As shown, the entire reasoning process is explained in detail. The step of inputting the stitched architectural design image, the first guiding condition, and the second guiding condition into the diffusion model to obtain the inferred architectural design image includes:

[0076] In step S151, the stitched architectural design image is input into a VAE (Variational Autoencoder) encoder, and the encoded features of the stitched architectural design image in the latent space are output.

[0077] In step S152, the encoding features of the latent space, the first guiding condition, and the second guiding condition are input into the U-Net model, and the encoded features of the latent space based on the first guiding condition and the second guiding condition are output in the stitched architectural design image.

[0078] In step S153, the encoded features of the potential space based on the first and second guiding conditions are input into the VAE decoder to obtain the inference architectural design image.

[0079] The diffusion model applied in this invention is described in detail below. Before the input image and control conditions are input into the U-Net model, the high-dimensional input stitched architectural design image is encoded into a low-dimensional latent space representation using a VAE encoder. This significantly reduces data storage and computation costs, making the U-Net model more efficient during inference. Secondly, VAE encoding can remove noise and redundant information from the input stitched architectural design image, extracting more meaningful features, which helps the U-Net model generate higher-quality samples during the generation process. Furthermore, VAE can generalize the input stitched architectural design image to a certain extent during encoding, enhancing the generalization ability of the U-Net model. Similarly, VAE decoding after the U-Net model can improve the quality of the generated architectural design image, enhance the robustness of the diffusion model, and provide more flexible generation control.

[0080] After the input data is encoded by the VAE encoder, before it is input into the U-Net model, it is necessary to determine the main operating objects and control conditions of the U-Net model. As one implementation method of step S152, for example... Figure 4 As shown, the step of inputting the encoded features of the latent space, the first guiding condition, and the second guiding condition into the U-Net model, and outputting the encoded features of the latent space of the stitched architectural design image based on the first and second guiding conditions, includes:

[0081] In step S152a, the encoded features of the latent space are used as the basic carrier and input into the U-Net model;

[0082] In step S152b, the first guidance condition and the second guidance condition are used together as control conditions and input into the U-Net model to guide the U-Net model to perform inference on the basic carrier.

[0083] In step S152c, the output spliced ​​architectural design image is based on the encoded features of the potential space under the first guiding condition and the second guiding condition.

[0084] Unlike existing technologies, this invention uses two guiding conditions as control conditions for the U-Net model, so that the U-Net model generates corresponding image features only for the target region.

[0085] The generation method provided by this invention not only makes corresponding improvements in diffusion inference, but also includes many operations in the preprocessing stage. The original architectural design image of this invention includes a reference architectural design image and a redrawn architectural design image; the mask image includes a reference mask and a redrawn mask; the reference mask and the reference architectural design image have the same size, and the redrawn mask and the redrawn architectural image have the same size; the local architectural design image includes a local reference architectural design image and a local redrawn architectural design image; as an optional implementation of step S120, such as... Figure 5 As shown, the step of preprocessing the original architectural design image based on the mask image to obtain a partial architectural design image and a stitched architectural design image, and then compositing the mask image based on the partial architectural design image to obtain a composite mask image, includes:

[0086] In step S121, the reference architectural design image is cropped according to the target area of ​​the reference mask, and a partial reference architectural design image is output.

[0087] In step S122, the architectural design image is cropped and redrawn based on the target area of ​​the redraw mask, and the partially redrawn architectural design image is output.

[0088] In step S123, the local reference architectural design image is scaled based on the size of the locally redrawn architectural design image, and the aligned reference architectural design image is output.

[0089] In step S124, the aligned reference architectural design image and the partially redrawn architectural design image are stitched together in the row or column direction to output the stitched architectural design image;

[0090] In step S125, the target area of ​​the redrawn mask and the aligned reference architectural design image are stitched together in a row or column direction to output a composite mask image.

[0091] It should be noted that the image size of the reference architectural design image and the redrawn architectural design image given in the above embodiments are different, and the sizes of the local image blocks cropped from the target areas of the corresponding masks are also different. Therefore, scaling and alignment operations are required after cropping. In practical applications, if local image blocks of the same size are cropped from the target areas of the corresponding masks, scaling is not required, and they can be directly stitched together to form a stitched architectural design image according to row alignment or height alignment. Thus, the preprocessing process ensures that only the areas of the redrawn architectural design image that need to be regenerated and the areas of the reference architectural design image carrying the architectural style to be generated are retained. Since the first guiding condition, which acts as a mask, is needed in the subsequent diffusion model inference, a synthetic mask of the same size as the stitched architectural design image needs to be generated. Considering that the stitched architectural design image includes the alignment reference architectural design image and the locally redrawn architectural design image, and we only need to focus on the key features in the diffusion model inference, although the synthetic mask is stitched together from the target area of ​​the redrawn mask and the alignment reference architectural design image, it only uses the size information of the alignment reference architectural design image. The alignment reference architectural design image in the synthetic mask is an occlusion area composed of all-black pixel values.

[0092] After the diffusion model completes the inference of the local architectural design image, the generation method provided by this invention further includes post-processing operations. As an optional implementation of the post-processing operation, such as... Figure 6 As shown, the step of post-processing the inference architectural design image based on the mask image to obtain the target architectural design image includes:

[0093] In step S161, the inference architectural design image is cropped according to the target area of ​​the redraw mask, and the cropped inference architectural design image is output.

[0094] In step S162, the cropped inference architectural design image is filled into a local image block of the redrawn architectural design image that matches the target area of ​​the redrawn mask, and the target architectural design image is output.

[0095] In practical applications, in order to further improve the image quality of the target building design image, post-processing operations such as correction and optimization can be added.

[0096] It is also necessary to specifically explain step S130, as an optional implementation of step S130, such as Figure 7 As shown, the extraction of text features and image features from local architectural design images includes:

[0097] In step S131, if the local reference architectural design image includes text content, the local reference architectural design image is input into the Clip (Contrastive Language-Image Pre-Training) model, and the text features of the local reference architectural design image are output.

[0098] In step S132, the local reference architectural design image is input into the Clip visual model, and the image features of the local reference architectural design image are output.

[0099] In step S133, when the local reference architectural design image only includes image content, the local reference architectural design image is input into the language model, and the text features of the local reference architectural design image are output.

[0100] The steps S131-S133 above are intended to illustrate that in order to improve the quality of architectural design images generated by the diffusion model, when there is no text information in the local reference architectural design image or no additional text information input by the user, the language model outputs the text features corresponding to the local reference architectural design image, so as to increase the feature information as much as possible to generate an accurate architectural style.

[0101] The method for generating architectural images provided by the present invention will be further explained below with reference to two embodiments:

[0102] Example 1:

[0103] The following is in conjunction with the appendix Figure 8Example 1 is described in detail below. First, a reference image, the image to be redrawn, a reference image mask, and a redraw mask are obtained. The reference image mask has the same size as the reference image, and the target area coordinates of the reference image mask are used to crop the corresponding image block of the reference image based on the target area coordinates. The redraw mask has the same size as the image to be redrawn, and the target area coordinates of the redraw mask are used to crop the corresponding image block to be redrawn based on the target area coordinates. Then, the sizes of the cropped images are adjusted respectively. The adjusted reference image and the adjusted image to be redrawn are stitched together in a row or column direction, and the VAE encoder of the VAE model is loaded to encode the stitched image. The adjusted reference image mask and the redraw mask are then stitched together in a row or column direction. The image is stitched together, with both stitching operations performed in the same direction. The language model describes the adjusted reference image, and the Clip text encoding of the reference image is output based on the Clip model, while the Clip visual encoding of the reference image is output based on the Clip visual model. The style model is loaded, and the Clip visual encoding and Clip text encoding of the reference image are input into the style model to obtain conditional guidance (i.e., a specific architectural style). The stitched result of the VAE encoding, the reference image mask, and the redrawn mask are then used as conditions for the inpainting model. The conditional guidance and inpainting model conditions are then input into the U-Net model (VAE encoding as input, and the stitched result of the reference image mask and the redrawn mask as the first guiding condition). Figure 8 The condition in the middle is used as the second guiding condition to perform diffusion inference, output the latent space image and use the VAE model to decode it. After the decoded image is cropped, the final building image that meets the user's expectations is obtained and this building image is saved.

[0104] Example 2:

[0105] The following is in conjunction with the appendix Figure 9 Example 2 will be described in detail. Figure 9 The images on the left, from top to bottom, are: a redrawn architectural design image, a redrawn mask, a reference architectural design image, and a reference mask. The redrawn architectural design image and redrawn mask are 2500×2106 pixels in size, while the reference architectural design image and reference mask are 1700×1895 pixels in size. The target area of ​​the redrawn mask is 1024×1152 pixels. The redrawn architectural design image is cropped based on the target area of ​​the redrawn mask, resulting in a cropped redrawn architectural design image of 1024×1152 pixels. Figure 9As can be seen, the cropped redrawn architectural design image mainly includes buildings with a glass curtain wall style. The target area of ​​the reference mask is 1344×1056 pixels. The reference architectural design image is cropped based on the target area of ​​the reference image, resulting in a cropped reference architectural design image of 1344×1056 pixels. It can also be seen that the cropped reference architectural design image mainly includes buildings with green gardens and open staircases. Since the cropped reference architectural design image and the cropped redrawn architectural design image are not the same size, the height of the cropped redrawn architectural design image is used as the reference, and the cropped reference architectural design image is scaled to have the same height as the cropped redrawn architectural design image. Align the reference architectural design image, which is currently 1472×1152 pixels. Then, stitch the aligned reference architectural design image and the cropped redrawn architectural design image, both with the same height, along the height direction to generate a stitched architectural design image of size 2496×1152. This stitched image includes both the area to be generated from the redrawn architectural design image and the area of ​​the reference architectural design image desired by the user. Since the target area of ​​the redrawn mask is 1024×1152 pixels, and the stitched architectural design image is 2496×1152 pixels, to ensure the diffusion model focuses on key features, a [missing information - likely a specific element or feature] can be created to match the stitched architectural design. A composite mask image with the same dimensions (2496×1152 pixels) is generated. Since the composite mask image and the stitched architectural design image have a one-to-one correspondence in pixel coordinates, the pixel values ​​of the target area of ​​the redrawn mask are filled into the corresponding composite mask image. It should be noted that although an alignment reference architectural design image is used in the generation of the composite mask image, it does not depend on the pixel values ​​of the alignment reference architectural design image, but rather on its size. This completes the preprocessing operation. The Clip model is used to describe the cropped reference architectural design image, outputting text features. The Clip visual model is then used to extract image features from the cropped reference architectural design image. Before being input into the U-Net model, the stitched architectural design image is encoded using a VAE encoder to obtain the encoded features of the low-dimensional latent space. The encoded features of the latent space are used as the main input of the U-Net model. The synthesized mask image, text features, and image features are used as control conditions input into the U-Net model to guide the U-Net model to perform inference according to the control conditions. The model outputs the image features of the stitched architectural design image in the latent space. The image features of the latent space are then input into the VAE decoder to obtain a 2496×1152 inferred architectural design image. At this time, the inferred architectural design image includes image blocks of the cropped reference architectural design drawing and the architectural design drawing expected to be generated by the user.Finally, the inference architectural design image is cropped based on the target area of ​​the 1024×1152 redraw mask to obtain the cropped inference architectural design image of 1024×1152. The cropped inference architectural design image is the architectural style expected by the user. Then, this cropped inference architectural design image is filled into the local image block of the redraw architectural design image corresponding to the target area of ​​the redraw mask to generate the target architectural design image of 2048×1728. Figure 9 As can be seen from the generated target architectural design image, the generated target architectural design image includes the style of green gardens and open staircases in the reference architectural design image, and compared with the redrawn architectural design image, there are no problems such as style confusion and unclear boundaries in non-target areas.

[0106] The generation method provided by this invention has two aspects. First, the guiding condition can lock the target region and reduce the influence of non-target regions on the generation result during the generation process, thus solving the problem of feature focus imbalance in the generation process of the diffusion model. Second, the guiding condition can output accurate style features, thus solving the problem of semantic alignment difficulty of multimodal input. This dual guiding condition, as a control condition, solves the problems of style disorder, image blur and other poor image quality in the architectural image generated by the diffusion model.

[0107] As a second aspect of the present invention, an electronic device is provided, such as... Figure 10 As shown, it includes:

[0108] One or more processors 101;

[0109] The memory 102 stores one or more computer programs that, when executed by the one or more processors 101, cause the one or more processors 101 to implement the scheduling method provided according to the first and second aspects of the present invention.

[0110] The tool may also include one or more I / O interfaces 103 connected between the processor 101 and the memory 102, configured to enable information interaction between the processor 101 and the memory 102.

[0111] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit 101 (CPU); the first memory 102 is a device with data storage capabilities, including but not limited to random access memory 102 (RAM, more specifically SDRAM, DDR, etc.), read-only memory 102 (ROM), electrically erasable programmable read-only memory 102 (EEPROM), and flash memory (FLASH); the I / O interface 103 (read-write interface) is connected between the processor 101 and the memory 102, enabling information interaction between the processor 101 and the memory 102, including but not limited to a data bus 104 (Bus).

[0112] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0113] Furthermore, as a third aspect of the present invention, a computer-readable medium is also provided having a computer program stored thereon, such as... Figure 11 As shown, when the computer program is executed by a processor, it implements the generation method provided in the first aspect of the present invention.

[0114] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can implement the methods of any of the above embodiments. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0115] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments above. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.

Claims

1. A method for generating architectural images, characterized in that, The method includes: Obtain an original architectural design image and a mask image. The mask image includes an occlusion area and a target area. The shape of the target area matches at least one local image block in the original architectural design image. The original architectural design image includes a reference architectural design image and a redrawn architectural design image. The mask image includes a reference mask and a redrawn mask. The reference mask has the same size as the reference architectural design image, and the redrawn mask has the same size as the redrawn architectural image. The original architectural design image is preprocessed based on the mask image to obtain a partial architectural design image and a stitched architectural design image. The mask image is then synthesized based on the partial architectural design image to obtain a composite mask image. The composite mask image serves as the first guiding condition. The partial architectural design image includes a partial reference architectural design image and a partial redrawn architectural design image. The preprocessing includes: The reference architectural design image is cropped based on the target area of ​​the reference mask, and a partial reference architectural design image is output. Based on the target area of ​​the redraw mask, the architectural design image is cropped and redrawn, and the partially redrawn architectural design image is output. Based on the size of the partially redrawn architectural design image, the local reference architectural design image is scaled and output to align with the reference architectural design image; The aligned reference architectural design image and the partially redrawn architectural design image are stitched together in a row or column direction to output a stitched architectural design image. The target area of ​​the redrawn mask and the aligned reference architectural design image are stitched together in a row or column direction to output a composite mask image; Extract textual and image features from local reference architectural design images; Text features and image features are input into a pre-trained application style model, which outputs a second guiding condition, which corresponds to a specific architectural style. The stitched architectural design image, the first guiding condition, and the second guiding condition are input into the diffusion model to obtain the inferred architectural design image. The target architectural design image is obtained by post-processing the inference architectural design image based on the mask image.

2. The generation method according to claim 1, characterized in that, The process of inputting text features and image features into a pre-trained application style model and outputting a second guiding condition includes: The text features and the image features are fused to generate a fused feature; The fused features are input into the application style model, and the output corresponds to a specific architectural style.

3. The generation method according to claim 2, characterized in that, The specific architectural style is selected from the architectural image's style information, texture information, environmental information, color and material information, and shape and structure information.

4. The generation method according to claim 1, characterized in that, The process of inputting the stitched architectural design image, the first guiding condition, and the second guiding condition into the diffusion model to obtain the inferred architectural design image includes: The stitched architectural design image is input into the VAE encoder, and the encoded features of the stitched architectural design image in the latent space are output. The encoding features of the latent space, the first guiding condition, and the second guiding condition are input into the U-Net model, and the stitched architectural design image is output based on the encoding features of the latent space under the first and second guiding conditions. The encoded features of the latent space based on the first and second guiding conditions are input into the VAE decoder to obtain the inferred architectural design image.

5. The generation method according to claim 4, characterized in that, The process of inputting the encoded features of the latent space, the first guiding condition, and the second guiding condition into the U-Net model, and outputting the encoded features of the latent space of the stitched architectural design image based on the first and second guiding conditions, includes: The encoded features of the latent space are used as the basic carrier and input into the U-Net model; The first and second guiding conditions are used together as control conditions and input into the U-Net model to guide the U-Net model to perform inference on the basic carrier. The output spliced ​​architectural design image is based on the encoded features of the potential space under the first and second guiding conditions.

6. The generation method according to any one of claims 1 to 5, characterized in that, The post-processing of the inference architectural design image based on the mask image to obtain the target architectural design image includes: Cropping the inference architectural design image based on the target area of ​​the redrawn mask, and outputting the cropped inference architectural design image; The cropped inference architectural design image is filled into a local image block of the redrawn architectural design image that matches the target area of ​​the redrawn mask, and the target architectural design image is output.

7. The generation method according to any one of claims 1 to 5, characterized in that, The extraction of text features and image features from the local reference architectural design image includes: When the local reference architectural design image includes text content, the local reference architectural design image is input into the Clip model, and the text features of the local reference architectural design image are output. Input a local reference architectural design image into the Clip visual model and output the image features of the local reference architectural design image; When the local reference architectural design image only includes the image content, the local reference architectural design image is input into the language model, and the text features of the local reference architectural design image are output.

8. An electronic device, characterized in that, include: One or more processors; A memory having stored one or more computer programs that, when executed by one or more processors, cause the one or more processors to implement the generation method according to any one of claims 1 to 7.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the generation method according to any one of claims 1 to 7.

Images