Building rendering graph generation method and system fused with multi-modal input

Abstract

The invention relates to the field of building visualization, and discloses a multi-modal input fused building rendering graph generation method and system, and the method comprises the steps: obtaining multi-modal input data of a building scene; preprocessing the multi-modal input data and constructing a multi-branch feature extraction module; constructing an improved diffusion model backbone network; and starting a generation process of the improved diffusion model backbone network, and iteratively generating a building rendering graph from the random noise image. According to the technical scheme, the generation of the high-quality building rendering graph is realized through multi-modal input fusion and multi-level control.

Description

Technical Field

[0001] This application relates to the field of architectural visualization, and in particular to a method and system for generating architectural renderings that integrates multimodal inputs. Background Technology

[0002] Architectural visualization is typically achieved through traditional 3D rendering techniques. Designers first build a complete 3D architectural model, assigning material textures and lighting environments, and then use algorithms such as ray tracing to render realistic renderings. Traditional rendering methods can accurately reproduce design details, resulting in highly realistic visual effects. However, this process is often time-consuming and labor-intensive, requiring advanced technical skills and advanced hardware. Designers need to perform complex modeling and parameter adjustments, and rendering a high-quality architectural rendering can take several hours or even longer. Therefore, relying solely on traditional rendering to create architectural visualizations suffers from long lead times and low efficiency.

[0003] In recent years, emerging AI image generation technologies (represented by diffusion models such as StableDiffusion) have provided new approaches to architectural visualization. Using pre-trained diffusion models, a corresponding architectural scene image can be automatically generated within seconds simply by inputting a text description or providing a sample image. This generative model-based approach is extremely fast, significantly lowering the production threshold and allowing designers to quickly produce multiple renderings for conceptual design and communication.

[0004] However, AI-generated images based on pure text or a single modality have significant shortcomings: on the one hand, the generated images often lack accuracy and may contain detail deviations or distortions that do not match the actual design; on the other hand, because the model is learned from large-scale data, its output sometimes exhibits uncontrollable flaws or artifacts, such as missing building components or abnormal material textures. Therefore, current AI-generated architectural images are more suitable for the conceptual sketch stage and cannot completely replace traditional rendering for the final presentation that requires accuracy and consistency.

[0005] Solving this technical problem is a technical challenge that needs to be overcome by those skilled in the art. Summary of the Invention

[0006] This application provides a method for generating architectural renderings that integrates multimodal inputs, so as to at least partially solve the above-mentioned technical problems.

[0007] To achieve the above objectives, according to a first aspect of this application, a method for generating architectural renderings that integrates multimodal inputs is provided, comprising: Acquire multimodal input data of the architectural scene; the multimodal input data includes depth map, semantic segmentation map, basic color map, roughness map and text prompt words; The multimodal input data is preprocessed and a multi-branch feature extraction module is constructed. The multi-branch feature extraction module includes a depth branch, a semantic branch, a color branch, a material branch, and a text branch. The depth branch extracts multi-scale spatial structure features; the semantic branch extracts semantic features; the color branch extracts color features; the material branch extracts material features; and the text branch encodes text prompts into semantic vectors. An improved diffusion model backbone network is constructed. This improved backbone network is based on a U-shaped network structure, which includes an encoder, a bottleneck, and a decoder. Multiple feature injection interfaces are set at the initial stage, intermediate stage, and later stage of the encoder in the U-shaped network structure. The intermediate stage is the connection interval between the end of the encoder, the bottleneck, and the beginning of the decoder. Each multiple feature injection interface is connected to the output of the depth branch, semantic branch, color branch, and material branch of the multi-branch feature extraction module. The output of the text branch establishes a cross-modal attention connection with each layer of the U-shaped network structure. Features extracted from each branch are injected into the improved diffusion model backbone network according to preset corresponding stages through the multiple feature injection interfaces. After injection, the features of each branch are fused with the feature maps of the corresponding stages of the improved diffusion model backbone network. The semantic vectors of the text branch are applied to the calculation of each layer of the U-shaped network structure through a cross-modal attention mechanism, adjusting the feature weights of the improved diffusion model backbone network. The generation process of the improved diffusion model backbone network is initiated, iteratively generating architectural renderings starting from random noise images.

[0008] Optionally, acquire multimodal input data of the architectural scene, including: Export the depth map, semantic segmentation map, basic color map, and roughness map of the building scene from the current viewpoint using architectural modeling software or rendering engine; wherein, the semantic segmentation map is automatically annotated by the modeling software; the basic color map is exported from the diffuse reflection attribute parameters of the building material, and the roughness map is exported from the surface roughness attribute parameters of the building material. The system receives text prompts input by the user. These prompts include environmental descriptions, style requirements, and detailed features of the architectural scene. The text prompts are used to define the overall atmosphere and key local features of the generated image.

[0009] Optionally, the multimodal input data is preprocessed, including: The depth values ​​of the depth map are linearly scaled to fit the range of the neural network input, eliminating the differences in depth scale between different scenes; One-hot encoding is performed on the semantic segmentation graph to convert the category labels into a multi-channel semantic mask tensor, with each channel corresponding to a single category; The pixel values ​​of the base color map and roughness map are normalized to the range that fits the neural network input. Remove special symbols and meaningless characters from text prompts; if the text length exceeds a preset threshold, truncate it. Extract the core semantic fragment and input it into the text branch.

[0010] Optionally, the depth branch, semantic branch, color branch, and material branch all employ convolutional neural networks; each layer of the convolutional neural network includes a convolutional layer, a batch normalization layer, and an activation function layer; the convolutional layer is used to extract local detail features or global structural features of the corresponding image modality; the batch normalization layer is used to stabilize the network training process; the activation function layer is used to introduce nonlinear mapping, and the activation function layer uses the ReLU activation function or the GELU activation function; the depth branch, semantic branch, color branch, and material branch achieve multiple downsampling through pooling layers; The text branch uses a pre-trained text encoder, which is a CLIP text encoder. After encoding the text prompts into semantic vectors, the CLIP text encoder optimizes the distribution of semantic vectors through normalization operations, so that the semantic vectors are adapted to image features in attention calculation.

[0011] Optionally, the output of the text branch establishes a cross-modal attention connection with each layer of the U-shaped network structure, including: In the U-shaped network structure, each layer of the encoder, the bottleneck section, and each layer of the decoder are equipped with an independent cross-attention layer; the cross-attention layer is connected in series with the convolutional feature calculation module of the corresponding layer. The semantic vector output by the text branch is mapped to a query vector that matches the length of the corresponding layer's image feature sequence through a fully connected layer. The image feature map of the corresponding layer is flattened into a feature sequence, which serves as both the key vector and value vector of the cross-attention layer. The attention weights of the query vector and the key vector are calculated using an attention algorithm, and the attention weights are normalized to obtain the weight matrix. The weight matrix and the value vector are weighted and summed to obtain the image features that fuse textual semantics, and the original image features of the corresponding layer are replaced with the image features that fuse textual semantics.

[0012] Optionally, the U-shaped network structure specifically includes: The encoder section includes several downsampling units; each downsampling unit includes a convolutional layer, a batch normalization layer, and an activation function layer; the input image features are downsampled sequentially through several downsampling units to gradually reduce the feature map resolution in order to extract high-rise abstract features of the building scene; The bottleneck section includes several convolutional units and a self-attention module; the convolutional units are used to fuse local information of various modal features, and the self-attention module captures the global dependencies of architectural scene features by calculating the correlation weights within the feature sequence. The decoder section includes several upsampling units, each consisting of a transposed convolutional layer, a normalization layer, and an activation function layer. The feature map output from the bottleneck section is upsampled sequentially by several upsampling units to gradually restore the feature map resolution. Each upsampling unit is fused with the feature map of the corresponding layer of the encoder through a channel splicing operation.

[0013] Optionally, the process of initiating the generation of the improved diffusion model backbone network, iteratively generating architectural renderings starting from a random noise image, includes: Initialize a random noise image; the size of the random noise image is consistent with the size of the input layer of the improved diffusion model backbone network; The number of iterations is set; the number of iterations is determined based on the generation efficiency requirements and image quality requirements of the architectural rendering: Perform the following operations in each iteration: S701. Input the current random noise image into the improved diffusion model backbone network, and extract the multi-scale features of the current noise image through the downsampling unit of the encoder. S702. According to the preset corresponding stage, inject the features extracted from the depth branch, semantic branch, color branch, and material branch through the multi-path feature injection interface; after completing the feature fusion, integrate the semantic vector of the text branch through the cross attention layer of each layer, and adjust the feature weights of the improved diffusion model backbone network. S703. Based on the noise prediction value output by the backbone network of the improved diffusion model, and combined with the attenuation coefficient of the current iteration step, reduce the noise intensity in the random noise image. S704. Determine if the current iteration count has reached the preset iteration count: If not, use the updated random noise image as the input for the next iteration and return to S701; if it has reached the preset iteration count, stop the iteration and output the final generated architectural rendering.

[0014] Optionally, the feature weights of the improved diffusion model backbone network can be adjusted, including: In each iteration of the improved diffusion model backbone network, an adaptive weight adjustment process is initiated; The similarity between the generated image and the input data of each modality is calculated. Specifically, the structural similarity index is used to calculate the similarity between the generated image and the depth map and semantic segmentation map to evaluate the matching degree of spatial structure and semantic layout; the peak signal-to-noise ratio index is used to calculate the similarity between the generated image and the basic color map and roughness map to evaluate the matching degree of color distribution and material features; and the cosine similarity index is used to calculate the similarity between the generated image and the semantic vector of the text prompt words to evaluate the matching degree of text semantic guidance. The weight coefficients of each branch feature are dynamically adjusted based on the calculated similarity: if the similarity between the input data of a certain modality and the generated image is higher, the weight coefficient of the corresponding branch feature is increased; if the similarity between the input data of a certain modality and the generated image is lower, the weight coefficient of the corresponding branch feature is decreased. The adjusted feature weight coefficients of each branch are applied to the feature fusion process of the improved diffusion model backbone network.

[0015] Optionally, after iteratively generating the architectural renderings, the method further includes: A nonlocal mean denoising algorithm is used to remove high-frequency noise from the generated architectural renderings. A contrast-limited adaptive histogram equalization algorithm is used to adjust the local color contrast of architectural renderings, thereby improving the visual clarity of architectural component details. The Canny edge detection algorithm is used to extract the edge features of the building outline in the architectural rendering image; the extracted edge features are then fused with the original architectural rendering image according to preset weights to optimize the sharpness of the building outline.

[0016] According to a second aspect of this application, a system for generating architectural renderings that integrates multimodal inputs is provided, comprising: The first processing module is used to: acquire multimodal input data of the building scene; the multimodal input data includes depth map, semantic segmentation map, basic color map, roughness map and text prompt words; The second processing module is used to: preprocess the multimodal input data and construct a multi-branch feature extraction module; the multi-branch feature extraction module includes a depth branch, a semantic branch, a color branch, a material branch, and a text branch; the depth branch extracts multi-scale spatial structure features; the semantic branch extracts semantic features; the color branch extracts color features; the material branch extracts material features; and the text branch encodes text prompts into semantic vectors. The third processing module is used for: constructing an improved diffusion model backbone network; the improved diffusion model backbone network is based on a U-shaped network structure, which includes an encoder part, a bottleneck part, and a decoder part; setting multiple feature injection interfaces in the initial stage, intermediate stage, and late stage of the encoder part of the U-shaped network structure; the intermediate stage is the connection interval between the end of the encoder part, the bottleneck part, and the beginning of the decoder part; the multiple feature injection interfaces are respectively connected to the output ends of the depth branch, semantic branch, color branch, and material branch of the multi-branch feature extraction module, and the output end of the text branch establishes a cross-modal attention connection with each layer of the U-shaped network structure; injecting the features extracted from each branch into the improved diffusion model backbone network according to the preset corresponding stage through the multiple feature injection interfaces; after each branch feature is injected through the multiple feature injection interfaces, it is fused with the feature map of the corresponding stage of the improved diffusion model backbone network, and the semantic vector of the text branch is applied to the calculation of each layer of the U-shaped network structure through the cross-modal attention mechanism to adjust the feature weights of the improved diffusion model backbone network; The fourth processing module is used to: initiate the generation process of the improved diffusion model backbone network, and iteratively generate architectural renderings starting from random noise images.

[0017] In summary, the embodiments of this application, through the above technical solutions, achieve the generation of high-quality architectural renderings by multimodal input fusion and multi-level control. The generated renderings combine the structural accuracy and detail credibility of traditional renderings with the high efficiency and diversity of AI image generation.

[0018] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the steps of a method for generating architectural renderings that integrates multimodal inputs, provided in an exemplary embodiment of this application. Figure 2 This is a schematic diagram of a building rendering system that integrates multimodal input, provided in an exemplary embodiment of this application. Explanation of reference numerals in the attached drawings: 01, First processing module; 202, Second processing module; 203, Third processing module; 204, Fourth processing module. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0022] This application provides a method for generating architectural renderings that integrates multimodal inputs. Please refer to [link / reference]. Figure 1 The method for generating architectural renderings that integrates multimodal inputs, provided in this application embodiment, includes the following steps: Step 101: Obtain multimodal input data of the building scene; the multimodal input data includes depth map, semantic segmentation map, basic color map, roughness map and text prompt words.

[0023] Step 102: Preprocess the multimodal input data and construct a multi-branch feature extraction module; the multi-branch feature extraction module includes a depth branch, a semantic branch, a color branch, a material branch, and a text branch; the depth branch extracts multi-scale spatial structure features; the semantic branch extracts semantic features; the color branch extracts color features; the material branch extracts material features; the text branch encodes text prompts into semantic vectors.

[0024] Step 103: Construct an improved diffusion model backbone network. The improved diffusion model backbone network is based on a U-shaped network structure, which includes an encoder, a bottleneck, and a decoder. Multiple feature injection interfaces are set at the initial stage, intermediate stage, and later stage of the encoder in the U-shaped network structure. The intermediate stage is the connection interval between the end of the encoder, the bottleneck, and the beginning of the decoder. The multiple feature injection interfaces are connected one-to-one with the outputs of the depth branch, semantic branch, color branch, and material branch of the multi-branch feature extraction module. The output of the text branch establishes a cross-modal attention connection with each layer of the U-shaped network structure. Features extracted from each branch are injected into the improved diffusion model backbone network according to preset corresponding stages through the multiple feature injection interfaces. After injection, the features of each branch are fused with the feature maps of the corresponding stages of the improved diffusion model backbone network. The semantic vectors of the text branch are applied to the calculation of each layer of the U-shaped network structure through a cross-modal attention mechanism, adjusting the feature weights of the improved diffusion model backbone network.

[0025] Step 104: Start the generation process of the improved diffusion model backbone network, and iteratively generate architectural renderings starting from random noise images.

[0026] Specifically, a depth map is an image reflecting the distance of each point in the architectural scene from the camera. It is used to strictly constrain the spatial structure and perspective relationships of the generated image. In this embodiment, the depth map of the architectural scene from the current viewpoint can be exported using architectural design software such as Revit and SketchUp, or rendering engines such as V-Ray and Arnold. A semantic segmentation map is an image that labels the semantic categories of different areas in the architectural scene. Each category is usually coded with different colors or labels. It can be automatically generated based on the architectural model using design software, or it can be manually edited and labeled by the designer. The aforementioned semantic categories include the sky, the main building, windows, and greenery. A base color map, also known as a diffuse map, is an image that provides the base color information of each surface of the building. It does not include lighting and darkness factors and can be directly exported from the diffuse attribute parameters in the building material settings. It is used to control the surface color tone of the generated image. A roughness map is an image describing the surface roughness of the building material. It is usually represented by grayscale values. The lower the value, the smoother the surface and the more obvious the highlights. The higher the value, the stronger the diffuse reflection and the weaker the specular reflection. The roughness map of each surface can be exported from the building material parameters and used to adjust the light and shadow reflection effects of the generated image. Text prompts: These are textual descriptions provided by the user that depict the scene environment, art style, and detailed features of the desired architectural rendering.

[0027] Existing technologies for AI-generated architectural renderings face three core technical challenges: 1. Difficulty in fusing multimodal control signals, unable to simultaneously process multiple types of inputs such as depth, semantics, and materials, and the fusion of different signals is prone to conflict; 2. Single-modal constraints can easily lead to generated results deviating from design expectations; 3. Mismatch between the content of the generated image region and semantic annotations or text descriptions.

[0028] This application constructs a multi-branch feature extraction module, designing dedicated processing branches for each modal input. The depth branch extracts spatial structure, the semantic branch extracts region categories, the material branch extracts surface attributes, and the text branch encodes style semantics, extracting core features of various modalities and avoiding conflicts caused by direct fusion of multiple signals. Feature injection interfaces are set at different levels of the improved U-shaped network structure. Deep feature constraints are injected in the initial stage of the encoder, semantic features are injected in the intermediate stage to guide the layout, and material features are injected in the later stage of the decoder to optimize details. Cross-modal attention ensures that text semantics are integrated throughout the entire process, forming multi-dimensional and multi-modal constraints to solve the problem of weak generation controllability. The semantic branch's definition of region categories and guidance of text semantics ensures that the content of each region in the generated image matches the semantic annotations and text descriptions, solving the problem of inconsistency between graphics and semantics.

[0029] The above solution combines the accuracy of traditional rendering with the efficiency of AI generation in the generated architectural renderings. The solution can directly use intermediate data in the traditional architectural rendering process as input, achieving seamless integration with the existing design process, eliminating manual rendering and post-processing steps, and greatly improving the efficiency of architectural rendering production.

[0030] In some embodiments, acquiring multimodal input data of an architectural scene includes: Export the depth map, semantic segmentation map, basic color map, and roughness map of the building scene from the current viewpoint using architectural modeling software or rendering engine; wherein, the semantic segmentation map is automatically annotated by the modeling software; the basic color map is exported from the diffuse reflection attribute parameters of the building material, and the roughness map is exported from the surface roughness attribute parameters of the building material. The system receives text prompts input by the user. These prompts include environmental descriptions, style requirements, and detailed features of the architectural scene. The text prompts are used to define the overall atmosphere and key local features of the generated image.

[0031] In some embodiments, preprocessing of the multimodal input data includes: The depth values ​​of the depth map are linearly scaled to fit the range of the neural network input, eliminating the differences in depth scale between different scenes; One-hot encoding is performed on the semantic segmentation graph to convert the category labels into a multi-channel semantic mask tensor, with each channel corresponding to a single category; The pixel values ​​of the base color map and roughness map are normalized to the range that fits the neural network input. Remove special symbols and meaningless characters from text prompts; if the text length exceeds a preset threshold, truncate it. Extract the core semantic fragment and input it into the text branch.

[0032] Specifically, due to differences in the original format and numerical scale of data from different modalities—for example, depth map depth values ​​can vary by several orders of magnitude depending on scene size, semantic segmentation maps exist in the form of category labels, and text prompts may contain invalid characters—directly using these as inputs can lead to feature extraction bias. Therefore, preprocessing of multimodal input data is necessary to ensure that its format and scale are consistent and adaptable to the input requirements of the multi-branch feature extraction module and the improved diffusion model backbone network. Linearly scaling the depth map depth values ​​to a range suitable for neural network input eliminates depth scale differences between different architectural scenes, avoids large-scale depth values ​​interfering with feature extraction or causing abnormal model gradients, and ensures that depth information from different scenes is comparable at the same scale.

[0033] One-hot encoding is performed on the semantic segmentation graph to convert category labels such as sky, building body, window, and green plants into multi-channel semantic mask tensors, with each channel corresponding to a single category; the above one-hot encoding is an encoding method that converts discrete category labels into binary vectors.

[0034] Normalizing the pixel values ​​of the base color map and roughness map to fit the range of neural network input means mapping the original values ​​of image pixels to a fixed range of 0-1 through a linear transformation.

[0035] Special symbols and meaningless characters are removed from text prompts. If the text length exceeds a preset threshold, the core semantic segment is extracted before inputting the text branch. Special symbols and meaningless characters refer to characters in the text that do not carry effective semantics, such as special symbols, extra spaces, garbled characters, and repeated meaningless words. These characters can interfere with the semantic extraction accuracy of the pre-trained text encoder. The preset threshold is the upper limit of text length set according to the input length limit of the pre-trained text encoder used. For example, the CLIP text encoder usually limits the number of input characters to no more than 77. Exceeding this threshold will cause the encoder to be unable to encode completely.

[0036] In some embodiments, the depth branch, semantic branch, color branch, and material branch all employ convolutional neural networks; each layer of the convolutional neural network includes a convolutional layer, a batch normalization layer, and an activation function layer; the convolutional layer is used to extract local detail features or global structural features of the corresponding image modality; the batch normalization layer is used to stabilize the network training process; the activation function layer is used to introduce nonlinear mapping, and the activation function layer uses the ReLU activation function or the GELU activation function; the depth branch, semantic branch, color branch, and material branch achieve multiple downsampling through pooling layers; The text branch uses a pre-trained text encoder, which is a CLIP text encoder. After encoding the text prompts into semantic vectors, the CLIP text encoder optimizes the distribution of semantic vectors through normalization operations, so that the semantic vectors are adapted to image features in attention calculation.

[0037] Specifically, to extract core features from multimodal input data and ensure that information from each modality can be utilized by the improved diffusion model backbone network, this application constructs a multi-branch feature extraction module. Based on the characteristics of different modalities, it designs a depth branch, a semantic branch, a color branch, and a material branch. These four branches are collectively referred to as the image branch and text branch network structure. The depth branch, semantic branch, color branch, and material branch all employ convolutional neural networks, and each layer of each convolutional neural network includes a convolutional layer, a batch normalization layer, and an activation function layer, balancing feature extraction accuracy and network training stability.

[0038] Convolutional layers are used to extract local detail features or global structural features of the corresponding image modality. Local detail features refer to information about small areas in the image, such as the texture details of the wall in the base color image and the graininess of the material in the roughness image. Global structural features refer to the overall spatial layout information of the image, such as the floor height ratio of buildings in the depth image and the division of buildings and sky in the semantic segmentation image. Convolutional layers use convolutional kernels of preset size, such as 3×3 or 5×5 convolutional kernels, to slide and calculate on the image, aggregating local pixel information into feature values.

[0039] Batch normalization layers are used to stabilize the network training process. By standardizing the mean and variance of each batch of input feature maps, batch normalization layers eliminate the distribution differences between different batches of data, avoid gradient vanishing or training oscillation problems caused by excessive fluctuations in feature values ​​during training, and accelerate the convergence speed of the network.

[0040] Activation function layers are used to introduce nonlinear mappings. Image modalities have nonlinear characteristics, such as the correlation between depth values ​​and spatial structures, and the matching relationship between color and material. Activation functions are needed to break the linear limitations of the network so that it can fit complex feature patterns. This application uses the ReLU activation function or the GELU activation function: the ReLU activation function introduces nonlinearity by setting negative input values ​​to 0 and keeping positive input values ​​unchanged, which can effectively alleviate the gradient vanishing problem; the GELU activation function achieves nonlinear activation through Gaussian probability weighting.

[0041] To enable each image branch to output multi-scale features and adapt to the feature fusion requirements of different layers of the improved diffusion model backbone network, the depth branch, semantic branch, color branch, and material branch implement multiple downsampling through pooling layers. The aforementioned pooling layer refers to a network layer that selects the maximum or average value of a local region of the feature map to lower the feature map resolution, thereby compressing the amount of data while retaining key features. For example, the pooling layer of the depth branch can retain building outline features, and the pooling layer of the semantic branch can retain region category features. Multiple downsampling refers to mapping the original image features to 2-3 different resolution feature maps through 2-3 pooling operations. For example, the 512×512 depth map features are downsampled to 256×256 and 128×128 resolutions.

[0042] Each image branch outputs target features: the depth branch extracts multi-scale spatial structure features to constrain the geometric outline of the building; the semantic branch extracts multi-scale semantic features to guide the layout of regional content; the color branch extracts multi-scale color features to control the surface color distribution; and the material branch extracts multi-scale material features to adjust the light and shadow texture.

[0043] The text branch uses a pre-trained text encoder, which is the CLIP text encoder; the CLIP text encoder encodes text prompts into semantic vectors.

[0044] In some embodiments, the output of the text branch establishes a cross-modal attention connection with each layer of the U-shaped network structure, including: In the U-shaped network structure, each layer of the encoder, the bottleneck section, and each layer of the decoder are equipped with an independent cross-attention layer; the cross-attention layer is connected in series with the convolutional feature calculation module of the corresponding layer. The semantic vector output by the text branch is mapped to a query vector that matches the length of the corresponding layer's image feature sequence through a fully connected layer. The image feature map of the corresponding layer is flattened into a feature sequence, which serves as both the key vector and value vector of the cross-attention layer. The attention weights of the query vector and the key vector are calculated using an attention algorithm, and the attention weights are normalized to obtain the weight matrix. The weight matrix and the value vector are weighted and summed to obtain the image features that fuse textual semantics, and the original image features of the corresponding layer are replaced with the image features that fuse textual semantics.

[0045] Specifically, this application establishes cross-modal attention connections across all layers of a U-shaped network structure, deeply integrating text semantics into the feature calculation process of each layer. Specifically, independent cross-attention layers are set in each layer of the encoder, the bottleneck section, and each layer of the decoder in the U-shaped network structure. These cross-attention layers differ from self-attention layers that only process single-modal features; they establish a mapping relationship between text semantics and image features, ensuring that the style and details of the text description can be directed to the corresponding image regions.

[0046] The cross-attention layer is concatenated with the corresponding convolutional feature calculation module. This concatenation means that the calculation steps of the cross-attention layer are embedded within the convolutional feature calculation process. Specifically, image features of the current layer are first extracted through convolutional layers and batch normalization layers, then input into the cross-attention layer to fuse textual semantics, and finally, the fused features are passed to the next layer. This ensures that textual semantics participate in the constraint during the generation of image features at each layer, avoiding the accumulation of semantic biases in subsequent layers due to the lack of semantic guidance from initial features. For example, when the encoder extracts global structural features of a building, the semantics of the text "modern minimalist style" can be incorporated, avoiding the generation of complex and redundant decorative structures; when the decoder restores wall texture details, the semantics of "clear stone texture" in the text can be used to directionally optimize texture representation.

[0047] The semantic vector output from the text branch is mapped to a query vector with a length matching the image feature sequence of the corresponding layer through a fully connected layer. The fully connected layer is a network component that realizes vector dimension transformation. Its role is to solve the problem of incompatibility between the text semantic vector and the image feature sequence length: the semantic vector output from the text branch, such as the 768-dimensional vector generated by the CLIP encoder, has a fixed dimension, while the image features of different layers of the U-shaped network have different graph resolutions. The fully connected layer maps the semantic vector to a query vector with a length consistent with the image feature sequence of the current layer.

[0048] The image feature maps of the corresponding layers are flattened into feature sequences, which simultaneously serve as the key and value vectors of the cross-attention layer. Attention weights between the query vector and the key vector are calculated using an attention algorithm. Specifically, the dot product of the query vector (textual semantics) and the key vector (image features) is first calculated, then divided by the square root of the feature dimension to obtain an initial relevance score. This score represents the degree of matching between the textual semantics and the corresponding image features. For example, the dot product of the query vector for the text "a glass curtain wall reflecting a sunset" and the key vector for the glass area features in the image will have a significantly higher score than that for the wall area, ensuring that textual details are directed to the target area.

[0049] Attention weights are normalized to obtain a weight matrix. Specifically, the Softmax function is used for normalization, which converts the initial relevance scores into probability weights summing to 1, resulting in a more reasonable weight distribution: high-matching image features receive higher weights, and low-matching features receive lower weights, preventing irrelevant image features from being excessively influenced by text semantics. The weight matrix and value vector are weighted and summed to obtain image features incorporating text semantics. These incorporating text semantics image features replace the original image features in the corresponding layers. In the encoder stage, the replaced features guide high-level abstract features to align with the text style; in the bottleneck stage, they ensure that semantic constraints are carried during global feature fusion; and in the decoder stage, they can be used to optimize the matching degree between detailed features and text descriptions. This solves the problem in existing technologies where text semantics are only superficially superimposed on image features, enabling text to guide the generation process.

[0050] In some embodiments, the U-shaped network structure specifically includes: The encoder section includes several downsampling units; each downsampling unit includes a convolutional layer, a batch normalization layer, and an activation function layer; the input image features are downsampled sequentially through several downsampling units to gradually reduce the feature map resolution in order to extract high-rise abstract features of the building scene; The bottleneck section includes several convolutional units and a self-attention module; the convolutional units are used to fuse local information of various modal features, and the self-attention module captures the global dependencies of architectural scene features by calculating the correlation weights within the feature sequence. The decoder section includes several upsampling units, each consisting of a transposed convolutional layer, a normalization layer, and an activation function layer. The feature map output from the bottleneck section is upsampled sequentially by several upsampling units to gradually restore the feature map resolution. Each upsampling unit is fused with the feature map of the corresponding layer of the encoder through a channel splicing operation.

[0051] In some embodiments, the process of initiating the generation of the improved diffusion model backbone network, iteratively generating architectural renderings starting from a random noise image, includes: Initialize a random noise image; the size of the random noise image is consistent with the size of the input layer of the improved diffusion model backbone network; The number of iterations is set; the number of iterations is determined based on the generation efficiency requirements and image quality requirements of the architectural rendering: Perform the following operations in each iteration: S701. Input the current random noise image into the improved diffusion model backbone network, and extract the multi-scale features of the current noise image through the downsampling unit of the encoder. S702. According to the preset corresponding stage, inject the features extracted from the depth branch, semantic branch, color branch, and material branch through the multi-path feature injection interface; after completing the feature fusion, integrate the semantic vector of the text branch through the cross attention layer of each layer, and adjust the feature weights of the improved diffusion model backbone network. S703. Based on the noise prediction value output by the backbone network of the improved diffusion model, and combined with the attenuation coefficient of the current iteration step, reduce the noise intensity in the random noise image. S704. Determine if the current iteration count has reached the preset iteration count: If not, use the updated random noise image as the input for the next iteration and return to S701; if it has reached the preset iteration count, stop the iteration and output the final generated architectural rendering.

[0052] Specifically, the random noise image refers to the initial input carrier for the diffusion model to generate a clear image. Its pixel values ​​follow a standard normal distribution, and the size of the random noise image is consistent with the input layer size of the improved diffusion model backbone network. The number of iterations is determined based on the generation efficiency and image quality requirements of the architectural rendering. The number of iterations of the diffusion model directly affects the generation speed and image accuracy. If high efficiency is desired, fewer iterations can be set, in which case the model can complete denoising in a short time, but the generated image may have blurred details. If high quality is desired, more iterations are required, and the model can gradually optimize details through multiple denoising processes.

[0053] The current random noise image is input into the improved diffusion model backbone network. The downsampling unit of the encoder extracts multi-scale features of the current noise image. The downsampling unit is the core component of the encoder. Its structure consists of convolutional layers, batch normalization layers and activation function layers. Its function is to gradually reduce the resolution of the noise image and extract abstract features at different levels: the initial downsampling layer extracts the local texture features of the noise, the deep downsampling layer extracts the global distribution features of the noise, and finally forms a multi-scale feature map. According to the preset corresponding stages, features extracted from the depth branch, semantic branch, color branch, and material branch are injected through the multi-path feature injection interface; in the initial stage of the encoder part, i.e., the low-resolution feature layer: multi-scale spatial structure features of the depth branch are injected, and fused with noise features through element-level addition to constrain the architectural geometric contours and spatial perspective of the generated image; in the middle stage, i.e., the end of the encoder, the bottleneck part, and the beginning of the decoder: multi-scale semantic features of the semantic branch are injected, and the semantic features and noise feature channels are merged by channel splicing and then fused after 1×1 convolution to compress the number of channels, guiding the noise features to optimize towards semantic partitioning; in the later stage of the decoder part, i.e., high-resolution features: color features of the color branch and material features of the material branch are injected simultaneously, and fused with noise features through weighted summation to control the color distribution and light and shadow texture of the noise features.

[0054] After feature fusion, the semantic vectors of the text branches are incorporated through the cross-attention layers of each layer, and the feature weights of the improved diffusion model backbone network are adjusted. Based on the noise prediction values ​​output by the improved diffusion model backbone network, and combined with the attenuation coefficient of the current iteration step, the noise intensity in the random noise image is reduced. Ensure that the generated images retain multimodal feature guidance information while gradually revealing the realistic details of the architectural scene.

[0055] Determine if the current iteration count has reached the preset iteration count: if not, use the updated random noise image as the input for the next iteration and return to S701; if it has reached the preset number of iterations, stop the iteration and output the final generated building rendering. When the preset number of iterations is reached, the noise image is complete, and the output building rendering is shown.

[0056] Reference Figure 2 The second embodiment of the present invention provides a building rendering system that integrates multimodal inputs, comprising: The first processing module 201 is used to: acquire multimodal input data of the building scene; the multimodal input data includes a depth map, a semantic segmentation map, a basic color map, a roughness map, and text prompts; The second processing module 202 is used to: preprocess the multimodal input data and construct a multi-branch feature extraction module; the multi-branch feature extraction module includes a depth branch, a semantic branch, a color branch, a material branch, and a text branch; the depth branch extracts multi-scale spatial structure features; the semantic branch extracts semantic features; the color branch extracts color features; the material branch extracts material features; and the text branch encodes text prompts into semantic vectors. The third processing module 203 is used for: constructing an improved diffusion model backbone network; the improved diffusion model backbone network is based on a U-shaped network structure, which includes an encoder part, a bottleneck part, and a decoder part; setting multiple feature injection interfaces in the initial stage, intermediate stage, and late stage of the encoder part of the U-shaped network structure; the intermediate stage is the connection interval between the end of the encoder part, the bottleneck part, and the beginning of the decoder part; the multiple feature injection interfaces are respectively connected to the output ends of the depth branch, semantic branch, color branch, and material branch of the multi-branch feature extraction module, and the output end of the text branch establishes a cross-modal attention connection with each layer of the U-shaped network structure; injecting the features extracted by each branch into the improved diffusion model backbone network according to the preset corresponding stage through the multiple feature injection interfaces; after each branch feature is injected through the multiple feature injection interfaces, it is fused with the feature map of the corresponding stage of the improved diffusion model backbone network, and the semantic vector of the text branch is applied to the calculation of each layer of the U-shaped network structure through the cross-modal attention mechanism to adjust the feature weights of the improved diffusion model backbone network; The fourth processing module 204 is used to: initiate the generation process of the improved diffusion model backbone network, and iteratively generate architectural renderings starting from random noise images.

[0057] It should be noted that the architectural rendering system for integrating multimodal input provided in this embodiment of the invention is used to execute all the process steps of the architectural rendering system for integrating multimodal input in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0058] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0059] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0060] The embodiments, implementation methods, and related technical features of this application can be combined and substituted for each other without conflict.

[0061] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this application without departing from the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A method for generating an architectural rendering image fusing multi-modal inputs, the method comprising: The method comprises the following steps: Obtain multi-modal input data of a building scene; the multi-modal input data comprises a depth map, a semantic segmentation map, a base color map, a roughness map, and a text prompt word; Preprocess the multi-modal input data and construct a multi-branch feature extraction module; the multi-branch feature extraction module comprises a depth branch, a semantic branch, a color branch, a material branch, and a text branch; the depth branch extracts multi-scale spatial structure features; the semantic branch extracts semantic features; the color branch extracts color features; and the material branch extracts material features; The text branch encodes the text prompt word into a semantic vector; Construct an improved diffusion model backbone network; the improved diffusion model backbone network is based on a U-shaped network structure, which comprises an encoder part, a bottleneck part, and a decoder part; a multi-path feature injection interface is arranged at the initial stage, the middle stage, and the late stage of the decoder part of the U-shaped network structure, respectively; The middle stage is the connection interval of the end of the encoder part, the bottleneck part, and the start of the decoder part; the multi-path feature injection interface is connected with the output end of the depth branch, the semantic branch, the color branch, and the material branch of the multi-branch feature extraction module one by one, respectively; the output end of the text branch is connected with each layer of the U-shaped network structure through a cross-modal attention connection; the features extracted by each branch are injected into the improved diffusion model backbone network through the multi-path feature injection interface according to the corresponding stages; after the features of each branch are injected through the multi-path feature injection interface, the features are fused with the feature maps of the corresponding stages of the improved diffusion model backbone network, and the semantic vector of the text branch is applied to the calculation of each layer of the U-shaped network structure through the cross-modal attention mechanism to adjust the feature weight of the improved diffusion model backbone network; Start the generation process of the improved diffusion model backbone network, and iteratively generate a building rendering image from a random noise image.

2. The method of claim 1, wherein, Obtain multi-modal input data of a building scene, comprising: Export a depth map, a semantic segmentation map, a base color map, and a roughness map under the current view angle of the building scene through a building modeling software or a rendering engine; wherein the semantic segmentation map is automatically labeled by the modeling software; the base color map is derived from the diffuse reflection attribute parameters of the building material, and the roughness map is derived from the surface roughness attribute parameters of the building material; Receive a text prompt word input by a user, which contains the environmental description, style requirement, and detail feature of the building scene, and is used to define the overall atmosphere and local performance emphasis of the generated image.

3. The method of claim 2, wherein, Preprocess the multi-modal input data, comprising: Linearly scale the depth values of the depth map to adapt to the input range of the neural network, and eliminate the depth scale differences of different scenes; One-hot encode the semantic segmentation map to convert the class labels into a multi-channel semantic mask tensor, with each channel corresponding to a single class; Normalize the pixel values of the base color map and the roughness map to adapt to the input range of the neural network; remove special symbols and meaningless characters from the text prompt word; if the length of the text exceeds a preset threshold, input the core semantic fragment to the text branch after truncation. ​ 4. The method of claim 3, wherein, The depth branch, the semantic branch, the color branch and the material branch all adopt a convolutional neural network; each layer of the convolutional neural network comprises a convolution layer, a batch normalization layer and an activation function layer; the convolution layer is used for extracting local detail features or global structure features of a corresponding image modality; The batch normalization layer is used for stabilizing a network training process; the activation function layer is used for introducing a nonlinear mapping, and the activation function layer adopts a ReLU activation function or a GELU activation function; the depth branch, the semantic branch, the color branch and the material branch realize multiple times of down-sampling through a pooling layer; The text branch adopts a pre-trained text encoder, and the pre-trained text encoder is a CLIP text encoder; after the CLIP text encoder encodes a text prompt word into a semantic vector, a normalization operation is performed on the semantic vector to optimize a distribution of the semantic vector, so that the semantic vector is adapted to image features in attention calculation.

5. The method of claim 4, wherein, An output end of the text branch is connected to each layer of the U-shaped network structure through cross-modal attention connection, including: An independent cross-attention layer is arranged at each layer of an encoder, a bottleneck part and each layer of a decoder of the U-shaped network structure; the cross-attention layer is connected in series with a convolution feature calculation module of a corresponding layer; A semantic vector output by the text branch is mapped into a query vector matching a length of an image feature sequence of a corresponding layer through a fully connected layer, and an image feature map of the corresponding layer is flattened into a feature sequence, which is used as a key vector and a value vector of the cross-attention layer at the same time; An attention weight of the query vector and the key vector is calculated through an attention algorithm, and the attention weight is normalized to obtain a weight matrix; The weight matrix and the value vector are weighted and summed to obtain image features fused with text semantics, and the image features fused with the text semantics are used to replace original image features of the corresponding layer.

6. The method of claim 5, wherein, The U-shaped network structure specifically comprises: The encoder part comprises a plurality of down-sampling units; each down-sampling unit comprises a convolution layer, a batch normalization layer and an activation function layer; the input image features are sequentially down-sampled through the plurality of down-sampling units, and the resolution of the feature map is gradually reduced to extract high-level abstract features of the building scene; The bottleneck part comprises a plurality of convolution units and a self-attention module; the convolution units are used for fusing local information of the features of each modality, and the self-attention module captures global dependency relationships of the building scene features by calculating correlation weights between the feature sequences; The decoder part comprises a plurality of up-sampling units, each of which comprises a transposed convolution layer, a batch normalization layer and an activation function layer; the feature map output by the bottleneck part is sequentially up-sampled through the plurality of up-sampling units, and the resolution of the feature map is gradually restored; each up-sampling unit is fused with the feature map of the corresponding layer of the encoder through a channel concatenation operation.

7. The method of claim 6, wherein, The generation process of the improved diffusion model backbone network is started, and a building rendering image is iteratively generated from a random noise image, comprising: A random noise image is initialized; the size of the random noise image is consistent with the size of an input layer of the improved diffusion model backbone network; An iteration number is set; the iteration number is determined according to the generation efficiency requirement and the image quality requirement of the building rendering image; The following operations are performed in each iteration: S701, input the current random noise image into the improved diffusion model backbone network, and extract multi-scale features of the current noise image through the downsampling unit of the encoder; S702, according to the preset corresponding stage, the features extracted by the deep branch, the semantic branch, the color branch and the material branch are injected through the multi-path feature injection interface; after feature fusion, the semantic vector of the text branch is fused through the cross attention layer of each layer to adjust the feature weight of the improved diffusion model backbone network; S703, according to the noise prediction value output by the improved diffusion model backbone network, combined with the decay coefficient of the current iteration step, the noise intensity in the random noise image is reduced; S704, judge whether the current iteration number reaches the preset iteration number. If not, the updated random noise image is used as the input of the next iteration, and returns to S701; if yes, stop iteration and output the finally generated building rendering image.

8. The method of claim 7, wherein, Adjusting the feature weight of the improved diffusion model backbone network includes: In each iteration process of the improved diffusion model backbone network, start the adaptive weight adjustment process; Calculate the similarity between the generated image and each modality input data; wherein the structural similarity index is used to calculate the similarity between the generated image and the depth map, the semantic segmentation map to evaluate the matching degree of spatial structure and semantic layout; the peak signal-to-noise ratio index is used to calculate the similarity between the generated image and the base color map, the roughness map to evaluate the matching degree of color distribution and material characteristics; the cosine similarity index is used to calculate the similarity between the generated image and the text prompt semantic vector to evaluate the matching degree of text semantic guidance; According to the calculated similarity, dynamically adjust the weight coefficient of each branch feature: the higher the similarity between the modality input data and the generated image, the higher the weight coefficient of the corresponding branch feature; the lower the similarity between the modality input data and the generated image, the lower the weight coefficient of the corresponding branch feature; Apply the adjusted weight coefficients of each branch feature to the feature fusion process of the improved diffusion model backbone network.

9. The method of claim 8, wherein, After generating the building rendering image, the method further includes: Using a non-local mean denoising algorithm to remove high-frequency noise in the generated building rendering image; Using a limited contrast adaptive histogram equalization algorithm to adjust the local color contrast of the building rendering image and improve the visual clarity of the building component details; Using a Canny edge detection algorithm to extract the building contour edge features in the building rendering image; fuse the extracted edge features with the original building rendering image according to a preset weight to optimize the sharpness of the building contour.

10. A building rendering map generation system that fuses multi-modal inputs, characterized by, It includes: A first processing module is configured to obtain multi-modal input data of a building scene; the multi-modal input data includes a depth map, a semantic segmentation map, a base color map, a roughness map, and a text prompt word; The second processing module is configured to pre-process the multi-modal input data and construct a multi-branch feature extraction module; the multi-branch feature extraction module comprises a depth branch, a semantic branch, a color branch, a material branch and a text branch; the depth branch extracts multi-scale spatial structure features; the semantic branch extracts semantic features; the color branch extracts color features; and the material branch extracts material features; The text branch encodes text prompt words into semantic vectors; The third processing module is configured to construct an improved diffusion model backbone network; the improved diffusion model backbone network is based on a U-shaped network structure, which comprises an encoder part, a bottleneck part and a decoder part; and a multi-path feature injection interface is arranged at an initial stage, an intermediate stage and a late stage of a decoder part of the encoder part of the U-shaped network structure, respectively; The intermediate stage is a connection interval of the end of the encoder part, the bottleneck part and the start of the decoder part; the multi-path feature injection interface is connected to the output end of the depth branch, the semantic branch, the color branch and the material branch of the multi-branch feature extraction module one by one, and the output end of the text branch is connected to each layer of the U-shaped network structure through a cross-modal attention connection; features extracted by each branch are injected into the improved diffusion model backbone network through the multi-path feature injection interface according to a preset corresponding stage, each branch feature is fused with a feature map of a corresponding stage of the improved diffusion model backbone network after being injected through the multi-path feature injection interface, and a semantic vector of the text branch is applied to the calculation of each layer of the U-shaped network structure through a cross-modal attention mechanism to adjust the feature weight of the improved diffusion model backbone network; The fourth processing module is configured to start a generation process of the improved diffusion model backbone network, and iteratively generate an architectural rendering image from a random noise image.

Images