Dynamic coupling generation method for building information model (BIM) of data center based on multi-modal data

Abstract

The invention provides a dynamic coupling generation method for a building information model (BIM) of a data center based on multi-modal data. The dynamic coupling generation method comprises the following steps: step 1, preparing and preprocessing data; step 2, carrying out image segmentation and edge segmentation; step 3, carrying out image feature coding; step 4, carrying out forward diffusion; step 5, training the ResU-Net model; step 6, image feature decoding is carried out; and step 7, result generation and application. According to the method, natural language processing, computer vision and deep learning technologies are fused, and remarkable beneficial effects are shown in the field of automatic generation of the building effect picture of the data center. According to the method, crossing from text description to image generation can be realized, and high consistency between the style of the generated image and the user demand is ensured.

Description

Technical Field

[0001] The present invention relates to the field of construction engineering, and in particular to a data center building information model (BIM) dynamic coupling generation method based on multimodal data. Background Art

[0002] With the rapid development of artificial intelligence technology, deep learning has gradually emerged in the field of architectural design. The existing architectural cultural image technology methods include: generation technology based on GAN (generative adversarial network), Transformer and diffusion model, and comprehensive platforms integrating multiple technologies and cultural image functions in professional architectural design software. These methods can generate architectural design renderings that meet specific styles based on the text description entered by the user or the selection of preset styles, bringing more innovative possibilities to the field of architectural design. Its advantages are that it can quickly generate high-quality images, support user-defined content and required design styles, and improve the design efficiency of designers. However, the disadvantage of the prior art is that the generated images may lack realism and lack key pixel details. It cannot generate accurate architectural images for a certain type of building, that is, the accuracy is low, and the model size is large and the generation time is long. In the present invention, by collecting a large number of data center images as data sets for training, the accuracy of the generated images is improved, and an image segmentation method is introduced at the input end to improve the realism and fine granularity of the generated images in terms of edges and colors. At the codec end, by introducing a residual network and integrating it into a recurrent neural network, the efficiency and accuracy of the codec are improved. At the output end, a generation adversarial loss function and a classification loss function are introduced to improve the accuracy of the generated images. This improves the accuracy and efficiency of the generated images of the model as a whole. Summary of the invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a data center building information model (BIM) dynamic coupling generation method based on multimodal data in view of the shortcomings of the prior art, comprising the following steps:

[0004] Step 1: Data preparation and preprocessing;

[0005] Step 2, perform image segmentation and edge segmentation;

[0006] Step 3, perform image feature encoding;

[0007] Step 4, forward diffusion;

[0008] Step 5: Train the ResU-Net model;

[0009] Step 6, decoding the image features;

[0010] Step 7: Generate and apply results.

[0011] Step 1 includes:

[0012] Data collection: The architectural rendering image data comes from the Internet public data set, design company or personal portfolio, which is used as the basic data set. While collecting the data set, the text information of the architectural rendering described in the image data is recorded for the labeling of the data set.

[0013] Data screening and cleaning: The obtained architectural effect image dataset is deduplicated by using a hash algorithm to reduce the interference of duplicates. The PIL library in Python is used to remove damaged or unclear image data, and the image size and format are unified to ensure the integrity and uniformity of the images in the dataset.

[0014] Data annotation: By converting the raw image data into a format that can be understood and learned by the deep learning network, the corresponding text description is annotated for each image in the Text-to-Image related tasks. Common data annotation methods include manual annotation, semi-automatic annotation, and automatic annotation. In this task, the preliminary text annotation is generated by using an automatic annotation tool that combines existing image recognition and natural language processing models, and then manually reviewed and corrected by the backend, thereby achieving the relevant text annotation of the architectural image data;

[0015] Data enhancement: The obtained architectural image dataset is enhanced by performing image enhancement, including rotation, flipping, scaling and cropping. The rotation operation is to adjust the image by 90°, 180°, and -90°; the flipping operation is to adjust the image horizontally or vertically; the scaling operation is to enlarge and reduce the image by 0.5, 1.5, and 2 times; the cropping operation is to divide the larger image into smaller images. At the same time, the number of datasets is increased, and the robustness and generalization of the model are improved.

[0016] Step 2 includes:

[0017] Step 2-1, edge segmentation: Gaussian filter the input image f0:

[0018]

[0019] Where G(x,y) represents a two-dimensional Gaussian filter kernel, e is a natural constant, and σ represents the standard deviation. represents the normalization constant, G represents the Gaussian kernel, the pixel point (x, y) in the image is smoothed and filtered, the input image f0 is convolved with the Gaussian kernel to obtain the smoothed image F f :

[0020] F f =G*f0 (2)

[0021] Among them, F f represents the output after Gaussian filtering;

[0022] After smoothing the image with Gaussian filtering, the next step is to calculate the gradient strength and direction of the image. This step is crucial for edge detection because gradient information can perceive and identify areas in the image with significant intensity changes, which usually correspond to the edges of image features. In order to accurately calculate the gradient, the Sobel gradient operator is used to calculate the horizontal and vertical gradients to obtain the horizontal gradient component G x , the longitudinal gradient component G y , sum the horizontal and vertical gradients to obtain the gradient strength M of each pixel, and introduce θ to represent the gradient direction:

[0023]

[0024] in, It means to perform partial differentiation on the horizontal axis of the feature image to find its horizontal axis gradient value. Indicates partial differentiation of the vertical axis of the feature image to find its vertical axis gradient value, arctan() represents the inverse tangent function, which is used to find the gradient angle between different pixel points; the gradient strength M and the gradient direction θ provide important information about the edge in the image. The larger the gradient strength M, the more obvious the edge at that point; the gradient direction θ indicates the direction of the edge, which is helpful for the subsequent non-maximum suppression and edge connection steps.

[0025] In traditional Canny edge detection, the high and low thresholds are usually fixed globally, while in adaptive threshold Canny edge detection, the high and low thresholds are calculated dynamically based on the characteristics of the local image. First, the average value μ(x,y) of each pixel (x,y) in the local window δ(x,y) is calculated, and the variance δ(x,y) is calculated:

[0026]

[0027] Where N represents the number of pixels in the local sliding window, and M(i,j) represents the gradient strength of the pixel (i,j);

[0028] Calculate the overall high threshold T H(x,y) , low threshold T L(x,y) :

[0029] T H(x,y) =μ(x,y)+k1·δ(x,y) (8)

[0030] T L(x,y) =μ(x,y)-k2·δ(x,y) (9)

[0031] Among them, k1 and k2 represent constants, which are generally set to 0~1.

[0032] The gradient value in the above formula (4) is used to obtain the gradient strength and gradient direction of each pixel. In order to further suppress the non-maximum points in each pixel, retain the local maximum value, and then retain the true edge points, refine the extracted edges, for each pixel (x, y), check the gradient strength of the adjacent pixel points in the gradient direction, and calculate the local high and low thresholds T according to the adaptive threshold. H(x,y) and T L(x,y) , the gradient strength is divided into three categories:

[0033]

[0034] Among them, M'(x,y) represents the gradient strength, E(x,y) represents the double threshold detection of each pixel, 1 represents a strong edge, weak represents a weak edge, and 0 represents a non-edge; when M (x,y) If it is greater than the maximum threshold, the adjacent value is set to M (x,y) , otherwise, it is 0. The purpose of dual threshold detection and edge connection is to distinguish strong edges, weak edges (weak) and non-edges by setting two high and low thresholds, and to connect weak edges to strong edges through edge connection to form a complete edge.

[0035] Step 2-2, color segmentation: The goal of the K-Means++ algorithm is to divide the pixels into K clusters so that the pixels within each cluster are as similar as possible, while the data points between different clusters are as different as possible. The algorithm optimizes the distance within the cluster by selecting cluster centers and dividing clusters through an iterative process, and updates the cluster centers to minimize the sum of the squared distances between each data point and its nearest cluster center. The smaller the value of E, the higher the similarity of the samples within the cluster and the more accurate the color segmentation.

[0036] The image that has been prepared and preprocessed is used as the input sample and K pixels of the sample are randomly selected as the initial cluster center. The minimum Euclidean distance from the remaining pixels to the initial cluster center is calculated. The number of pixels in the entire input sample is used as the weight denominator to select the next cluster center. The formula is:

[0037] C m =x n ,m=1,2,3....K,n∈[1,T] (12)

[0038]

[0039] Among them, C m Indicates that the mth pixel point is randomly selected as the cluster center, where m is an integer between 1 and K, and xn represents the nth random pixel of the input image, where n is a random integer between 1 and T, |C m | represents the vector value of the selected mth cluster center, φ m represents the mean vector value of the selected mth cluster, and E(·) represents the shortest Euclidean distance from each pixel to the selected cluster center;

[0040] By taking the distance from each pixel to the existing cluster center as the denominator E(x m ), the distance from the randomly selected pixel to the existing cluster center is used as the numerator E(x i ), the probability P(x) selected as the new cluster center i ) and E(x i ), that is, the farther the newly selected cluster center is from the previous cluster center, the greater the probability that the newly selected cluster center will be selected as the new cluster center. The selection of new cluster centers is repeated until K cluster centers are selected:

[0041]

[0042] Among them, P(x i ) represents the i-th pixel x selected in the input image i The probability of being selected as the new cluster center, E(xi) represents the selected i-th pixel x i E(·) represents the Euclidean distance from a selected pixel to the existing cluster center.

[0043] Step 3 includes:

[0044] Image encoding: After the input image is processed by convolution, adaptive pooling and batch normalization, a randomly generated noise vector z is introduced as the initial noise vector. The feature vector in the feature image after image segmentation is fused with the initial noise vector to generate the initial image features, and then forward diffusion is performed;

[0045] Use multimodal deep learning models, such as a hybrid model that combines BERT (for text processing) and ResU-Net (for image processing). Use the BERT model to extract key information from text descriptions, such as architectural style, color preference, etc. Use the ResU-Net model to extract visual features from image samples, combining an encoder and decoder, and a skip connection to achieve feature sampling, segmentation, detection, and fusion.

[0046] A feature fusion strategy is designed, which specifically includes: converting text descriptions into text feature vectors through the BERT model, normalizing the text feature vectors and image feature vectors through batch normalization to ensure that the text and image features are on the same scale; concatenating text features and image features to generate a joint feature vector, and then introducing a cross-modal attention mechanism to achieve the fusion of text features and image features.

[0047] Step 4 includes: in the forward diffusion process, gradually adding noise vectors to the feature vectors in the feature image after image segmentation to generate a series of noise image features F t :

[0048]

[0049] where β t represents the control parameter, z t represents the noise at the tth time step, F t-1 Represents the noise image features generated in the previous round.

[0050] Step 5 includes: inputting the data preprocessed in step 1 into the ResU-Net model for training. Specifically, ResU-Net is a semantic segmentation neural network that combines the advantages of U-Net and residual neural network (ResNet). First, the residual unit will simplify the training of the network; second, the jump connection within the residual unit and between the low and high layers of the network will promote the propagation of information without degradation. The ResU-Net model includes an encoder, a skip connection, and a decoder; the encoder encodes the input image into a compact representation. The skip connection end connects the encoding path and the decoding path, and the decoder completes these representations into pixel-level classification, i.e., semantic segmentation.

[0051] The encoder, skip connection and decoder all introduce residual modules, which include two 3×3 convolution blocks and an identity map. Each convolution block includes a batch normalization (BN) layer, a ReLU activation layer and a convolution layer.

[0052] The encoder includes three residual modules. In each residual module, the size of the input feature map is halved by setting the stride of the first convolution block to 2.

[0053] The decoder includes three residual modules, each of which upsamples the feature map from the previous input level and concatenates it with the feature map in the corresponding encoding path. After the last layer of the decoder, a 1×1 convolution and a sigmoid activation layer are used to project the multi-channel feature map onto the desired segmentation result to obtain the final output.

[0054] First, add a direct connection between the convolutional layers of the residual module in the encoder so that the network can more easily learn the information of the forward input features, alleviate the problem of gradient vanishing during pooling and normalization, and improve the training effect of the network. The formula of the residual module is:

[0055]

[0056] in, They represent the edge segmentation and color segmentation of the input image after preliminary image segmentation, Conv 1×1 Represents 1×1 convolution, Conv 2×2 represents 2×2 convolution, ReLU() represents activation function, and F(f) represents output feature;

[0057] The encoder gradually reduces the spatial resolution of the feature map through a series of convolutional layers and pooling layers, while increasing the number of channels of the feature map to improve the high-level features of the image. The formula of the encoder is:

[0058] B i =R(AdaptivePool(B i-1 )) (18)

[0059] Among them, AdaptivePool() represents adaptive maximum pooling, which adaptively adjusts the size of the convolution frame according to the pixel clustering in the feature map, R() represents the residual module, and B i represents the feature map of the i-th layer in the encoder;

[0060] The decoder gradually restores the spatial resolution of the feature map through a series of upsampling layers and convolutional layers, while reducing the number of channels of the feature map. The decoder also combines the low-level features in the encoder with the high-level features through jump connections to improve the accuracy of segmentation. The decoder formula is:

[0061] J i =R(Concat(Upsample(J i-1 ),B i )) (19)

[0062] Among them, J irepresents the feature map of the jth layer in the decoder, Upsample(J i-1 ) indicates an upsampling operation on the feature map of the j-1th layer in the decoder, and Concat() indicates a concatenation operation;

[0063] In order to further quantify the difference between the obtained output results and the original results, a loss function is introduced at the output end to calculate and measure the difference between the predicted results and the real data, thereby improving the network training process. The Wasserstein generative adversarial loss function (WGANs) and the Focal loss function are introduced at the output end of this improved model. The WGANs function is used to measure the difference between the generated distribution and the real distribution, thereby generating more stable high-quality images. The formula is as shown in (19). The Focal loss function is mainly used to deal with the problem of class imbalance. By adjusting the weights of difficult-to-classify samples, the model's attention to difficult-to-classify samples is increased. In image generation tasks, especially when generating images with specific categories, the Focal loss can improve the accuracy and diversity of generated images. The output layer of the decoder in the ResU-Net module is subjected to a 1×1 convolution operation as the output of the model, and the loss function is introduced:

[0064]

[0065] Among them, F T represents the real image, S(F T ) represents the real image dataset, D(F T ) represents the output result of the discriminator for the real image, represents the expected value of the discriminator's output result for the real image; z represents a random noise vector, Z(z) represents a noise set, D(G(z)) represents the discriminator's output result for the generated image, and E z～Z(z) [D(G(z))] represents the discriminator’s expectation of the output of the generated image;

[0066]

[0067] Among them, N pixel Represents the total number of pixels, P i represents the predicted probability of the i-th pixel, y i represents the true label of the i-th pixel, and γ represents the focus parameter, which is used to adjust the classification weight of difficult samples.

[0068] L total =L WGANs +λL Focal (twenty two)

[0069] Among them, L WGANs represents the Wasserstein generative adversarial loss, L FocalRepresents the Focal function loss, L total represents the total loss, λ represents the balance factor, λ∈(0,1];

[0070] Reverse diffusion is performed. Compared with the forward diffusion process in which the input image feature vector is gradually denoised, the reverse diffusion process gradually denoises the feature vector to further generate a higher quality image. The formula is:

[0071]

[0072] Step 6 includes: the decoder uses deconvolution (transposed convolution) and upsampling operations to gradually restore the spatial resolution of the feature map, and finally generates a high-quality image corresponding to the input features; the decoder ensures that the generated image is consistent with the original image in structure and details by learning the mapping relationship from the low-dimensional feature space to the high-dimensional image space.

[0073] Step 7 includes:

[0074] Set input parameters: The user enters a text description and reference image describing the architectural style;

[0075] Rendering generation: Input parameters are fed into the trained ResU-Net model to generate preliminary building renderings;

[0076] Effect evaluation and adjustment: By comparing the generated effect pictures with the real effect pictures or the user's expected effect, the quality and accuracy of the generated effect are evaluated, and the model is fine-tuned or retrained according to the evaluation results to improve the generation effect;

[0077] The final architectural rendering is output to the user, who can make further edits or adjustments as needed.

[0078] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0079] The present invention also provides a storage medium storing a computer program or instruction. When the computer program or instruction is run on a computer, the steps of the method described are executed.

[0080] The method of the invention can be widely used in the fields of architectural design, urban planning, real estate marketing, etc., providing designers and customers with efficient and convenient design tools.

[0081] The method of the present invention aims to automatically generate a data center building rendering that conforms to a specific style based on the detailed text description and reference image input by the user by integrating natural language processing, computer vision and deep learning technology. The entire image generation process includes seven main steps: image preparation and preprocessing, image segmentation, image feature encoding operation, feature vector denoising and forward diffusion, multiple rounds of feature denoising and extraction combined with CLIP text editor and ResU-Net model, image feature decoding, and output and application of generated results. In the data preparation and preprocessing stage, a large number of architectural renderings and their corresponding text descriptions are collected and processed to form a data set; in the image segmentation stage, the input image is segmented by edges and colors to obtain a more refined image encoding; in the forward diffusion stage, the feature vector is subjected to multiple rounds of random noise addition operations; in the reverse diffusion process, the input text vector and the forward diffusion image are sent to the ReU-Net model for multiple rounds of feature denoising to obtain the denoised image; in the feature decoding stage, the final generation result is obtained through decoding; in the result generation stage, the Wasserstein generation adversarial loss function and the Focal loss function are used to measure the differences in feature distribution and classification between the generated image and the real image, which are used for backpropagation to the model for further iteration. Finally, the final rendering is output to the user and applied to the field of data center architectural design.

[0082] Beneficial effects: The present invention integrates natural language processing, computer vision and deep learning technology, and has shown significant beneficial effects in the field of automatic generation of data center architectural renderings. The method achieves a leap from text description to image generation through seven closely connected steps, ensuring that the style of the generated image is highly consistent with user needs. In the data preparation and preprocessing stage, the collection and processing of a large number of data center architectural renderings and their corresponding text descriptions provide rich learning materials for the model and ensure the accuracy of the generated image. The refined processing in the image segmentation stage further improves the accuracy of image encoding and lays a solid foundation for subsequent feature extraction, feature fusion and image generation. Through the feature denoising and denoising operations in the forward diffusion and reverse diffusion processes, the model is continuously optimized in multiple iterations and gradually approaches the feature distribution of the real image. At the same time, combined with the CLIP text editor and the ResU-Net model, the deep fusion of text and image features is achieved, so that the generated image can accurately reflect the user's intention. Under the joint action of the Wasserstein generative adversarial loss function and the Focal loss function, the difference between the generated image and the real image is effectively measured and fed back to the model, promoting the continuous iterative optimization of the model. This process not only improves the quality of the generated image, but also ensures the wide application prospects of the model in the field of data center architectural design. In summary, the method of the present invention is of great significance for promoting the intelligent and efficient design of data center architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0084] Figure 1 The process of generating images for a data center building.

[0085] Figure 2 This is the overall model architecture diagram of Wenshengtu.

[0086] Figure 3 This is the ResU-Net model architecture diagram.

[0087] Figure 4 This is the residual module architecture diagram.

[0088] Figure 5 It is a flowchart of node image processing used in an embodiment of the present invention.

[0089] Figure 6 It is a diagram showing the results of the present invention. DETAILED DESCRIPTION

[0090] like Figure 1 , Figure 2 As shown, an embodiment of the present invention provides a method for dynamically coupling and generating a data center building information model (BIM) based on multimodal data, comprising the following steps:

[0091] Step 1: Data preparation and preprocessing;

[0092] Step 2, perform image segmentation and edge segmentation;

[0093] Step 3, perform image feature encoding;

[0094] Step 4, forward diffusion;

[0095] Step 5: Train the ResU-Net model;

[0096] Step 6, decoding the image features;

[0097] Step 7: Generate and apply results.

[0098] Step 1 includes:

[0099] Data collection: The architectural rendering image data comes from the Internet public data set, design company or personal portfolio, which is used as the basic data set. While collecting the data set, the text information of the architectural rendering described in the image data is recorded for the labeling of the data set.

[0100] Data screening and cleaning: The obtained architectural effect image dataset is deduplicated by using a hash algorithm to reduce the interference of duplicates. The PIL library in Python is used to remove damaged or unclear image data, and the image size and format are unified to ensure the integrity and uniformity of the images in the dataset.

[0101] Data annotation: By converting the raw image data into a format that can be understood and learned by the deep learning network, the corresponding text description is annotated for each image in the Text-to-Image related tasks. Common data annotation methods include manual annotation, semi-automatic annotation, and automatic annotation. In this task, the preliminary text annotation is generated by using an automatic annotation tool that combines existing image recognition and natural language processing models, and then manually reviewed and corrected by the backend, thereby achieving the relevant text annotation of the architectural image data;

[0102] Data enhancement: The obtained architectural image dataset is enhanced by performing image enhancement, including rotation, flipping, scaling and cropping. The rotation operation is to adjust the image by 90°, 180°, and -90°; the flipping operation is to adjust the image horizontally or vertically; the scaling operation is to enlarge and reduce the image by 0.5, 1.5, and 2 times; the cropping operation is to divide the larger image into smaller images. At the same time, the number of datasets is increased, and the robustness and generalization of the model are improved.

[0103] Step 2 includes:

[0104] Step 2-1, edge segmentation: Gaussian filter the input image f0:

[0105]

[0106] Where G(x,y) represents a two-dimensional Gaussian filter kernel, e is a natural constant, and σ represents the standard deviation. represents the normalization constant, G represents the Gaussian kernel, the pixel point (x, y) in the image is smoothed and filtered, the input image f0 is convolved with the Gaussian kernel to obtain the smoothed image F f :

[0107] F f =G*f0 (2)

[0108] Among them, F f represents the output after Gaussian filtering;

[0109] After smoothing the image with Gaussian filtering, the next step is to calculate the gradient strength and direction of the image. This step is crucial for edge detection because gradient information can perceive and identify areas in the image with significant intensity changes, which usually correspond to the edges of image features. In order to accurately calculate the gradient, the Sobel gradient operator is used to calculate the horizontal and vertical gradients to obtain the horizontal gradient component G x , the longitudinal gradient component G y , sum the horizontal and vertical gradients to obtain the gradient strength M of each pixel, and introduce θ to represent the gradient direction:

[0110]

[0111] in, It means to perform partial differentiation on the horizontal axis of the feature image to find its horizontal axis gradient value. Indicates partial differentiation of the vertical axis of the feature image to find its vertical axis gradient value, arctan() represents the inverse tangent function, which is used to find the gradient angle between different pixel points; the gradient strength M and the gradient direction θ provide important information about the edge in the image. The larger the gradient strength M, the more obvious the edge at that point; the gradient direction θ indicates the direction of the edge, which is helpful for the subsequent non-maximum suppression and edge connection steps.

[0112] In traditional Canny edge detection, the high and low thresholds are usually fixed globally, while in adaptive threshold Canny edge detection, the high and low thresholds are calculated dynamically based on the characteristics of the local image. First, the average value μ(x,y) of each pixel (x,y) in the local window δ(x,y) is calculated, and the variance δ(x,y) is calculated:

[0113]

[0114] Where N represents the number of pixels in the local sliding window, and M(i,j) represents the gradient strength of the pixel (i,j);

[0115] Calculate the overall high threshold T H(x ,y ) , low threshold T L(x ,y ) :

[0116] T H(x,y) =μ(x,y)+k1·δ(x,y) (8)

[0117] T L(x,y) =μ(x,y)-k2·δ(x,y) (9)

[0118] Among them, k1 and k2 represent constants, which are generally set to 0~1.

[0119] The gradient value in the above formula (4) is used to obtain the gradient strength and gradient direction of each pixel. In order to further suppress the non-maximum points in each pixel, retain the local maximum value, and then retain the true edge points, refine the extracted edges, for each pixel (x, y), check the gradient strength of the adjacent pixel points in the gradient direction, and calculate the local high and low thresholds T according to the adaptive threshold. H(x,y) and T L(x,y) , the gradient strength is divided into three categories:

[0120]

[0121] Among them, M'(x,y) represents the gradient strength, E(x,y) represents the double threshold detection of each pixel, 1 represents a strong edge, weak represents a weak edge, and 0 represents a non-edge; when M (x,y) If it is greater than the maximum threshold, the adjacent value is set to M (x,y) , otherwise, it is 0. The purpose of dual threshold detection and edge connection is to distinguish strong edges, weak edges (weak) and non-edges by setting two high and low thresholds, and to connect weak edges to strong edges through edge connection to form a complete edge.

[0122] Step 2-2, color segmentation: The goal of the K-Means++ algorithm is to divide the pixels into K clusters so that the pixels within each cluster are as similar as possible, while the data points between different clusters are as different as possible. The algorithm optimizes the distance within the cluster by selecting cluster centers and dividing clusters through an iterative process, and updates the cluster centers to minimize the sum of the squared distances between each data point and its nearest cluster center. The smaller the value of E, the higher the similarity of the samples within the cluster and the more accurate the color segmentation.

[0123] The image that has been prepared and preprocessed is used as the input sample and K pixels of the sample are randomly selected as the initial cluster center. The minimum Euclidean distance from the remaining pixels to the initial cluster center is calculated. The number of pixels in the entire input sample is used as the weight denominator to select the next cluster center. The formula is:

[0124] C m =x n ,m=1,2,3....K,n∈[1,T] (12)

[0125]

[0126] Among them, C m Indicates that the mth pixel point is randomly selected as the cluster center, where m is an integer between 1 and K, and x n represents the nth random pixel of the input image, where n is a random integer between 1 and T, |Cm | represents the vector value of the selected mth cluster center, φ m represents the mean vector value of the selected mth cluster, and E(·) represents the shortest Euclidean distance from each pixel to the selected cluster center;

[0127] By taking the distance from each pixel to the existing cluster center as the denominator E(x m ), the distance from the randomly selected pixel to the existing cluster center is used as the numerator E(x i ), the probability P(x) selected as the new cluster center i ) and E(x i ), that is, the farther the newly selected cluster center is from the previous cluster center, the greater the probability that the newly selected cluster center will be selected as the new cluster center. The selection of new cluster centers is repeated until K cluster centers are selected:

[0128]

[0129] Among them, P(x i ) represents the i-th pixel x selected in the input image i The probability of being selected as the new cluster center, E(xi) represents the selected i-th pixel x i E(·) represents the Euclidean distance from a selected pixel to the existing cluster center.

[0130] Step 3 includes:

[0131] Image encoding: After the input image is processed by convolution, adaptive pooling and batch normalization, a randomly generated noise vector z is introduced as the initial noise vector. The feature vector in the feature image after image segmentation is fused with the initial noise vector to generate the initial image features, and then forward diffusion is performed;

[0132] Use multimodal deep learning models, such as a hybrid model that combines BERT (for text processing) and ResU-Net (for image processing). Use the BERT model to extract key information from text descriptions, such as architectural style, color preference, etc. Use the ResU-Net model to extract visual features from image samples, combining an encoder and decoder, and a skip connection to achieve feature sampling, segmentation, detection, and fusion.

[0133] A feature fusion strategy is designed, which specifically includes: converting text descriptions into text feature vectors through the BERT model, normalizing the text feature vectors and image feature vectors through batch normalization to ensure that the text and image features are on the same scale; concatenating text features and image features to generate a joint feature vector, and then introducing a cross-modal attention mechanism to achieve the fusion of text features and image features.

[0134] Step 4 includes: in the forward diffusion process, gradually adding noise vectors to the feature vectors in the feature image after image segmentation to generate a series of noise image features F t :

[0135]

[0136] where β t represents the control parameter, z t represents the noise at the tth time step, F t-1 Represents the noise image features generated in the previous round.

[0137] Step 5 includes: inputting the data preprocessed in step 1 into the ResU-Net model for training. Specifically, ResU-Net is a semantic segmentation neural network that combines the advantages of U-Net and residual neural network (ResNet). First, the residual unit will simplify the training of the network; second, the jump connection within the residual unit and between the low and high layers of the network will promote the propagation of information without degradation. The ResU-Net model includes an encoder, a skip connection, and a decoder; the encoder encodes the input image into a compact representation. The skip connection end connects the encoding path and the decoding path, and the decoder completes these representations into pixel-level classification, i.e., semantic segmentation.

[0138] like Figure 3 , Figure 4 As shown, the encoder, skip connection and decoder all introduce residual modules, and the residual module includes two 3×3 convolution blocks and an identity map. Each convolution block includes a batch normalization BN layer, a ReLU activation layer and a convolution layer.

[0139] The encoder includes three residual modules. In each residual module, the size of the input feature map is halved by setting the stride of the first convolution block to 2.

[0140] The decoder includes three residual modules, each of which upsamples the feature map from the previous input level and concatenates it with the feature map in the corresponding encoding path. After the last layer of the decoder, a 1×1 convolution and a sigmoid activation layer are used to project the multi-channel feature map onto the desired segmentation result to obtain the final output.

[0141] First, add a direct connection between the convolutional layers of the residual module in the encoder so that the network can more easily learn the information of the forward input features, alleviate the problem of gradient vanishing during pooling and normalization, and improve the training effect of the network. The formula of the residual module is:

[0142]

[0143] in, They represent the edge segmentation and color segmentation of the input image after preliminary image segmentation, Conv 1×1 Represents 1×1 convolution, Conv 2×2 represents 2×2 convolution, ReLU() represents activation function, and F(f) represents output feature;

[0144] The encoder gradually reduces the spatial resolution of the feature map through a series of convolutional layers and pooling layers, while increasing the number of channels of the feature map to improve the high-level features of the image. The formula of the encoder is:

[0145] B i =R(AdaptivePool(B i-1 )) (18)

[0146] Among them, AdaptivePool() represents adaptive maximum pooling, which adaptively adjusts the size of the convolution frame according to the pixel clustering in the feature map, R() represents the residual module, and B i represents the feature map of the i-th layer in the encoder;

[0147] The decoder gradually restores the spatial resolution of the feature map through a series of upsampling layers and convolutional layers, while reducing the number of channels of the feature map. The decoder also combines the low-level features in the encoder with the high-level features through jump connections to improve the accuracy of segmentation. The decoder formula is:

[0148] J i =R(Concat(Upsample(J i-1 ),B i )) (19)

[0149] Among them, J irepresents the feature map of the jth layer in the decoder, Upsample(J i-1 ) indicates an upsampling operation on the feature map of the j-1th layer in the decoder, and Concat() indicates a concatenation operation;

[0150] In order to further quantify the difference between the obtained output results and the original results, a loss function is introduced at the output end to calculate and measure the difference between the predicted results and the real data, thereby improving the network training process. The Wasserstein generative adversarial loss function (WGANs) and the Focal loss function are introduced at the output end of this improved model. The WGANs function is used to measure the difference between the generated distribution and the real distribution, thereby generating more stable high-quality images. The formula is as shown in (19). The Focal loss function is mainly used to deal with the problem of class imbalance. By adjusting the weights of difficult-to-classify samples, the model's attention to difficult-to-classify samples is increased. In image generation tasks, especially when generating images with specific categories, the Focal loss can improve the accuracy and diversity of generated images. The output layer of the decoder in the ResU-Net module is subjected to a 1×1 convolution operation as the output of the model, and the loss function is introduced:

[0151]

[0152] Among them, F T represents the real image, S(F T ) represents the real image dataset, D(F T ) represents the output result of the discriminator for the real image, represents the expected value of the discriminator's output result for the real image; z represents a random noise vector, Z(z) represents a noise set, D(G(z)) represents the discriminator's output result for the generated image, and E z～Z(z) [D(G(z))] represents the discriminator’s expectation of the output of the generated image;

[0153]

[0154] Among them, N pixel Represents the total number of pixels, P i represents the predicted probability of the i-th pixel, y i represents the true label of the i-th pixel, and γ represents the focus parameter, which is used to adjust the classification weight of difficult samples.

[0155] L total =L WGANs +λL Focal (twenty two)

[0156] Among them, L WGANs represents the Wasserstein generative adversarial loss, L FocalRepresents the Focal function loss, L total represents the total loss, λ represents the balance factor, λ∈(0,1];

[0157] Reverse diffusion is performed. Compared with the forward diffusion process in which the input image feature vector is gradually denoised, the reverse diffusion process gradually denoises the feature vector to further generate a higher quality image. The formula is:

[0158]

[0159] Step 6 includes: the decoder uses deconvolution (transposed convolution) and upsampling operations to gradually restore the spatial resolution of the feature map, and finally generates a high-quality image corresponding to the input features; the decoder ensures that the generated image is consistent with the original image in structure and details by learning the mapping relationship from the low-dimensional feature space to the high-dimensional image space.

[0160] Step 7 includes:

[0161] Set input parameters: The user enters a text description and reference image describing the architectural style;

[0162] Rendering generation: Input parameters are fed into the trained ResU-Net model to generate preliminary building renderings;

[0163] Effect evaluation and adjustment: By comparing the generated effect pictures with the real effect pictures or the user's expected effect, the quality and accuracy of the generated effect are evaluated, and the model is fine-tuned or retrained according to the evaluation results to improve the generation effect;

[0164] The final architectural rendering is output to the user, who can make further edits or adjustments as needed.

[0165] The embodiment of the present invention provides a data center building information model (BIM) dynamic coupling generation method based on multimodal data. The main node images in the embodiment of the present invention are as follows: Figure 5 As shown in the figure, it mainly includes seven main steps: image preparation and preprocessing, image segmentation, image feature encoding operation, feature vector denoising and forward diffusion, combining CLIP text editor with ResU-Net model for multiple rounds of feature denoising and extraction, image feature decoding and output results.

[0166] In the data preparation and preprocessing stage, an image is randomly selected from the architectural data set as input; the input image is segmented by edge and color to highlight the characteristic edges and color clustering of the image; the edge segmentation and color segmentation maps are feature fused and encoded, and then forward multiple rounds of noise diffusion operations are performed on them; in the reverse diffusion stage, the generated conditional semantic vector is sent to the ReU-Net model for multiple rounds of feature denoising to obtain the denoised image; in the feature decoding stage, the final generated result is obtained through feature decoding; in the result generation stage, the difference in feature distribution and classification between the generated image and the real image is calculated through the Wasserstein generation adversarial loss function and the Focal loss function, and the calculated loss is back-propagated iteratively. The final rendering is output to the user.

[0167] like Figure 6 As shown in the generated results, in order to further highlight the advantages of the method of the embodiment of the present invention, the DALL-E model (developed by OpenAI) and the Imagen model (developed by Googel) are introduced as comparison methods. First, the same data center exterior line drawing and the same prompt words are provided to the three large models. The prompt words are keywords such as modern industrial park, building exterior, and modern building. The images generated by the three models are as follows: Figure 6 As shown in . From the generated image, it can be seen that the image generated by the DALL-E model is distorted at the edge of the facade of the data center, and the line distortion phenomenon occurs (marked by the red frame), and the texture of the image is different, which makes the styles of several data centers inconsistent, greatly reducing the quality of the generated image. The image generated by the Imagen model is distorted at the edge of the data center, and introduces more debris, blocking the display of the normal building edge (marked by the red frame), indicating that the algorithm lacks the influence of text prompts during denoising, and several key pixels are distorted, which makes the overall generated image style deviate from the style of the data center. The data center image generated by the method of the present invention is the most in line with the needs in terms of the accuracy and fit of the edges, texture features and overall generation effects of the building facade, which can better meet the needs of designers, improve the efficiency of design, and achieve the purpose of the design of the present invention.

[0168] The present invention provides a method for dynamically coupling and generating a building information model (BIM) of a data center based on multimodal data. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A data center building information model (BIM) dynamic coupling generation method based on multimodal data, characterized in that: The following steps are involved: Step 1: Data preparation and preprocessing; Step 2, perform image segmentation and edge segmentation; Step 3, perform image feature encoding; Step 4, forward diffusion; Step 5: Train the ResU-Net model; Step 6, decoding the image features; Step 7: Generate and apply results.

2. The method according to claim 1, characterized in that Step 1 includes: Data collection: The architectural effect image data is used as the basic data set. While collecting the data set, the text information of the architectural effect described in the image data is recorded for the labeling of the data set. Data screening and cleaning: Use hashing algorithms to remove duplicates from the obtained architectural effect image dataset, use the PIL library in Python to remove damaged or unclear image data, and unify the image size and format; Data annotation: By converting the raw image data into a format that the deep learning network can understand and learn, in the text-image related tasks, the corresponding text description is annotated for each image; Data enhancement: The obtained architectural image dataset is subjected to image enhancement.

3. The method according to claim 2, characterized in that Step 2 includes: Step 2-1, edge segmentation: Gaussian filter the input image f0: Where G(x,y) represents a two-dimensional Gaussian filter kernel, e is a natural constant, and σ represents the standard deviation. represents the normalization constant, G represents the Gaussian kernel, the pixel point (x, y) in the image is smoothed and filtered, the input image f0 is convolved with the Gaussian kernel to obtain the smoothed image F f : F f =G*f0 (2) Among them, F f represents the output after Gaussian filtering; By using the Sobel gradient operator to calculate the horizontal and vertical gradients, the horizontal gradient component G is obtained. x , the longitudinal gradient component G y , sum the horizontal and vertical gradients to obtain the gradient strength M of each pixel, and introduce θ to represent the gradient direction: in, It means to perform partial differentiation on the horizontal axis of the feature image to find its horizontal axis gradient value. It indicates that the partial differentiation of the vertical axis of the feature image is performed to obtain the vertical axis gradient value. Arctan() represents the inverse tangent function, which is used to calculate the gradient angle between different pixel points. Calculate the mean μ(x,y) of each pixel (x,y) in the local window δ(x,y) and calculate the variance δ(x,y): Where N represents the number of pixels in the local sliding window, and M(i,j) represents the gradient strength of the pixel (i,j); Calculate the overall high threshold T H(x,y) , low threshold T L(x,y) : T H(x,y) =μ(x,y)+k1·δ(x,y) (8) T L(x,y) =μ(x,y)-k2·δ(x,y) (9) Where k1 and k2 represent constants; For each pixel (x, y), check the gradient strength of the adjacent pixels in the gradient direction, and calculate the local high and low thresholds T according to the adaptive threshold H(x,y) and T L(x,y) , the gradient strength is divided into three categories: Where M'(x,y) represents the gradient strength, E(x,y) represents the double threshold detection of each pixel, 1 represents a strong edge, weak represents a weak edge, and 0 represents a non-edge; Step 2-2, color segmentation: take the image after data preparation and preprocessing as the input sample and randomly select K pixels of the sample as the initial cluster center, and calculate the minimum Euclidean distance from the remaining pixels to the initial cluster center, and use the number of pixels in the entire input sample as the weight denominator to select the next cluster center. The formula is: C m =x n ,m=1,2,3.....K,n∈[1,T] (12) Among them, C m Indicates that the mth pixel point is randomly selected as the cluster center, where m is an integer between 1 and K, and x n represents the nth random pixel of the input image, where n is a random integer between 1 and T, |C m | represents the vector value of the selected mth cluster center, φ m represents the mean vector value of the selected mth cluster, and E(·) represents the shortest Euclidean distance from each pixel to the selected cluster center; By taking the distance from each pixel to the existing cluster center as the denominator E(x m ), the distance from the randomly selected pixel to the existing cluster center is used as the numerator E(x i ), the probability P(x) selected as the new cluster center i ) and E(x i ), that is, the farther the newly selected cluster center is from the previous cluster center, the greater the probability that the newly selected cluster center will be selected as the new cluster center. The selection of new cluster centers is repeated until K cluster centers are selected: Among them, P(x i ) represents the i-th pixel x selected in the input image i The probability of being selected as the new cluster center, E(xi) represents the selected i-th pixel x i E(·) represents the Euclidean distance from a selected pixel to the existing cluster center.

4. The method according to claim 3, characterized in that Step 3 includes: Image encoding: After the input image is processed by convolution, adaptive pooling and batch normalization, a randomly generated noise vector z is introduced as the initial noise vector. The feature vector in the feature image after image segmentation is fused with the initial noise vector to generate the initial image features, and then forward diffusion is performed; Adopt a multimodal deep learning model, use the BERT model to extract key information from text descriptions, use the ResU-Net model to extract visual features from image samples, combine the encoder and decoder, and a skip connection to achieve feature sampling, segmentation, detection, and fusion; A feature fusion strategy is designed, which specifically includes: converting text descriptions into text feature vectors through the BERT model, normalizing the text feature vectors and image feature vectors through batch normalization to ensure that the text and image features are on the same scale; concatenating text features and image features to generate a joint feature vector, and then introducing a cross-modal attention mechanism to achieve the fusion of text features and image features.

5. The method according to claim 4, characterized in that Step 4 includes: in the forward diffusion process, gradually adding noise vectors to the feature vectors in the feature image after image segmentation to generate a series of noise image features F t : where β t represents the control parameter, z t represents the noise at the tth time step, F t-1 Represents the noise image features generated in the previous round.

6. The method according to claim 5, characterized in that Step 5 includes: inputting the data preprocessed in step 1 into a ResU-Net model for training, wherein the ResU-Net model includes an encoder, a skip connection, and a decoder; The encoder, skip connection and decoder all introduce residual modules, which include two 3×3 convolution blocks and an identity map. Each convolution block includes a batch normalization (BN) layer, a ReLU activation layer and a convolution layer. The encoder includes three residual modules. In each residual module, the size of the input feature map is halved by setting the stride of the first convolution block to 2. The decoder includes three residual modules, each of which upsamples the feature map from the previous input level and concatenates it with the feature map in the corresponding encoding path. After the last layer of the decoder, a 1×1 convolution and a sigmoid activation layer are used to project the multi-channel feature map onto the desired segmentation result to obtain the final output. First, a direct connection is added between the convolutional layers of the residual module in the encoder. The formula of the residual module is: in, They represent the edge segmentation and color segmentation of the input image after preliminary image segmentation, Conv 1×1 Represents 1×1 convolution, Conv 2×2 represents 2×2 convolution, ReLU() represents activation function, and F(f) represents output feature; The formula of encoder is: B i =R(AdaptivePool(B i-1 )) (18) Among them, AdaptivePool() represents adaptive maximum pooling, which adaptively adjusts the size of the convolution frame according to the pixel clustering in the feature map, R() represents the residual module, and B i represents the feature map of the i-th layer in the encoder; The decoder formula is: J i =R(Concat(Upsample(J i-1 ),B i )) (19) Among them, J i represents the feature map of the jth layer in the decoder, Upsample(J i-1 ) indicates an upsampling operation on the feature map of the j-1th layer in the decoder, and Concat() indicates a concatenation operation; The output layer of the decoder in the ResU-Net module is subjected to a 1×1 convolution operation as the output of the model, and the loss function is introduced: Among them, F T represents the real image, S(F T ) represents the real image dataset, D(F T ) represents the output result of the discriminator for the real image, represents the expected value of the discriminator's output result for the real image; z represents a random noise vector, Z(z) represents a noise set, D(G(z)) represents the discriminator's output result for the generated image, and E z～Z(z) [D(G(z))] represents the discriminator’s expectation of the output of the generated image; Among them, N pixel Represents the total number of pixels, P i represents the predicted probability of the i-th pixel, y i represents the true label of the i-th pixel, and γ represents the focus parameter; THE total =L WGANs +λL Focal (22) Among them, L WGANs represents the Wasserstein generative adversarial loss, L Focal Represents the Focal function loss, L total represents the total loss, λ represents the balance factor, λ∈(0,1]; For reverse diffusion, the formula is:

7. The method according to claim 6, characterized in that Step 6 includes: the decoder uses deconvolution and upsampling operations to gradually restore the spatial resolution of the feature map, and finally generates a high-quality image corresponding to the input features; the decoder ensures that the generated image is consistent with the original image in structure and details by learning the mapping relationship from the low-dimensional feature space to the high-dimensional image space.

8. The method according to claim 7, characterized in that Step 7 includes: Set input parameters: The user enters a text description and reference image describing the architectural style; Rendering generation: Input parameters are fed into the trained ResU-Net model to generate preliminary building renderings; Effect evaluation and adjustment: By comparing the generated effect pictures with the real effect pictures or the user's expected effect, the quality and accuracy of the generated effect are evaluated, and the model is fine-tuned or retrained according to the evaluation results; The final architectural rendering is output to the user.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is run on a computer, the steps of the method according to any one of claims 1 to 8 are executed.

Images