A User Interface Layout Generation Method Based on a Latent Diffusion Model

Abstract

This invention relates to a user interface layout generation method based on a latent diffusion model, belonging to the field of image generation. This invention extracts components and constraints from interface requirement description text using the BERT model, utilizes a graph convolutional network module to predict complete constraint relationships and borders to generate a complete graph, and generates the user interface layout based on the latent diffusion model. On one hand, introducing a latent diffusion model into user interface generation can meticulously capture the complex relationships and details between UI elements, thereby generating a more accurate user interface that conforms to interaction specifications. On the other hand, introducing a graph convolutional network into user interface generation can effectively predict complete constraint relationships between components, effectively capturing and modeling the complex dependencies and interactions between user interface elements; through graph structures, multi-dimensional constraints such as the position, alignment, and function of elements are applied, generating a more coordinated layout that conforms to design specifications; improving the aesthetics and usability of the generated interface, and enhancing the user experience.

Description

Technical Field

[0001] This invention belongs to the field of image generation and relates to a user interface layout generation method based on a latent diffusion model. Background Technology

[0002] User interface layout plays a crucial role in visual communication. Currently, traditional user interface design primarily relies on the manual creation of interface designers, who complete interface designs of different styles based on relevant interface requirements, design specifications, and their accumulated design experience. However, with technological advancements, researchers have proposed some methods for automated interface design, but most of these methods use static templates or fixed pre-designed rules to automatically generate interface layouts. Although template- and rule-based methods can produce good design results, their applicability is limited, their application scope is restricted, and they struggle to meet diverse and dynamically changing application needs.

[0003] The automatic construction of well-structured and aesthetically pleasing interfaces has significant application value, effectively improving work efficiency and quality in both media content creation and application interface design. Its wide applicability and potential market demand indicate promising research prospects. Thanks to in-depth research into automatic interface generation technology, interface generation techniques can now learn and generate interfaces using machine learning models. Currently, automatic interface layout generation technology is mainly divided into two methods: those based on traditional machine learning and those based on deep learning. Automatic interface layout generation technology based on traditional machine learning relies primarily on pre-defined rules or templates, combined with machine learning algorithms to generate interface layouts. Therefore, the generation of interface layouts is limited by expert-defined rules or templates, resulting in significant limitations and relatively simple layouts. In the field of deep learning, user interface layout generation technology mainly employs an encoder-decoder architecture to achieve interface layout generation. Among these, Variational Auto-Encoders (VAEs), Generative Adversarial Networks (GANs), and Diffusion Models (DMs) constitute the mainstream models for interface generation. VAE is a generative model based on variational inference that learns the latent distribution of data by maximizing the likelihood function. LayoutVAE is a layout model that generates corresponding element layouts based on given element labels and a VAE generative model. However, VAE training is slow, resulting in poor generated interface layouts. GAN is a generative model consisting of a generator and a discriminator. The generator's task is to generate data similar to real data, while the discriminator's task is to determine whether the generated data is similar to real data. During training, the two will continuously engage in a game to achieve better generation results. LayoutGAN model models the geometric relationships and inverse models of different types of elements to generate wireframe layouts. Although GAN can generate very realistic samples, this also brings problems with training stability and convergence. DM generates new data by gradually diffusing high-dimensional data into low-dimensional data. Its training process is similar to a diffusion process, starting from the initial raw input data and gradually transforming the data into low-dimensional latent variable representations. LayoutDiffusion model fuses image and layout information and generates multi-object images based on DM. DM demonstrates good performance in various generation tasks, such as image and text tasks. However, most current methods in the field of user interface generation only consider generating interface layouts based on given elements and constraints between elements, neglecting to consider generating complete element descriptions and user interface layouts from the perspective of user-defined textual requirements. This results in limited ability to support personalized customization and automated layout generation for user interfaces, and the need for significant expertise to operate existing user interfaces, leading to a high design threshold. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a user interface layout generation method based on a potential diffusion model, which aims to lower the threshold of interface design, allow users to personalize the interface, and improve interaction efficiency and user experience.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A user interface layout generation method based on a latent diffusion model includes the following steps:

[0007] S1. Extract components and constraints from the interface requirement description text based on the Bidirectional Encoder Representation from Transformers (BERT) model. Specifically, the original interface requirement text data is first organized and segmented into text to be processed. Then, a text classification model is constructed using the BERT model to obtain the user interface component types and some constraint relationships between user interface components from the text to be processed.

[0008] S2. Use the Graph Convolutional Networks (GCN) module to predict complete constraint relationships and bounding boxes to generate a complete graph. Specifically, the user interface component types and partial constraint relationships between user interface components obtained from S1 are taken as input, processed by the GCN module, to obtain the complete constraint relationships between each user interface component.

[0009] S3. Generate user interface layout based on latent diffusion model. Specifically, the complete constraint relationship between user interface components obtained from S2 and the original interface image are taken as input and processed by the perceptual image compression, forward noise addition and reverse noise reduction process of latent diffusion model to obtain a user interface layout diagram.

[0010] The image compression process described above is used to achieve mutual conversion between the image and the feature space. Specifically, the encoder E of a Vector Quantized Variational Autoencoder (VQ-VAE) compresses the image x into a feature vector z; the decoder D of the VQ-VAE then restores the feature vector z back to an image with an interface layout.

[0011] The forward noise addition process is used to denoise the normally distributed variable to learn the data distribution p(x), and adds noise to the feature vector z to represent the latent variable z at time T during the diffusion process. T The reverse denoising process is used to convert zT To restore z, the core structure of the denoising process is a U-Net network composed of cross-attention.

[0012] The forward noise addition process is implemented using the following method:

[0013] Let T∈N be the total time step of the diffusion model. During the forward noise addition process, for a given K class z at time step t∈N... t z is a scalar discrete variable ∈{1,2,…,K}. t-1 Passed to z t The probability is obtained by using the transition matrix Q. t ∈[0,1] K×K By definition, then:

[0014] q[z t |z t-1 ]=v(z t ) T Q t v(z t-1 )]

[0015] Among them, [Q t ]=q(z t =m|z t-1 =n), v(z) t )∈{0,1} K Represents z t The one-hot vector; given z t-1 time z t The categorical distribution on the vector Q is determined by the column vector Q. t v(z t-1 )∈[0,1] K Calculations were performed, and the values ​​from z0 to z were derived based on Markov properties. t probability distribution:

[0016]

[0017] in,

[0018] The reverse denoising process is implemented using the following method:

[0019] Reverse the forward noise addition process, and from the feature vector z at time T during the diffusion process... T The initial features z0 and z are gradually restored. t To z t-1 The joint distribution is:

[0020]

[0021] Due to the Markov property, the formula q(z) t |z t-1,z0)=q(z t |z t-1 );

[0022] Through neural network p θ (z t-1 |z t )∈[0,1] N×K Consider z t-1 In z t Conditional distribution on; z t-1 Based on this distribution, sampling is performed, and p is parameterized using a bidirectional Transformer encoder block. θ (z t-1 |z t Overall noise reduction objective The calculation formula is as follows:

[0023]

[0024] Where λ is a hyperparameter balancing the two loss terms. For the variational lower bound objective, To assist in the denoising target.

[0025] The reverse denoising process also includes a conditional constraint mechanism, which is implemented using the following method:

[0026] Encode the preconditions into a feature vector τ θ In the initial state z T Injecting conditional information into the sampling information, and then performing a reverse denoising process p θ (z t-1 |z t The learning conditional distribution (y) is represented as follows:

[0027]

[0028] Among them, the forward noise addition process q(z) 1:T |z0) remains unchanged, and y is the true label vector of the sentence;

[0029] The encoder of the GCN model maps the user interface component types and the constraints between user interface components to a vector space, and the cross-attention mechanism incorporates the conditions into the intermediate layer of U-Net, ultimately obtaining the loss function of the latent diffusion model with added conditions. for:

[0030]

[0031] The beneficial effects of this invention are as follows:

[0032] Introducing a latent diffusion model into user interface generation allows for a more detailed capture of the complex relationships and details between UI elements, resulting in a more accurate user interface that conforms to interaction specifications. It also offers advantages such as more stable training and more controllable generation results, reducing pattern collapse. Furthermore, it enables a more effective integration of various design constraints and aesthetic standards, ensuring that the generated interface is not only aesthetically pleasing but also fully functional.

[0033] Furthermore, S2 is implemented using the following method:

[0034] S2 is implemented using the following method:

[0035] The graph is constructed from user interface components E = {e1, e2, ..., eM} and user interface component constraints C = {c1, c2, ..., cN}. When constructing links between element nodes and constraint nodes, an adjacency matrix A is used to describe all adjacency relationships on the graph. The GCN layer has a two-layer structure, with each layer consisting of multiple GCN units. Word embedding processing is performed on the user interface components and component relationships obtained from S1 to obtain the adjacency matrix A and the node feature matrix X. The node feature update formula for each layer is:

[0036]

[0037] Among them, H (l+1) H is the node feature matrix of layer l+1, representing the features of each node in this layer after convolution; (l) It is the input of the (l+1)th layer, representing the node feature matrix of the lth layer, that is, the feature of each node in this layer; H (0) Let X be the input feature matrix of the initial layer, where each row of X represents the initial feature of a node. This is the normalized adjacency matrix, representing the node connection relationships in the graph; I is the identity matrix, indicating the addition of a self-join to the adjacency matrix; To add a self-connected adjacency matrix, yes The degree matrix, W (l) Let be the weight matrix of the l-th layer, and let ReLU be the activation function.

[0038] The component and constraint relationship features output from the GCN layer are input into the relationship predictor to predict the complete constraint relationship; the multi-class cross-entropy loss function is used to predict the edge constraint relationship, specifically: assuming T uv The loss function represents the type of true edge constraint between node u and node v. The calculation formula is as follows:

[0039]

[0040] Where k is the number of edge types, T uvk S is the type label vector of edge (u,v). uvk This represents the probability that the constraint relationship between nodes u and v belongs to category k.

[0041] Further beneficial effects of this invention are as follows: It innovatively introduces graph convolutional networks into user interface generation, which can effectively predict the complete constraint relationships between components; it can effectively capture and model the complex dependencies and interactions between user interface elements, and through graph structures, it can generate more coordinated layouts that conform to design specifications by imposing multi-dimensional constraints on the position, alignment, and function of elements; it enhances the model's understanding of layout logic and consistency, improving the aesthetics and usability of the generated interface; in addition, it promotes constraint management during the layout generation process, ensuring that the generated interface achieves the best balance between diversity and standardization, significantly improving the user experience.

[0042] Furthermore, this invention also supports user input of interface description text to generate user interfaces. The trained model controls component alignment, arrangement, and relative size control when generating the layout, without requiring manual definition by the user, thus lowering the barrier to interface design. Users can customize the input of interface description text to control the generation of interface layout, supporting personalized customization of interface layout. By constructing multiple models to generate user interface layout, users only need to input the interface description according to their needs, and the model can generate the layout diagram of the user interface, resulting in a high degree of automation.

[0043] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0044] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0045] Figure 1 A flowchart of a user interface layout generation method based on a potential diffusion model;

[0046] Figure 2 This is a diagram of the model structure for recognizing and classifying user description text based on the BERT model.

[0047] Figure 3 The structural diagram of a potential diffusion model incorporating conditional controls. Detailed Implementation

[0048] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0049] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0050] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0051] Please see Figure 1 The flowchart of a user interface layout generation method based on a latent diffusion model provided by the present invention mainly includes the following steps:

[0052] Step 1: Extract components and constraints from the interface requirement text based on the BERT model, specifically through the following methods:

[0053] First, the original interface requirement text is preprocessed, such as spell correction, stop word removal, part-of-speech tagging, and sentence splitting, to clean and segment the original, unstructured interface requirement text data into sentences that are easy to process.

[0054] Then, a text classification model is built based on the Bidirectional Encoder Representation from Transformers (BERT) model. The trained BERT model can extract user interface component types and some constraint relationships between user interface components from the user interface requirement description text. See also... Figure 2 This is a diagram of the BERT model for recognizing and classifying user-described text. The BERT model consists of an input layer, an encoding layer, an attention mechanism layer, and a decision layer.

[0055] The input layer, comprising a tokenizer and an embedding layer, first segments the input text using the tokenizer. Then, the embedding layer encodes individual characters, positional information, and relationships between sentences. Finally, a vector embedding layer transforms the original input text into a specific-dimensional vector. The embedded vectors are then summed, and the resulting vector is normalized and randomly deactivated before being used as input to the encoding layer.

[0056] The encoding layer consists of encoders from multiple Transformers. Each Transformer encoder unit is generated by stacking a multi-head self-attention mechanism, a feedforward neural network, and a normalization layer. The self-attention head performs a linear transformation on the input vector using three weight matrices to obtain a new encoded vector, thus distinguishing the weights of different parts of the input. The multi-head self-attention unit connects multiple different sub-attention heads, and then passes them through a fully connected layer for dimensionality reduction. The feedforward neural network performs a linear transformation on the feature vectors of each character to enhance the model's expressive power. The normalization layer accelerates the model's convergence speed. The encoding layer ultimately obtains the encoded text matrix h (the feature vectors of each character), and uses h as the input to the attention mechanism layer and the decision layer.

[0057] The attention mechanism layer receives the feature matrix h from the encoding layer as input and generates a single-word level attention vector α. The sigmoid function is used as the activation function for the attention vector α, which restricts each component value of α to between 0 and 1. A larger value indicates greater importance of the corresponding word's feature vector in the classification result. The calculation formula is as follows:

[0058] α = Sigmoid(h*W)

[0059] Where W is the weight parameter.

[0060] The decision layer consists of a fully connected layer and an activation function layer. First, the output matrix h of the encoding layer is multiplied by the attention vector α to obtain the final sentence feature vector h*, thus altering the influence of different levels of importance information in h on the classification result. h* is then input into the fully connected layer and the activation function layer to obtain the probability vectors of the current sentence belonging to each category. The category corresponding to the highest probability value is the prediction result. The calculation formula is as follows:

[0061]

[0062] Where W' is the network weight and b' is the bias.

[0063] This invention employs cross-entropy loss and KL divergence as the model's loss functions. Cross-entropy loss is used to calculate the classification loss, and KL divergence loss is used to calculate the direction difference loss. and component feature vectors The difference between them. The formula for calculating the total loss function of the model is as follows:

[0064]

[0065] in Cross-entropy loss is used to measure the difference between the model's predicted probability distribution and the true label. The KL (Kullback-Leibler) divergence is used to measure the similarity between two probability distributions. The total loss function of the model is and The combination of M and y; M represents the number of categories, and y is the sentence's true label vector. and It is the normalized feature vector.

[0066] Step 2: Graph Convolutional Networks (GCNs) predict user interface constraints and bounding boxes to generate a complete graph. This is achieved through the following methods:

[0067] Graphical user interfaces (GUIs) contain various elements arranged in different ways, such as text boxes, buttons, icons, graphics, and sliders. The graph structure data, composed of components and their constraints, effectively conveys the spatial-semantic structure of the user interface. A typical graph can be represented by a tuple consisting of a vertex set and an edge set, i.e., G = (V, E). Here, G represents the graph, and V and E are the vertex and edge sets that constitute the graph. The graph structure data includes element nodes and constraint edges. Element nodes represent element attributes, including visual appearance, text content, element type, position, and size. Constraint edges represent positional constraints between element nodes, including orientation, relative size, alignment, and grouping constraints.

[0068] Although step 1 can extract component types and some user-defined constraints from the text description, it is difficult to fully obtain the constraint relationships between all components based on the text description alone, and unknown constraint relationships still exist between elements. Therefore, this invention proposes a graph convolutional network (GCN)-based method to predict the complete constraint relationships between components, adopting a "graph input, graph output" architecture.

[0069] The graph is constructed from the obtained components E = {e1, e2, ..., eM} and constraints C = {c1, c2, ..., cN}. When constructing links between element nodes and constraint nodes, an adjacency matrix A, representing the direct connections between nodes in the graph, describes all adjacency relationships. A two-layer GCN is constructed within the GCN layer, consisting of multiple GCN units. Word embedding processing is performed on the components and component relationships obtained from step 1 to obtain the adjacency matrix A and the node feature matrix X. The node feature update formula for each layer is:

[0070]

[0071] Where H (l+1) H is the node feature matrix of layer l+1, representing the features of each node in this layer after convolution; (l) H is the node feature matrix of layer l, representing the features of each node in that layer, and is the input of layer l+1; (0) X is the input feature matrix of the initial layer, where each row represents the initial features of a node; It is a normalized adjacency matrix, representing the node connection relationships in the graph. I is the identity matrix, which represents adding self-connections to the adjacency matrix, meaning that each node is connected to itself. Adding self-connections allows nodes to retain their own feature information during graph convolution. yes The degree matrix, W (l)Let be the weight matrix of the l-th layer, and ReLU be the activation function. The final result obtained from this layer effectively represents the spatial characteristics of the entire graph structure data, which is beneficial for subsequent network modules to predict the complete constraint relationships between various components.

[0072] The relation predictor predicts the constraints of components, ultimately obtaining a complete node representation. This layer consists of multiple fully connected networks, mainly divided into two sub-modules: a bounding box regression network and a relation prediction network, to predict element sizes and constraints. By inputting the component and constraint features output from the GCN layer into the relation predictor, the complete constraint relationships predicted by the model can be obtained. For edge constraint prediction, we use a multi-class cross-entropy loss function. Assuming Tuv represents the true edge constraint type between node u and node v, the loss function is calculated as follows:

[0073]

[0074] Where K is the number of edge types, Tuvk is the type label vector of edge (u,v), and S uvk This represents the probability that the constraint relationship between nodes u and v belongs to category k.

[0075] Because step 2 innovatively introduces Graph Convolutional Networks (GCNs) into the user interface generation, it can effectively predict the complete constraint relationships between components. GCNs can effectively capture and model the complex dependencies and interactions between user interface (UI) elements. Through graph structures, it constrains elements' positions, alignments, and functions in multiple dimensions, generating more harmonious layouts that conform to design specifications. Compared to traditional methods, GCNs enhance the model's understanding of layout logic and consistency, improving the aesthetics and usability of the generated interface. Furthermore, the introduction of GCNs facilitates constraint management during the layout generation process, ensuring that the generated interface achieves an optimal balance between diversity and standardization, significantly improving the user experience.

[0076] Step 3: Generate the user interface layout based on the potential diffusion model, specifically through the following methods:

[0077] The diffusion model comprises a forward process and a reverse process. The forward process, also known as the diffusion process, progressively adds noise until a set of pure noise is reached. The reverse process restores the input from a set of random noise, thus performing a denoising process. Both the forward and reverse processes are parameterized Markov chains, with the reverse process used to generate data samples. General diffusion models, such as the Denoising Diffusion Probabilistic Model (DDMP), operate in the image pixel space, making training and inference resource-intensive. To improve the inference and generation efficiency of diffusion models, this paper uses a Latent Diffusion Model (LDM), shifting the diffusion space from the pixel space to the latent space. The probability distribution in the latent space can be obtained using a trained VAE. Predicting the probability distribution not only improves generation efficiency but also avoids overtraining of the diffusion model on image pixels. Since these image details are handled by the VAE, the quality of the generated image is significantly improved. Finally, the latent diffusion model, through the cross-attention module, supports the addition of different conditions to the diffusion process, thereby achieving controllability of the generated content of the generative model.

[0078] Please see Figure 3 This is a structural diagram of a latent diffusion model with added conditional controls. The latent diffusion model mainly includes a perceptual image compression module, a diffusion model module, and a conditional mechanism module.

[0079] The Perceptual Image Compression module employs a Vector Quantized Variational Autoencoder (VQ-VAE) to achieve mutual conversion between image and feature space. VQ-VAE consists of an encoder E and a decoder D. The encoder compresses the image x into a feature vector z, and the decoder restores this feature vector z back to an image with an interface layout.

[0080] The diffusion model is a probabilistic model that aims to learn the data distribution p(x) by progressively denoising normally distributed variables. The forward denoising process adds noise to the feature z, resulting in z0. T The core structure of the reverse denoising process is a U-Net network composed of cross-attention, used to process z... TReverting to z. The latent diffusion model operates in the latent space, where the feature size is much smaller than the image space, thus making its inference speed much faster. Furthermore, the latent space in which the latent diffusion model operates is a visual codebook with practical meaning, which allows the diffusion model portion of the latent diffusion model to focus more on the semantic information of the generated image rather than the texture details of the image.

[0081] In the forward noise addition process, let T∈N be the total time step of the diffusion model. For a given K class z at time step t∈N, t z is a scalar discrete variable ∈{1,2,…,K}. t-1 Passed to z t The probability is obtained by using the transition matrix Q. t ∈[0,1] K×K Define, where [Q] t ]=q(z t =m|z t-1 =n):

[0082] q[z t |z t-1 ]=v(z t ) T Q t v(z t-1 )]

[0083] Where v(z) t )∈{0,1} K It is z t The one-hot vector. Given z t-1 time z t The categorical distribution on the vector Q is determined by the column vector Q. t v(z t-1 )∈[0,1] K Calculation. Based on the Markov property, the calculation from z0 to z can be derived. t The probability distribution, where

[0084]

[0085] The reverse denoising process requires reversing the aforementioned forward denoising process, starting from z... T The initial features z0 and z are gradually restored. t To z t-1 The joint distribution is:

[0086]

[0087] Due to the Markov property, the future state of a stochastic process depends only on the current state and is independent of its past states. Therefore:

[0088] q(zt |z t-1 ,z0)=q(z t |z t-1 )

[0089] In contrast to the forward noise addition process, the backward noise reduction process uses a neural network p θ (z t-1 |z t )∈[0,1] N×K Consider z t-1 In z t Conditional distribution on z. t-1 Based on this distribution, sampling is performed, and p is parameterized using a bidirectional Transformer encoder block. θ (z t-1 |z t ).

[0090] The overall denoising objective is a variational lower bound objective. The formula for calculating the addition of the auxiliary denoising target is as follows:

[0091]

[0092] Where λ is a hyperparameter that balances the two loss terms.

[0093] The conditional mechanism module introduces external constraints or information to ensure the generation process conforms to specific constraints or requirements. By injecting external conditions into the model generation process, the model can guide the output direction during the generation phase based on these conditions, thereby avoiding the generation of results that do not meet the constraints. The conditional encoder encodes the preconditions into a feature vector τ. θ In the initial state z T Injecting conditional information into the sampling information, and then performing a reverse denoising process p θ (z t-1 |z t ,y) learns the conditional distribution without modifying the forward noisy process q(z) 1:T |z0). The process can be represented as:

[0094]

[0095] The encoder of a pre-trained graph convolutional network model maps the graph-structured data (including category and constraint information) to a vector space, and a cross-attention mechanism is used to incorporate conditions into the intermediate layers of U-Net. Finally, the loss function of the conditional latent diffusion model is added. for:

[0096]

[0097] Step 3, by introducing a latent diffusion model into the user interface generation, effectively improves the quality and diversity of the generated layout. Through a progressive noise addition and denoising process, the latent diffusion model can more meticulously capture the complex relationships and details between UI elements, thereby generating a more accurate user interface that conforms to interaction specifications. Compared to GANs and VAEs, the diffusion model has the advantages of more stable training, more controllable generation results, and reduced pattern collapse. Furthermore, the latent diffusion model can more effectively integrate multiple design constraints and aesthetic standards, ensuring that the generated interface is not only aesthetically pleasing but also functionally complete.

[0098] To illustrate the differences between this model and existing models, this embodiment further proposes a comparative evaluation of its effectiveness in practical applications, specifically utilizing the following evaluation indicators:

[0099] FID (Fréchet Inception Distance) is used to measure the difference between the predicted layout and the actual layout to evaluate the generative model. The evaluation formula is as follows:

[0100]

[0101] Where Tr represents the sum of the elements on the diagonal of the matrix, which is the trace of the matrix. x represents the actual layout vector, and g represents the generated layout vector. A lower FID means that the two distributions are closer, which means that the generated layouts are of higher quality and more diverse.

[0102] Aesthetic standards, particularly alignment, are an important criterion for measuring whether a generated layout conforms to design guidelines. In most good designs, elements are either centered or edge-aligned.

[0103]

[0104] Where N D Indicates the number of layouts generated. This represents the k-th element in the d-th layout. l, m, and r represent the distances between the left, center, and right edges of the elements.

[0105] IoU is used to calculate the overlap ratio between the predicted bounding box and the ground truth bounding box. The calculation formula is as follows:

[0106]

[0107] Where So represents the area of ​​the overlapping portion between the predicted UI component layout and the actual component, and Su represents the combined area of ​​the predicted UI component layout and the actual component. A higher IoU value indicates a higher overlap rate between the borders of the two components, resulting in a more accurate generated layout.

[0108] This embodiment uses the LayoutVAE model and the LayoutGAN model as benchmarks to evaluate the performance of the user interface layout generation method based on the latent diffusion model proposed in this embodiment in the layout generation task. The specific results are shown in Table 1.

[0109] Table 1

[0110] Model FID↓ Aesthetic standards ↓ IoU↑ LayoutVAE 6.725 0.743 0.334 LayoutGAN 6.576 0.779 0.374 This embodiment 6.244 0.736 0.387

[0111] The experimental data used in this embodiment are as follows:

[0112] The Magazine dataset contains over 4,000 layout images of magazine pages, with a total of 6 component categories.

[0113] The Rico dataset is a dataset for mobile application user interface design, containing screenshots of over 90,000 Android mobile applications, categorized into 25 elements such as text buttons, toolbars, and icons.

[0114] The Ali-UI dataset contains 1,613 product ad layout datasets in e-commerce scenarios, covering 8 component categories.

[0115] As shown in Table 1, the user interface layout generation method based on the latent diffusion model proposed in this embodiment achieved the best results in both FID and IoU metrics in the layout generation task, thus verifying that the method shows a smaller difference between the predicted layout and the actual layout; it also obtained the highest alignment score in terms of aesthetic quality.

[0116] This invention enables the automatic generation of user interfaces by allowing users to input descriptive text. The trained model automatically controls the alignment, arrangement, and relative size of components during layout generation, eliminating the need for manual user definition and thus lowering the barrier to entry for interface design. Furthermore, this invention supports user-defined input of descriptive text to control the generation of the interface layout, enabling personalized customization. By constructing multiple models, users only need to input a description of the interface according to their needs, and the model can automatically generate a user interface layout diagram, achieving a high degree of automation.

[0117] This invention extracts components and constraints from interface requirement description text based on the BERT model, uses a graph convolutional network module to predict complete constraint relationships and generate a complete graph, and generates the user interface layout based on a latent diffusion model. On one hand, introducing a latent diffusion model into the user interface generation process can meticulously capture the complex relationships and details between UI elements, thus generating a more accurate user interface that conforms to interaction specifications. On the other hand, introducing a graph convolutional network into the user interface generation process can effectively predict complete constraint relationships between components, effectively capturing and modeling the complex dependencies and interactions between user interface elements; through graph structures, multi-dimensional constraints such as the position, alignment, and function of elements are applied, generating a more coordinated layout that conforms to design specifications; improving the aesthetics and usability of the generated interface, and enhancing the user experience.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A user interface layout generation method based on a latent diffusion model, characterized in that, Includes the following steps: S1. Extract components and constraints from the interface requirement description text based on the Bidirectional Encoder Representation from Transformers (BERT) model. Specifically, the original interface requirement text data is first organized and segmented into text to be processed. Then, a text classification model is constructed using the BERT model to obtain the user interface component types and some constraint relationships between user interface components from the text to be processed. S2. Use the Graph Convolutional Networks (GCN) module to predict complete constraint relationships and bounding boxes to generate a complete graph. Specifically, the user interface component types and partial constraint relationships between user interface components obtained from S1 are taken as input, processed by the GCN module, to obtain the complete constraint relationships between each user interface component. S3. Generate user interface layout based on latent diffusion model, specifically: take the complete constraint relationship between user interface components obtained from S2 and the original interface image as input, process them through the perceptual image compression, forward noise addition and reverse noise reduction process of latent diffusion model to obtain a user interface layout diagram; S2 is implemented using the following method: The graph is constructed from user interface components E = {e1, e2, ..., eM} and user interface component constraints C = {c1, c2, ..., cN}. When constructing links between element nodes and constraint nodes, an adjacency matrix A is used to describe all adjacency relationships on the graph. The GCN layer has a two-layer structure, with each layer consisting of multiple GCN units. Word embedding processing is performed on the user interface components and component relationships obtained from S1 to obtain the adjacency matrix A and the node feature matrix X. The node feature update formula for each layer is: Among them, H (l+1) H is the node feature matrix of layer l+1, representing the features of each node in this layer after convolution; (l) It is the input of the (l+1)th layer, representing the node feature matrix of the lth layer, that is, the feature of each node in this layer; H (0) Let X be the input feature matrix of the initial layer, where each row of X represents the initial feature of a node. This is the normalized adjacency matrix, representing the node connection relationships in the graph; I is the identity matrix, indicating the addition of a self-join to the adjacency matrix; To add a self-connected adjacency matrix, yes The degree matrix, W (l) Let be the weight matrix of the l-th layer, and let ReLU be the activation function. The component and constraint relationship features output from the GCN layer are input into the relationship predictor to predict the complete constraint relationship; the multi-class cross-entropy loss function is used to predict the edge constraint relationship, specifically: assuming T uv The loss function represents the type of true edge constraint between node u and node v. The calculation formula is as follows: Where k is the number of edge types, T uvk S is the type label vector of edge (u,v). uvk This represents the probability that the constraint relationship between nodes u and v belongs to category k; The forward noise addition process is used to denoise the normally distributed variable to learn the data distribution p(x), and adds noise to the feature vector z to represent the latent variable z at time T during the diffusion process. T The reverse denoising process is used to convert z T To restore z, the core structure of the denoising process is a U-Net network composed of cross-attention; The reverse denoising process also includes a condition constraint mechanism, which is implemented using the following method: Encode the preconditions into a feature vector τ θ In the initial state z T Injecting conditional information into the sampling information, and then performing a reverse denoising process p θ (z t-1 |z t The learning conditional distribution (y) is represented as follows: Among them, the forward noise addition process q(z) 1:T |z0) remains unchanged, and y is the true label vector of the sentence; The encoder of the GCN model maps the user interface component types and the constraints between user interface components to a vector space, and the cross-attention mechanism incorporates the conditions into the intermediate layers of U-Net, ultimately obtaining the loss function of the latent diffusion model with added conditions. for:

2. The user interface layout generation method based on a latent diffusion model according to claim 1, characterized in that: The image compression process is used to achieve mutual conversion between the image and the feature space. Specifically, the encoder E of a Vector Quantized Variational Autoencoder (VQ-VAE) compresses the image x into a feature vector z; the decoder D of the VQ-VAE then restores the feature vector z back to an image with an interface layout.

3. The user interface layout generation method based on a latent diffusion model according to claim 1, characterized in that, The forward noise addition process is implemented using the following method: Let T∈N be the total time step of the diffusion model. During the forward noise addition process, for a given K class z at time step t∈N... t z is a scalar discrete variable ∈{1,2,…,K}. t-1 Passed to z t The probability is obtained by using the transition matrix Q. t ∈[0,1] K×K By definition, then: q[z t |z t-1 ]=v(z t ) T Q t v(z t-1 )] Among them, [Q t ]=q(z t =n|z t-1 =n), v(z) t )∈{0,1} K Represents z t The one-hot vector; given z t-1 time z t The categorical distribution on the vector Q is determined by the column vector Q. t v(z t-1 )∈[0,1] K Calculations were performed, and the values ​​from z0 to z were derived based on Markov properties. t probability distribution: in, 4. The user interface layout generation method based on a latent diffusion model according to claim 3, characterized in that, The reverse denoising process is implemented using the following method: Reverse the forward noise addition process, and from the feature vector z at time T during the diffusion process... T The initial features z0 and z are gradually restored. t To z t-1 The joint distribution is: Due to the Markov property, the formula q(z) t |z t-1 ,z0)=q(z t |z t-1 ); Through neural network p θ (z t-1 |z t )∈[0,1] N×K Consider z t-1 In z t Conditional distribution on; z t-1 Based on this distribution, sampling is performed, and p is parameterized using a bidirectional Transformer encoder block. θ (z t-1 |z t Overall noise reduction objective The calculation formula is as follows: Where λ is a hyperparameter balancing the two loss terms. For the variational lower bound objective, To assist in the denoising target.

5. The user interface layout generation method based on a latent diffusion model according to claim 1, characterized in that: The method for constructing a text classification model using the BERT model is implemented as follows: The BERT model comprises an input layer, an encoding layer, an attention mechanism layer, and a decision layer. The input layer converts the input text into a vector of a specific dimension. The encoding layer processes the vector into a text-encoded matrix h. The attention mechanism layer uses □ to generate a single-word-level attention vector α, and employs the Sigmoid function to restrict each component value of vector α to between 0 and 1 according to its importance. The calculation formula is as follows: α = Sigmoid(h*W) Where W is the weight parameter; The decision layer consists of a fully connected layer and an activation function layer; first, ... Multiplying by α yields the sentence feature vector h*; h* is then input into the fully connected layer and the activation function layer to obtain the probability vectors of the current sentence belonging to each category. The category corresponding to the highest probability value in a given area is the prediction result, calculated using the following formula: Where W′ is the network weight and b′ is the bias.

6. The user interface layout generation method based on a latent diffusion model according to claim 5, characterized in that: The BERT model is also used to calculate the model's loss; the cross-entropy loss function is used to calculate the classification loss, and the KL divergence (KL divergence) loss function is used to calculate the difference between the normalized feature vectors. The formula for calculating the total model loss is as follows: in, Cross-entropy loss is used to measure the difference between the model's predicted probability distribution and the true label. The KL (Kullback-Leibler) divergence loss is used to measure the similarity between two probability distributions. The total loss of the model is and The combination of M and y; M represents the number of categories, and y is the sentence's true label vector. and It is the normalized feature vector.

Images