User interface layout generation method based on potential diffusion model

Abstract

The invention relates to a user interface layout generation method based on a potential diffusion model, and belongs to the field of image generation. Components and constraints are extracted from an interface demand description text based on a BERT model, a complete constraint relation and a frame are predicted by using a graph convolutional network module to generate a complete graph, and a user interface layout is generated based on a potential diffusion model. On one hand, a potential diffusion model is introduced into user interface generation, complex relations and details among UI elements can be carefully captured, and therefore the user interface which is more accurate and conforms to interaction specifications is generated. On the other hand, the graph convolutional network is introduced into user interface generation, the complete constraint relation between components can be effectively predicted, and complex dependence and interaction between user interface elements are effectively captured and modeled; a layout which is more coordinated and accords with design specifications is generated through multi-dimensional constraints of positions, alignment, functions and the like of elements by a graph structure; the attractiveness and practicability of the generated interface are improved, and the user experience is improved.

Description

Technical Field

[0001] The invention belongs to the field of image generation and relates to a user interface layout generation method based on a potential diffusion model. Background Art

[0002] User interface layout plays a vital role in the field of visual communication. At present, traditional user interface design mainly relies on the manual creation of interface designers. Designers complete interface design work of different styles based on corresponding interface requirements, design specifications and personal accumulated design experience. However, with the advancement of technology, researchers have proposed some methods for automatic interface design, but most of these methods use static templates or fixed pre-designed rules to achieve automatic generation of interface layout. Although template- and rule-based methods can produce better design results, these methods are not very applicable, have limited application scope, and are difficult to meet diverse and dynamically changing application needs.

[0003] Automatically building a layout-reasonable and beautiful interface has significant application value. It can effectively improve work efficiency and quality in both media content creation and application interface design. Its wide range of applications and potential market demand indicate good research prospects. Thanks to the in-depth research on automatic interface generation technology, interface generation technology can be learned and generated through machine learning models. At present, automatic interface layout generation technology is mainly divided into two methods: traditional machine learning-based and deep learning-based. The automatic interface layout generation technology based on traditional machine learning mainly relies on pre-set rules or templates, and combines machine learning algorithms to generate interface layouts. Therefore, the generation of interface layouts is limited to the rules or templates defined by experts, which has great limitations and the generated layout is relatively simple. In the field of deep learning, user interface layout generation technology mainly uses encoder-decoder architecture to realize the generation of interface layout. Among them, variational auto-encoders (VAE), generative adversarial networks (GAN) and diffusion models (DM) constitute the mainstream models of interface generation. VAE is a generative model based on variational inference. It learns the potential distribution of data by maximizing the likelihood function. The LayoutVAE layout model generates the layout model of the corresponding element layout based on the given element label and the VAE generation model. However, the training process of VAE is slow, resulting in poor results in the generated interface layout. GAN is a generative model composed of a generator and a discriminator. The task of the generator is to generate data similar to the real data, while the task of the discriminator is to determine whether the generated data is similar to the real data. During the training process, the two will continue to compete to obtain better generation results. The LayoutGAN layout model models the geometric relationship and retrograde of different types of elements to generate wireframe layouts. Although GAN can generate very realistic samples, it also brings about problems of training stability and convergence. DM generates new data by gradually diffusing high-dimensional data into low-dimensional data. Its training process is similar to a diffusion process. Starting from the initial raw input data, the data is gradually converted into a low-dimensional latent variable representation. The LayoutDiffusion layout model fuses the image and layout information and completes the generation of multi-object images based on DM. DM shows good performance in various generation tasks, such as image and text tasks. At present, in the field of user interface generation, most methods only consider the generation of interface layout given specified elements and constraints between elements, and do not consider the generation of complete element descriptions and user interface layouts from the perspective of user demand text. The user interaction interface has low ability to support personalized customization and automatic layout generation, and operating the existing user interaction interface also requires a lot of professional knowledge, resulting in a high design threshold. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a user interface layout generation method based on a potential diffusion model, aiming to lower the threshold of interface design, allow users to personalize the interface, and improve interaction efficiency and user experience.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for generating a user interface layout based on a potential diffusion model comprises the following steps:

[0007] S1. Extract components and constraints from the interface requirement description text based on the Bidirectional Encoder Representation from Transformers (BERT) model. Specifically, first organize the original interface requirement text data and divide it into text to be processed; then use the BERT model to build a text classification model to obtain the user interface component types and some constraint relationships between user interface components from the text to be processed;

[0008] S2. Use the Graph Convolutional Networks (GCN) module to predict the complete constraint relationship and bounding box to generate a complete graph. Specifically, the user interface component type and partial constraint relationship between user interface components obtained from S1 are used as input, and processed by the GCN module to obtain the complete constraint relationship between each user interface component.

[0009] S3. Generate a user interface layout based on the latent diffusion model. Specifically, the complete constraint relationship between the user interface components obtained from S2 and the original interface image are used as input, and the perceptual image compression, forward denoising and reverse denoising processes of the latent diffusion model are processed to obtain a user interface layout diagram.

[0010] The perceptual image compression process is used to realize the mutual conversion between image and feature space, specifically, the encoder E of Vector Quantised-Variational AutoEncoder (VQ-VAE) is used to compress the image x into a feature vector z; the decoder D of VQ-VAE is used to restore the feature vector z into an image with interface layout

[0011] The forward denoising process is used to denoise the normally distributed variables to learn the data distribution p(x), and the feature vector z is denoised as the latent variable z at time T in the diffusion process. T The reverse denoising process is used to convert zT Reducing to z, the core structure of the denoising process is a U-Net network composed of cross attention.

[0012] The forward noise addition process is implemented by the following method:

[0013] Let T∈N be the total time step of the diffusion model. In the forward noise addition process, for a class z with K at time step t∈N, t ∈{1,2,…,K} scalar discrete variable, z t-1 Pass to z t The probability is obtained by using the transfer matrix Q t ∈[0, 1] K×K Definition, then:

[0014] q[z t |z t-1 ]=v(z t ) T Q t v(z t-1 )]

[0015] Among them, [Q t ]=q(z t =m|z t-1 =n), v(z t )∈{0,1} K Represents z t One-hot vector of t-1 Time t The categorical distribution on is represented by the column vector Q t v(z t-1 )∈[0,1] K Calculate and deduce from z0 to z according to the Markov property t The probability distribution of is:

[0016]

[0017] in,

[0018] The reverse denoising process is implemented by the following method:

[0019] Reverse the forward noise addition process and obtain the feature vector z at time T during the diffusion process T Gradually restore the initial features z0, z t to z t-1 The joint distribution of is:

[0020]

[0021] Due to the Markov property, the formula q(z t |z t-1,z0)=q(z t |z t-1 );

[0022] Through the neural network θ (z t-1 |z t )∈[0,1] N×K Consider z t-1 In z t The conditional distribution on z t-1 Sampling is done according to this distribution and parameterizing p through a bidirectional Transformer encoder block. θ (z t-1 |z t ); Overall goal of denoising The calculation formula is as follows:

[0023]

[0024] Among them, λ is a hyperparameter that balances the two loss terms, is the variational lower bound objective, To assist in denoising.

[0025] The reverse denoising process also includes a conditional constraint mechanism, which is implemented by the following method:

[0026] Encode the precondition into a feature vector τ θ , in the initial state z T and inject conditional information into the sampling information, and then pass the reverse denoising process p θ (z t-1 |z t ,y) Learning conditional distribution, the process is expressed as:

[0027]

[0028] Among them, the forward noise process q(z 1:T |z0) remains unchanged, y is the true label vector of the sentence;

[0029] The encoder of the GCN model is used to map the types of user interface components and the constraint relationships between user interface components into the vector space, and the cross-attention mechanism is used to integrate the conditions into the middle layer of the U-Net, and finally the loss function of the latent diffusion model with added conditions is obtained. for:

[0030]

[0031] The beneficial effects of the present invention are:

[0032] Introducing the latent diffusion model in user interface generation can capture the complex relationships and details between UI elements in a more detailed manner, thereby generating a more accurate user interface that complies with interaction specifications. It has the advantages of more stable training and more controllable generation results, reducing the phenomenon of mode collapse. It can more effectively integrate multiple design constraints and aesthetic standards to ensure that the generated interface is not only beautiful but also fully functional.

[0033] Furthermore, S2 is implemented by the following method:

[0034] The S2 is implemented by the following method:

[0035] The graph is constructed by user interface components E = {e1, e2, ..., eM} and user interface component constraint relations C = {c1, c2, ..., cN}. When constructing links between element nodes and constraint nodes, the adjacency matrix A is used to describe all adjacency relations on the graph. The GCN layer is a two-layer structure, each layer consisting of multiple GCN units. The user interface components and component relations obtained from S1 are word-embedded to obtain the adjacency matrix A and the node feature matrix X. The node feature update formula for each layer is:

[0036]

[0037] Among them, H (l+1) is the node feature matrix of the l+1th layer, which represents the features of each node in this layer after convolution; H (l) is the input of the l+1th layer, representing the node feature matrix of the lth layer, that is, the features of each node in this layer; H (0) is the input feature matrix X of the initial layer, each row of X is the initial feature of a node; is the normalized adjacency matrix, which represents the node connection relationship in the graph; I is the identity matrix, which means adding self-connection to the adjacency matrix; To add the adjacency matrix with self-connection, yes The degree matrix, W (l) is the weight matrix of the lth layer, and the activation function uses the ReLU function;

[0038] The components and constraint relationship features output by the GCN layer are input into the relationship predictor to predict the complete constraint relationship; the edge constraint relationship is predicted using the multi-classification cross entropy loss function, specifically: Assume T uv Represents the true edge constraint type between node u and node v, loss function The calculation formula is as follows:

[0039]

[0040] Where k is the number of edge types, T uvk is the type label vector of the edge (u,v), S uvk Represents the probability that the constraint relationship between node u and node v belongs to category k.

[0041] Further beneficial effects of the present invention are: the innovative introduction of graph convolutional networks in user interface generation can effectively predict the complete constraint relationships between components; it can effectively capture and model the complex dependencies and interactions between user interface elements, and through the multi-dimensional constraints such as position, alignment and function between elements of the graph structure, it can generate a more coordinated and design-compliant layout; it enhances the model's understanding of layout logic and consistency, and improves the aesthetics and practicality of the generated interface; in addition, it also promotes constraint management in the layout generation process, ensuring that the generated interface achieves the best balance between diversity and standardization, significantly improving the user experience.

[0042] Furthermore, the present invention also supports users inputting interface description text to generate a user interface. The trained model will control component alignment, arrangement, relative size control, etc. when generating the layout, without the need for user manual definition, thereby lowering the interface design threshold. Users can customize the input interface description text to control interface layout generation, supporting personalized customization of interface layout. By constructing multiple models to achieve the generation of user interface layout, users only need to input the interface description according to their needs, and the model can generate a layout diagram of the user interface, with a high degree of automation.

[0043] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0045] Figure 1 A flowchart of a method for generating a user interface layout based on a latent diffusion model;

[0046] Figure 2 This is a model structure diagram for identifying and classifying user description text based on the BERT model;

[0047] Figure 3 Structural diagram of the latent diffusion model with conditional controls added. DETAILED DESCRIPTION

[0048] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0049] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0050] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0051] See also Figure 1 , which is a flow chart of a method for generating a user interface layout based on a potential diffusion model provided by the present invention, mainly comprising the following steps:

[0052] Step 1: Extract components and constraints from the interface requirement text based on the BERT model, which is implemented by the following method:

[0053] First, the original interface requirement text is preprocessed, such as spelling correction, stop word removal, part-of-speech tagging, sentence splitting, etc. The original, unstructured interface requirement text data is cleaned and segmented into sentences that are easy to process.

[0054] Then, a text classification model is built based on the Bidirectional Encoder Representation from Transformers (BERT) model. The trained BERT model can obtain the types of user interface components and some constraints between user interface components from the user interface requirement description text. Figure 2 , is a model structure diagram for recognizing and classifying user description text based on the BERT model. The BERT model includes an input layer, an encoding layer, an attention mechanism layer, and a decision layer.

[0055] The input layer includes a word segmenter and an embedding layer. The word segmenter first segments the input text, and then the embedding layer encodes the words, position information, and sentences of the sentence. The vector embedding layer converts the original input text into a vector of a specific dimension. The embedded vectors are added together, and the added vectors are processed by the normalization layer and random inactivation before being used as the input of the encoding layer.

[0056] The encoding layer is composed of encoders in multiple Transformers. The encoder unit of a Transformer is generated by the superposition of a multi-head self-attention mechanism, a feedforward neural network and a normalization layer. The self-attention head linearly transforms the input vector through three weight matrices to obtain a new vector encoding, thereby distinguishing the weights of different parts of the input. The multi-head self-attention unit connects multiple different sub-attention heads and then reduces the dimension of the output through a fully connected layer. The feedforward neural network is used to linearly transform the feature vectors of each word to enhance the expressiveness of the model. The normalization layer is used to accelerate the convergence of the model. The encoding layer finally obtains the matrix h (feature vectors of each word) after text encoding, and uses h as the input of the attention mechanism layer and the decision layer.

[0057] The attention mechanism layer receives the feature matrix h passed in by the encoding layer as input and generates a single-word level attention vector α. The Sigmoid function is used as the activation function of the attention vector α. This activation function limits the value of each component of the vector α to between 0 and 1. The larger the value, the greater the importance of the feature vector of the corresponding word in the classification result. The calculation formula is as follows:

[0058] α=Sigmoid(h*W)

[0059] Where W is the weight parameter.

[0060] The decision layer consists of a fully connected layer and an activation function layer. First, the encoding layer output matrix h is multiplied by the attention vector α to obtain the final sentence feature vector h*, thereby changing the impact of different importance information in h on the classification result. h* is input into the fully connected layer and the activation function layer to obtain the probability vector of the current sentence belonging to each category. The category corresponding to the highest probability value is the prediction result. The calculation formula is as follows:

[0061]

[0062] Where W' is the network weight and b' is the bias.

[0063] The present invention uses cross entropy loss and KL divergence (Kullback-Leibler Divergence) as the loss function of the model. Cross entropy loss is used to calculate the classification loss, and KL divergence loss is used to calculate the and the component feature vector The total loss function of the model is calculated as follows:

[0064]

[0065] in is the cross entropy loss, which is used to measure the gap between the model's predicted probability distribution and the true label; KL (Kullback-Leibler) divergence, which is used to measure the similarity of two probability distributions; is the total loss function of the model, and M represents the number of categories, y is the true label vector of the sentence, and is the normalized eigenvector.

[0066] Step 2: Graph Convolutional Networks (GCN) predicts UI constraints and bounding boxes to generate a complete graph. This is achieved by:

[0067] There are various elements arranged in various ways in the graphical user interface, such as text boxes, buttons, icons, graphics, sliders, etc. The graph structure data composed of components and component constraints can effectively convey the spatial-semantic structure of the user interface. A general graph can be represented by a tuple consisting of a vertex set and an edge set, that is, G = (V, E). G represents the graph, and V and E are the vertex set and edge set that constitute the graph. The graph structure data includes element nodes and constraint edges. Element nodes represent element attributes, including visual appearance, text content element type, position and size, etc. Constraint edges represent position constraints between element nodes, including: orientation, relative size, alignment and grouping constraints.

[0068] Although the component types and some user-defined constraints in the text description can be extracted in step 1, it is difficult to fully obtain the constraint relationships between all components based on the text description, and there are still unknown constraint relationships between elements. Therefore, the present invention proposes a method based on a graph convolutional network (GCN) to predict the complete constraint relationship between components, using a "graph input, graph output" architecture.

[0069] The graph is constructed by the obtained components E = {e1, e2, ..., eM} and the constraint relations C = {c1, c2, ..., cN}. When constructing the links between element nodes and constraint nodes, the adjacency matrix A representing the direct connection of nodes in the graph is used to describe all adjacency relations on the graph. A two-layer GCN is constructed in the GCN layer. The GCN layer consists of multiple GCN units. The components and component relations obtained in step 1 are word-embedded to obtain the adjacency matrix A and the node feature matrix X. The node feature update formula of each layer is:

[0070]

[0071] Among them, H (l+1) is the node feature matrix of the l+1th layer, which represents the features of each node in this layer after convolution; H (l) is the node feature matrix of the lth layer, representing the features of each node in this layer, and is the input of the l+1th layer; H (0) is the input feature matrix X of the initial layer, where each row is the initial feature of a node; It is a normalized adjacency matrix that represents the node connection relationship in the graph. I is the unit matrix, which means adding self-connection to the adjacency matrix, that is, each node is connected to itself. Adding self-connection can enable the node to retain its own feature information during the graph convolution process. yes The degree matrix, W (l)is the weight matrix of the lth layer, and the activation function is the ReLU function. The final result of this layer can well express the spatial characteristics of the entire graph structure data, which is conducive to the subsequent network module to predict the complete constraint relationship between each component.

[0072] In the relationship predictor, the constraint relationship of the component can be predicted, and finally a complete node representation can be obtained. This layer consists of multiple fully connected networks, which are mainly divided into two sub-modules, namely the bounding box regression network and the relationship prediction network, to predict the element size and constraint relationship. The features of the components and constraint relationships output by the GCN layer are input into the relationship predictor to obtain the complete constraint relationship predicted by the model. For edge constraint relationship prediction, we use a multi-classification cross entropy loss function. Assuming that Tuv represents the true edge constraint type between node u and node v, the loss function is calculated as follows:

[0073]

[0074] Where K is the number of edge types, Tuvk is the type label vector of the edge (u,v), and S uvk Represents the probability that the constraint relationship between node u and node v belongs to category k.

[0075] Because step 2 innovatively introduces graph convolutional networks (GCN) in user interface generation, it can effectively predict the complete constraint relationship between components. GCN can effectively capture and model the complex dependencies and interactions between user interface (UI) elements, and can generate a more coordinated and design-compliant layout through multi-dimensional constraints such as position, alignment, and function between elements in the graph structure. Compared with traditional methods, GCN enhances the model's understanding of layout logic and consistency, and improves the aesthetics and practicality of the generated interface. In addition, the introduction of GCN promotes constraint management in the layout generation process, ensuring that the generated interface achieves the best balance between diversity and standardization, significantly improving the user experience.

[0076] Step 3: Generate a user interface layout based on the potential diffusion model, which is specifically achieved by the following method:

[0077] The diffusion model includes a forward process and a reverse process. The forward process is also called a diffusion process. In the forward process, noise is gradually added to a set of pure noise. The reverse process restores a set of random noise to the input. This process is a denoising process. Both the forward process and the reverse process are parameterized Markov chains, and the reverse process can be used to generate data samples. General diffusion models, such as the Denoising Diffusion Probabilistic Model (DDMP), operate in the image pixel space. Training and reasoning are very resource-intensive. In order to improve the reasoning efficiency and generation efficiency of the diffusion model, the latent diffusion model (LDM) is used in this paper to transfer the diffusion space from the pixel space to the latent space. The probability distribution of the latent space can be obtained through the trained VAE. In addition to improving the generation efficiency, the predicted probability distribution can also avoid over-training of the diffusion model on the image pixels. These details of the image are completed by VAE, which can significantly improve the quality of the generated image. Finally, the latent diffusion model supports adding different conditions to the diffusion process through the cross-attention module, thereby achieving the controllability of the generated content of the generative model.

[0078] See also Figure 3 , which is the structural diagram of the latent diffusion model with conditional control. The latent diffusion model mainly includes the perceptual image compression module, the diffusion model module and the conditional mechanism module.

[0079] The Perceptual Image Compression module uses a vector quantized variational autoencoder (VQ-VAE) to achieve the conversion between image and feature space. VQ-VAE consists of an encoder E and a decoder D. The encoder is used to compress the image x into a feature vector z, and the decoder restores this feature vector z to an image with an interface layout.

[0080] Diffusion model, diffusion model is a probabilistic model that aims to learn the data distribution p(x) by gradually denoising the normally distributed variables. The forward denoising process adds noise to the feature z to z T The core structure of the reverse denoising process is a U-Net network composed of cross attention, which is used to transform z TRestore to z. The latent diffusion model works on the latent space, and the size of the feature is much smaller than that of the image space, so the inference speed of the latent diffusion model is much faster. And the latent space of the latent diffusion model is a meaningful visual codebook, which makes the diffusion model part of the latent diffusion model focus more on generating the semantic information of the image rather than the texture details of the image.

[0081] In the forward noise addition process, let T∈N be the total time step of the diffusion model. For a class z with K categories at time step t∈N, t ∈{1,2,…,K} scalar discrete variable, z t-1 Pass to z t The probability is obtained by using the transfer matrix Q t ∈[0, 1] K×K Definition, where [Q t ]=q(z t =m|z t-1 =n):

[0082] q[z t |z t-1 ]=v(z t ) T Q t v(z t-1 )]

[0083] where v(z t )∈{0,1} K Yes t A one-hot vector of . Given z t-1 Time t The categorical distribution on is represented by the column vector Q t v(z t-1 )∈[0,1] K Calculation. According to the Markov property, it can be deduced that from z0 to z t The probability distribution of

[0084]

[0085] In the reverse denoising process, the forward denoising process needs to be reversed. T Gradually restore the initial features z0, z t to z t-1 The joint distribution of is:

[0086]

[0087] Due to the Markov property, the future state of a random process depends only on the current state and has nothing to do with its past state, so:

[0088] q(zt |z t-1 ,z0)=q(z t |z t-1 )

[0089] In contrast to the forward denoising process, the reverse denoising process uses a neural network p θ (z t-1 |z t )∈[0,1] N×K Consider z t-1 In z t The conditional distribution on z. t-1 Sampling is done according to this distribution and parameterizing p through a bidirectional Transformer encoder block. θ (z t-1 |z t ).

[0090] The overall denoising goal is the variational lower bound goal Added to the auxiliary denoising target, the calculation formula is as follows:

[0091]

[0092] where λ is a hyperparameter that balances the two loss terms.

[0093] The conditional mechanism module introduces external constraints or information so that the generation process can meet specific constraints or requirements. By injecting external conditions into the generation process of the model, the model can guide the direction of the output in the generation phase according to these conditions, thereby avoiding the generation of results that do not meet the constraints. The conditional encoder encodes the precondition into a feature vector τ θ , in the initial state z T and inject conditional information into the sampling information, and then pass the reverse denoising process p θ (z t-1 |z t ,y) to learn the conditional distribution without modifying the forward noise addition process q(z 1:T |z0). The process can be expressed as:

[0094]

[0095] The encoder of the pre-trained graph convolutional network model is used to map the graph structure data of the element information (including category and constraint information) to the vector space, and the cross-attention mechanism is used to integrate the conditions into the middle layer of the U-Net. Finally, the loss function of the potential diffusion model with conditions is added for:

[0096]

[0097] Since step 3 introduces the latent diffusion model in user interface generation, the quality and diversity of the generated layout can be effectively improved. Through the gradual noise addition and denoising process, the latent diffusion model can capture the complex relationships and details between UI elements more meticulously, thereby generating a more accurate and interactive user interface. Compared with GAN and VAE, the diffusion model has the advantages of more stable training and more controllable generation results, reducing the phenomenon of mode collapse. In addition, the latent diffusion model can more effectively integrate multiple design constraints and aesthetic standards to ensure that the generated interface is not only beautiful but also functional.

[0098] In order to illustrate the difference between this model and the existing model, this embodiment further proposes a comparative evaluation of the effects in the actual application process, specifically using the following evaluation indicators:

[0099] FID (Fréchet Inception Distance) is used to measure the difference between the predicted layout and the true layout to evaluate the generated model. The evaluation formula is as follows:

[0100]

[0101] Where Tr represents the sum of the elements on the diagonal of the matrix, that is, the trace of the matrix. x represents the true layout vector, and g represents the generated layout vector. A lower FID means that the two distributions are closer, which means that the quality of the generated layout is higher and the diversity is better.

[0102] Aesthetic standards use alignment as an important criterion to measure whether the generated layout meets the design specifications. In most good designs, elements are either centered or aligned to the edges:

[0103]

[0104] Where N D Indicates the number of generated layouts, Represents the kth element in the dth layout. l, m and r represent the distances between the left, middle and right lines of the elements.

[0105] IoU is used to calculate the overlap rate between the predicted border and the real border. The calculation formula is as follows:

[0106]

[0107] Where So represents the overlapped area of ​​the predicted interface component layout and the real component, and Su represents the combined area of ​​the predicted interface component layout and the real component. The higher the IoU value, the higher the overlap rate of the two component borders, and the more accurate the generated layout.

[0108] This embodiment uses the LayoutVAE model and the LayoutGAN model as benchmarks to evaluate the performance of the user interface layout generation method based on the latent diffusion model proposed in this embodiment in the layout generation task. The specific results are shown in Table 1.

[0109] Table 1

[0110] Model FID↓ Aesthetic standards↓ IoU↑ LayoutVAE 6.725 0.743 0.334 LayoutGAN 6.576 0.779 0.374 This embodiment 6.244 0.736 0.387

[0111] The experimental data used in this embodiment are as follows:

[0112] Magazine dataset, contains 4k+ layout images of magazine pages, with 6 component categories.

[0113] The Rico dataset is a dataset of mobile application user interface design, which contains 90k+ screenshots of Android mobile applications and 25 element categories, such as text buttons, toolbars, and icons.

[0114] The Ali-UI dataset contains a total of 1,613 product advertisement layout datasets in e-commerce scenarios, covering 8 component categories.

[0115] As shown in Table 1, the user interface layout generation method based on the latent diffusion model proposed in this embodiment achieved the best results in the layout generation task in terms of FID and IoU indicators, thereby verifying that the method shows a smaller difference between the predicted layout and the actual layout; it also obtained the highest alignment score in the evaluation of aesthetic quality.

[0116] The present invention supports users to input interface description text to automatically generate user interfaces. When generating a layout, the trained model can automatically control the alignment, arrangement and relative size of components without the need for manual definition by the user, thereby lowering the threshold for interface design. In addition, the invention also supports users to customize the input of interface description text to control the generation of interface layout and achieve personalized customization. By building multiple models, users only need to input interface descriptions according to their own needs, and the model can automatically generate a user interface layout diagram, achieving a high degree of automation.

[0117] The present invention extracts components and constraints from the interface requirement description text based on the BERT model, uses a graph convolutional network module to predict complete constraint relationships and borders to generate a complete graph, and generates a user interface layout based on a potential diffusion model. On the one hand, the introduction of the potential diffusion model in user interface generation can capture the complex relationships and details between UI elements in detail, thereby generating a more accurate user interface that complies with interaction specifications. On the other hand, the introduction of a graph convolutional network in user interface generation can effectively predict the complete constraint relationships between components, and effectively capture and model the complex dependencies and interactions between user interface elements; through the graph structure, multi-dimensional constraints such as position, alignment, and function between elements are imposed, generating a more coordinated layout that complies with design specifications; improving the aesthetics and practicality of the generated interface, and enhancing the user experience.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for generating user interface layout based on a potential diffusion model, characterized in that: The following steps are involved: S1. Extract components and constraints from the interface requirement description text based on the Bidirectional Encoder Representation from Transformers (BERT) model. Specifically, first organize the original interface requirement text data and divide it into text to be processed; then use the BERT model to build a text classification model to obtain the user interface component types and some constraint relationships between user interface components from the text to be processed; S2. Use the Graph Convolutional Networks (GCN) module to predict the complete constraint relationship and bounding box to generate a complete graph. Specifically, the user interface component type and partial constraint relationship between user interface components obtained from S1 are used as input, and processed by the GCN module to obtain the complete constraint relationship between each user interface component. S3. Generate a user interface layout based on the latent diffusion model. Specifically, the complete constraint relationship between the user interface components obtained from S2 and the original interface image are used as input, and the perceptual image compression, forward denoising and reverse denoising processes of the latent diffusion model are processed to obtain a user interface layout diagram.

2. The method for generating a user interface layout based on a potential diffusion model according to claim 1, characterized in that: The perceptual image compression process is used to realize the mutual conversion between image and feature space, specifically, the encoder E of Vector Quantised-Variational AutoEncoder (VQ-VAE) is used to compress the image x into a feature vector z; the decoder D of VQ-VAE is used to restore the feature vector z into an image with interface layout 3. The method for generating a user interface layout based on a potential diffusion model according to claim 2, characterized in that: The forward denoising process is used to denoise the normally distributed variables to learn the data distribution p(x), and the feature vector z is denoised as the latent variable z at time T in the diffusion process. T The reverse denoising process is used to convert z T Reducing to z, the core structure of the denoising process is a U-Net network composed of cross attention.

4. The method for generating a user interface layout based on a potential diffusion model according to claim 3, characterized in that: The forward noise addition process is implemented by the following method: Let T∈N be the total time step of the diffusion model. In the forward noise addition process, for a class z with K at time step t∈N, t ∈{1, 2, ..., K} scalar discrete variable, z t-1 Pass to z t The probability is obtained by using the transfer matrix Q t ∈[0, 1] K×K Definition, then: q[zt|z t-1 ]=v(z t )TQtv(from t-1 )] Among them, [Q t ]=q(z t =m|z t-1 =n), v(z t )∈{0,1} K Represents z t The one-hot vector of t-1 Time t The categorical distribution on is represented by the column vector Q t v(z t-1 )∈[0,1] K Calculate and deduce from z0 to z according to the Markov property t The probability distribution of is: in, 5. The method for generating a user interface layout based on a potential diffusion model according to claim 4, characterized in that: The reverse denoising process is implemented by the following method: Reverse the forward noise addition process and obtain the feature vector z at time T during the diffusion process T Gradually restore the initial features z0, z t to z t-1 The joint distribution of is: Due to the Markov property, the formula q(z t |z t-1 , z0)=q(z t |z t-1 ); Through the neural network θ (z t-1 |z t )∈[0,1] N×K Consider z t-1 In z t The conditional distribution on z t-1 Sampling is done according to this distribution and parameterizing p through a bidirectional Transformer encoder block. θ (z t-1 |z t ); Overall goal of denoising The calculation formula is as follows: Among them, λ is a hyperparameter that balances the two loss terms, is the variational lower bound objective, To assist in denoising.

6. The method for generating a user interface layout based on a potential diffusion model according to claim 5, characterized in that: The reverse denoising process also includes a conditional constraint mechanism, which is implemented by the following method: Encode the precondition into a feature vector τ θ , in the initial state z T and inject conditional information into the sampling information, and then pass the reverse denoising process p θ (z t-1 |z t , y) learning conditional distribution, the process is expressed as: Among them, the forward noise process q(z 1：T |z0) remains unchanged, y is the true label vector of the sentence; The encoder of the GCN model is used to map the types of user interface components and the constraint relationships between user interface components into the vector space, and the cross-attention mechanism is used to integrate the conditions into the middle layer of the U-Net, and finally the loss function of the latent diffusion model with added conditions is obtained. for:

7. The method for generating a user interface layout based on a potential diffusion model according to claim 1, characterized in that: The S2 is implemented by the following method: The graph is constructed by user interface components E = {e1, e2, .., eM} and user interface component constraint relations C = {c1, c2, ..., cN}; when constructing links between element nodes and constraint nodes, the adjacency matrix A is used to describe all adjacency relations on the graph; the GCN layer is a two-layer structure, each layer is composed of multiple GCN units; the user interface components and component relations obtained from S1 are word-embedded to obtain the adjacency matrix A and the node feature matrix X, and the node feature update formula of each layer is: Among them, H (l+1) is the node feature matrix of the l+1th layer, which represents the features of each node in this layer after convolution; H (l) is the input of the l+1th layer, representing the node feature matrix of the lth layer, that is, the features of each node in this layer; H (0) is the input feature matrix X of the initial layer, each row of X is the initial feature of a node; is the normalized adjacency matrix, which represents the node connection relationship in the graph; I is the identity matrix, which means adding self-connection to the adjacency matrix; To add the adjacency matrix with self-connection, yes The degree matrix, W (l) is the weight matrix of the lth layer, and the activation function uses the ReLU function; The components and constraint relationship features output by the GCN layer are input into the relationship predictor to predict the complete constraint relationship; the edge constraint relationship is predicted using the multi-classification cross entropy loss function, specifically: Assume T uv Represents the true edge constraint type between node u and node v, loss function The calculation formula is as follows: Where k is the number of edge types, T uvk is the type label vector of edge (u, v), S uvk Represents the probability that the constraint relationship between node u and node v belongs to category k.

8. The method for generating a user interface layout based on a potential diffusion model according to claim 1, characterized in that: The method of constructing a text classification model using the BERT model is implemented using the following method: The BERT model includes an input layer, an encoding layer, an attention mechanism layer and a decision layer; the input layer converts the input text into a specific dimensional vector; the encoding layer processes the specific dimensional vector into a matrix h after text encoding; The attention mechanism layer uses h to generate a single-word level attention vector α, and uses the Sigmoid function to limit the value of each component of the vector α to between 0 and 1 according to its importance. The calculation formula is as follows: α=Sigmoid(h*W) Among them, W is the weight parameter; The decision layer consists of a fully connected layer and an activation function layer. First, multiply h by α to obtain the sentence feature vector h*. Then input h* into the fully connected layer and the activation function layer to obtain the probability vector of the current sentence belonging to each category. The category corresponding to the highest probability value in the area is the prediction result, and the calculation formula is as follows: Among them, W' is the network weight and b' is the bias.

9. The method for generating a user interface layout based on a potential diffusion model according to claim 8, characterized in that: The BERT model is also used to calculate the loss of the model; the classification loss is calculated using the cross entropy loss function, and the difference between the normalized feature vectors is calculated using the KL divergence (Kullback-Leibler Divergence) loss function. The total loss of the model is calculated as follows: in, is the cross entropy loss, which is used to measure the gap between the model's predicted probability distribution and the true label; KL (Kullback-Leibler) divergence loss is used to measure the similarity of two probability distributions; is the total loss of the model, and M represents the number of categories, y is the true label vector of the sentence, and is the normalized eigenvector.

Images