User interface generation method and system based on neural network, and medium

Abstract

The invention relates to the technical field of graphical user interfaces, in particular to a user interface generation method and system based on a neural network and a medium. The method comprises the steps of obtaining a large number of sample UI design drafts; preprocessing the sample UI design draft, and taking the preprocessed data as a training set; constructing a user interface generation model based on a convolutional neural network framework and a generative adversarial network framework, wherein the user interface generation model comprises a visual feature extraction layer, a full connection layer, a feature recombination layer and a decision output layer which are connected in sequence; training the user interface generation model by adopting the training set to obtain a trained user interface generation model; and preprocessing the original UI design draft, inputting the trained user interface generation model, and outputting a new UI design draft, so that the visual effect of the user interface design draft and the use experience of a user are improved, and the output efficiency of the user interface design draft is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of graphical user interfaces, and in particular to a method, system and medium for generating a user interface based on a neural network. Background Art

[0002] With the popularization of digital terminals in the information age, user interface (UI) design has gradually attracted the attention of the public. As an interface presented to users through digital terminal displays, the design of the user interface directly affects the user experience and is an important way of presenting information. With the rapid development of mobile Internet and cloud computing technologies, the UI design industry has shown a booming development trend. Companies are increasingly aware of the importance of excellent UI design to product competitiveness, so the demand for UI designers is increasing.

[0003] Currently, designers usually manually design layers based on existing design rules using design software such as Photoshop, Figma, and Blue Lake. The design needs to be completed within a certain period of time and provided to software developers. However, due to the short design time and the lack of creativity in the large number of repetitive designs, the designed UI drafts are relatively monotonous, which is difficult to meet the user's visual and usage experience, and the design efficiency is low, which prolongs the software project development cycle and blocks the progress of software project development. Summary of the invention

[0004] In response to the above technical problems, the present invention proposes a user interface generation method, system and medium based on neural network, aiming to enhance the visual effect of user interface design drafts and the user experience, and greatly improve the output efficiency of user interface design drafts.

[0005] In a first aspect, the present application provides a method for generating a user interface based on a neural network, characterized in that the method comprises the following steps:

[0006] Step 102, obtaining a large number of sample UI design drafts;

[0007] Step 104, preprocessing the sample UI design draft, and using the preprocessed data as a training set;

[0008] Step 106, constructing a user interface generation model based on a convolutional neural network framework and a generative adversarial network framework, wherein the user interface generation model includes a visual feature extraction layer, a fully connected layer, a feature recombination layer, and a decision output layer connected in sequence;

[0009] Step 108, using the training set to train the user interface generation model to obtain a trained user interface generation model;

[0010] Step 110, pre-process the original UI design draft and input it into the trained user interface generation model to output a new UI design draft.

[0011] In some embodiments, in step 104, the sample UI design draft is preprocessed and the preprocessed data is used as a training set, including:

[0012] Step 202, converting the sample UI design draft into image data, wherein the image data includes static images and dynamic images;

[0013] Step 204, digitizing the image data to obtain a pixel matrix corresponding to the image data;

[0014] Step 206: Use the pixel matrix corresponding to all the image data as a training set.

[0015] In some embodiments, in step 204, the image data is digitized to obtain a pixel matrix corresponding to the image data, including:

[0016] When the image data is a static image, each pixel in the static image is converted into a corresponding pixel value, and a pixel matrix corresponding to the static image is obtained based on the pixel values ​​of all the pixels in the static image;

[0017] When the image data is a dynamic image, several frames in the dynamic image are extracted to obtain the frame sequence of the dynamic image, and then each pixel in each frame is converted into a corresponding pixel value. Based on the frame sequence number of the dynamic image and the pixel values ​​of all pixels in each frame, the pixel matrix corresponding to the dynamic image is obtained.

[0018] In some embodiments, in step 106, building a user interface generation model based on a convolutional neural network framework and a generative adversarial network framework includes:

[0019] Construct visual feature extraction layer and fully connected layer based on convolutional neural network framework;

[0020] The generator part based on the generative adversarial network framework constructs a feature recombination layer and a decision output layer, and the decision output layer is used to output a new UI design draft.

[0021] In some embodiments, in step 108, the user interface generation model is trained using the training set to obtain the trained user interface generation model, including:

[0022] Step 302, using the visual feature extraction layer and the fully connected layer to learn the visual features of each sample UI design draft in the training set;

[0023] Step 304, generating a design specification library based on the visual features of each sample UI design draft learned by the visual feature extraction layer and the fully connected layer;

[0024] Step 306, the feature reorganization layer reorganizes the visual features of the sample UI design draft based on the design specification library;

[0025] Step 308: The decision output layer generates a new UI design draft based on the reorganized visual features.

[0026] In some embodiments, in step 302, the visual features of each sample UI design draft in the training set are learned using the visual feature extraction layer and the fully connected layer, including:

[0027] The visual feature extraction layer is used to extract the component features, color features, font features, and icon features of each sample UI design draft in the training set;

[0028] Use the fully connected layer to learn the page layout features of each sample UI design draft;

[0029] The set of component features, color features, font features, icon features and page layout features of the sample UI design draft constitutes the visual features of the sample UI design draft.

[0030] In some embodiments, in step 306, the feature reorganization layer reorganizes the visual features of the sample UI design draft based on the design specification library, including:

[0031] The similarity measurement method is used to calculate the similarity between the visual features of the input sample UI design draft and the visual features in the design specification library;

[0032] Based on the similarity value, the visual features of the sample UI design draft are matched with the visual features in the design specification library;

[0033] Based on the matching results, the visual features of the sample UI design draft are reorganized.

[0034] In some embodiments, after step 308, the method further includes:

[0035] Forward propagate the sample UI design draft and the corresponding generated new UI design draft to the fully connected layer;

[0036] The content loss is calculated based on the feature graph of the sample UI design draft and the corresponding generated new UI design draft in the fully connected layer;

[0037] The Gram matrix algorithm is used to calculate the style loss of the sample UI design draft and the generated new UI design draft in the visual feature extraction layer and the fully connected layer;

[0038] Generate a total loss function based on content loss and style loss;

[0039] The total loss function is used to optimize the user interface generation model.

[0040] In a second aspect, the present application provides a user interface generation system based on a neural network, characterized by comprising:

[0041] Data acquisition module, used to obtain a large number of sample UI design drafts;

[0042] The preprocessing module is used to preprocess the sample UI design draft and use the preprocessed data as the training set;

[0043] A model building module, used to build a user interface generation model based on a convolutional neural network framework and a generative adversarial network framework, wherein the user interface generation model includes a visual feature extraction layer, a fully connected layer, a feature recombination layer, and a decision output layer connected in sequence;

[0044] A model training module, used to train the user interface generation model using a training set to obtain a trained user interface generation model;

[0045] The user interface generation module pre-processes the original UI design draft, inputs the trained user interface generation model, and outputs a new UI design draft.

[0046] In a third aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the aforementioned neural network-based user interface generation method.

[0047] The beneficial technical effects of the present invention include at least:

[0048] 1. Adopting the user interface generation method, system and medium based on neural network, creatively constructing the user interface generation model, the visual feature extraction layer deeply learns the components, colors, fonts, icons and other features of the sample UI design draft, which can more accurately capture the core visual elements of UI design, and combined with the learning of page layout features by the fully connected layer, help the user interface generation model understand the overall design structure of the user interface. At the same time, through the construction of the design specification library, a large number of common patterns in excellent UI designs are standardized to facilitate the learning and reproduction of the user interface generation model, so that the feature recombination layer can reorganize the visual features based on the design specification library, and use the similarity measurement method to accurately match the visual features of the sample UI design draft with the features in the design specification library. While ensuring that the generated UI design meets the design specifications, it generates diversified UI designs, improves the visual effect of the user interface design draft, and greatly improves the output efficiency of the user interface design draft;

[0049] 2. UI design drafts are usually static pictures. When it comes to individual components containing animation features, the present invention can retain the animation features of the components, such as motion trajectory, state change, etc., by converting the components containing animation features into dynamic picture data, and provides additional feature dimensions, so that the model can learn more complex features in the subsequent training process, such as the fluency and coherence of animation, which is crucial for the model to understand the interactivity and animation effects of UI design. Combining static picture data and dynamic picture data can build a more comprehensive and diverse training set, which helps to improve the generalization ability of the model and generate more realistic and interactive UI design drafts;

[0050] 3. By calculating the content loss of the feature maps of the sample UI design draft and the corresponding generated new UI design draft in the fully connected layer, the user interface generation model can retain the core content information of the original UI design draft (such as important content and page layout structure), ensuring that the information conveyed by the design is not lost. At the same time, the Gram matrix algorithm is used to calculate the style loss of the sample UI design draft and the generated new UI design draft in the visual feature extraction layer and the fully connected layer, ensuring that the new UI design draft inherits the style features of the sample design draft, such as color, texture, etc., and optimizes the user interface generation model based on the total loss function combining content loss and style loss. The user interface generation model can be trained to generate a UI design that not only retains the content information and style information of the sample UI design draft content image, but also incorporates the innovative visual features in the design specification library, ensuring that the UI design draft generated in the subsequent model application process is consistent with the original UI design draft in terms of content and style. At the same time, it can also combine new design elements to create a UI design draft that is both in line with the original UI design intent and innovative.

[0051] Other features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention will be further described below in conjunction with the accompanying drawings:

[0053] Figure 1 This is a flow chart of a method for generating a user interface based on a neural network according to an embodiment of the present invention.

[0054] Figure 2 This is an example diagram of an original UI design draft for inputting a user interface generation model into an embodiment of the present invention.

[0055] Figure 3 This is an example diagram of a new UI design draft output by the user interface generation model in an embodiment of the present invention.

[0056] Figure 4 The figure is a schematic diagram of the structure of a user interface generation system based on a neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The technical solutions of the embodiments of the present invention are explained and described below in conjunction with the drawings of the embodiments of the present invention, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.

[0058] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description, and do not indicate or imply that the referred device or element must have a specific direction, be constructed and operate in a specific direction. Therefore, it should not be understood as a limitation of the present invention.

[0059] Please refer to the attached Figure 1 , Figure 1 A flowchart of a method for generating a user interface based on a neural network provided in one embodiment of the present specification is shown.

[0060] like Figure 1 As shown, the neural network-based user interface generation method may at least include the following steps:

[0061] Step 102, obtaining a large number of sample UI design drafts.

[0062] Specifically, this embodiment obtains a large number of excellent UI design drafts designed by a large number of designers as samples. The sample UI design drafts cover a variety of styles, colors, page layouts and interactive elements, so as to ensure the diversity of samples.

[0063] Step 104, pre-processing the sample UI design draft, and using the pre-processed data as a training set.

[0064] Furthermore, in this embodiment, in step 104, the sample UI design draft is preprocessed, and the preprocessed data is used as a training set, including:

[0065] Step 202: convert the sample UI design draft into image data, where the image data includes static images and dynamic images.

[0066] Specifically, the implementation method of converting the sample UI design draft into image data is: based on the preset static component and animation component templates, the static components of the original design draft and the components containing animation features are identified from the sample UI design draft through a matching algorithm, and then the static components are converted into static image data in png format, and the components containing animation features are converted into dynamic image data in gif format. Among them, components include but are not limited to buttons, cards, dates, input boxes, drawers, form navigation bars, search boxes, drop-down menus, etc.

[0067] Step 204: digitize the image data to obtain a pixel matrix corresponding to the image data.

[0068] Optionally, before step 204, the image data may be normalized in size and pixel value to ensure the uniformity of subsequent training data and the stability of the model.

[0069] Furthermore, in this embodiment, in step 204, the image data is digitized to obtain a pixel matrix corresponding to the image data, including:

[0070] When the image data is a static image, each pixel in the static image is converted into a corresponding pixel value, and a pixel matrix corresponding to the static image is obtained based on the pixel values ​​of all pixels in the static image, that is, a three-dimensional array, in which each element corresponds to the height, width and pixel value of a pixel;

[0071] When the image data is a dynamic image, several frames in the dynamic image are extracted to obtain the frame sequence of the dynamic image, and then each pixel in each frame is converted into a corresponding pixel value. Based on the frame sequence number of the dynamic image and the pixel values ​​of all pixels in each frame, the pixel matrix corresponding to the dynamic image is obtained, that is, a four-dimensional array, in which each element corresponds to the frame sequence number, height, width and pixel value of a pixel.

[0072] Among them, for grayscale images, the pixel value represents the brightness of the pixel, which is between 0-255. For color images, the pixel value represents the color of the pixel, which is three values ​​0-255 (corresponding to RGB channels). You only need to determine the specific pixel value based on the actual situation of the UI design draft. This embodiment does not limit the meaning of the pixel value.

[0073] Step 206: Use the pixel matrix corresponding to all the image data as a training set.

[0074] It is understandable that UI design drafts are usually static images. When it comes to individual components containing animation features, this embodiment can retain the animation features of the components, such as motion trajectories, state changes, etc., by converting the components containing animation features into dynamic image data, and provides additional feature dimensions, so that the model can learn more complex features in the subsequent training process, such as the smoothness and coherence of the animation, which is crucial for the model to understand the interactivity and animation effects of the UI design. Combining static image data and dynamic image data can construct a more comprehensive and diverse training set, which helps to improve the generalization ability of the model and generate more realistic and interactive UI design drafts.

[0075] Step 106, constructing a user interface generation model based on a convolutional neural network framework and a generative adversarial network framework, wherein the user interface generation model includes a visual feature extraction layer, a fully connected layer, a feature recombination layer, and a decision output layer connected in sequence;

[0076] Specifically, in this embodiment, in step 106, a user interface generation model is constructed based on a convolutional neural network framework and a generative adversarial network framework, including:

[0077] Construct visual feature extraction layer and fully connected layer based on convolutional neural network framework;

[0078] The generator part based on the generative adversarial network framework constructs the feature reconstruction layer and the decision output layer, and the decision output layer is used to output the new UI design draft.

[0079] For example, the network framework of the user interface generation model in this embodiment may specifically include:

[0080] 1. Visual feature extraction layer (convolutional neural network part, which can use MS-CNN, VGG, etc.)

[0081] Input layer: Receives image data corresponding to UI design drafts in the training set.

[0082] Convolutional layer: Use multiple convolution kernels to extract local visual features of the image, such as image structure, edges, texture, color features, etc.

[0083] Activation layer: Apply the ReLU activation function to enhance the network's expressiveness and make the model more complex.

[0084] Pooling layer: Use maximum pooling or average pooling to reduce the spatial size of the feature map.

[0085] Normalization layer: applies batch normalization.

[0086] 2. Fully connected layer (convolutional neural network part)

[0087] Fully connected layer: One or more fully connected layers are used to further extract and learn higher-level visual features of UI design drafts.

[0088] Activation layer: ReLU activation function is applied after the fully connected layer.

[0089] 3. Feature Recombination Layer (GAN generator, CycleGAN etc.)

[0090] Latent Space Input: Receives a feature vector from a library of design specifications as input.

[0091] Fully connected layer: maps the input vector to a higher dimensional feature space.

[0092] Upsampling layer: Use deconvolution or transposed convolution layers to increase the spatial size of the feature map.

[0093] Skip connections: Skip connections are optionally added to preserve low-level features.

[0094] Normalization layer: Apply layer normalization (Layer Normalization) or instance normalization (InstanceNormalization).

[0095] Activation layer: applies ReLU or other activation function.

[0096] 4. Decision output layer (GAN generator output part)

[0097] Output convolution layer: Use the output convolution layer to generate the final new UI design draft.

[0098] Activation function: Apply sigmoid or tanh activation function to generate images with pixel values ​​in the range [0,255] or [-1,1].

[0099] Among them, the construction of the generative adversarial network framework includes:

[0100] 1. Generator: The feature recombination layer and the decision output layer together constitute the generator of GAN, which generates UI design drafts from random vectors or feature vectors in the design specification library.

[0101] 2. Discriminator: Another CNN network, used to distinguish between real UI designs and fake designs generated by the generator. The discriminator can specifically include:

[0102] Convolutional layer: Similar to the feature extraction layer, it is used to extract features;

[0103] Activation layer: Apply Leaky ReLU activation function;

[0104] Fully connected layer: outputs a single value representing whether the image is real or generated;

[0105] Activation function: Sigmoid activation function is applied after the last fully connected layer.

[0106] Step 108: Use the training set to train the user interface generation model to obtain a trained user interface generation model.

[0107] Furthermore, in this embodiment, in step 108, the user interface generation model is trained using the training set to obtain the trained user interface generation model, including:

[0108] Step 302: Use the visual feature extraction layer and the fully connected layer to learn the visual features of each sample UI design draft in the training set.

[0109] Specifically, in step 302, the visual features of each sample UI design draft in the training set are learned using the visual feature extraction layer and the fully connected layer, including:

[0110] The visual feature extraction layer is used to extract the component features, color features, font features, and icon features of each sample UI design draft in the training set;

[0111] Use the fully connected layer to learn the page layout features of each sample UI design draft;

[0112] The collection of component features, color features, font features, icon features and page layout features of the sample UI design draft constitutes the visual features of the sample UI design draft.

[0113] Step 304: Generate a design specification library based on the visual features of each sample UI design draft learned by the visual feature extraction layer and the fully connected layer.

[0114] Based on the visual features learned by the visual feature extraction layer and the fully connected layer, a design specification library is constructed, that is, a set of feature vectors representing common patterns in UI design. Specifically, the generation of the design specification library is achieved by first using encoding techniques (such as autoencoders, PCA, t-SNE or vector quantization) to convert the extracted visual features into a compact vector representation, then clustering the features, assigning one or more representative feature vectors to each category, and finally storing the encoded feature vectors and their corresponding category information in the design specification library.

[0115] Step 306: The feature reorganization layer reorganizes the visual features of the sample UI design draft based on the design specification library.

[0116] Furthermore, in this embodiment, in step 306, the feature reorganization layer reorganizes the visual features of the sample UI design draft based on the design specification library, including:

[0117] Step 402: Use a similarity measurement method to calculate the similarity between the visual features of the input sample UI design draft and the visual features in the design specification library.

[0118] Optionally, similarity calculation may be performed using a similarity measurement method such as Euclidean distance, cosine similarity, or Hamming distance, which is not limited in this embodiment.

[0119] Step 404 , based on the similarity value, matching the visual features of the sample UI design draft with the visual features in the design specification library.

[0120] Specifically, feature matching can be implemented by assigning the visual feature vector of the input sample UI design draft to the existing clusters in the design specification library, and then selecting the feature vector in each cluster in the design specification library that is most similar to the input visual feature.

[0121] Step 406: reorganize the visual features of the sample UI design draft based on the matching results.

[0122] Specifically, in this embodiment, a generator of a generative adversarial network in a user interface generation model architecture is used to perform feature reorganization based on visual features in a design specification library that match visual features of an input sample UI design draft, so as to output reorganized visual features.

[0123] Step 308: The decision output layer generates a new UI design draft based on the reorganized visual features.

[0124] This embodiment creatively constructs a user interface generation model, and the visual feature extraction layer deeply learns the components, colors, fonts, icons and other features of the sample UI design draft, which can more accurately capture the core visual elements of the UI design. Combined with the learning of the page layout features by the fully connected layer, the user interface generation model is helped to understand the overall design structure of the user interface. At the same time, through the construction of a design specification library, a large number of common patterns in excellent UI designs are standardized to facilitate the learning and reproduction of the user interface generation model, so that the feature recombination layer can reorganize the visual features based on the design specification library, and use the similarity measurement method to accurately match the visual features of the sample UI design draft with the features in the design specification library. While ensuring that the generated UI design meets the design specifications, it generates diversified UI designs, improves the visual effect of the user interface design draft, and greatly improves the output efficiency of the user interface design draft.

[0125] On the other hand, in this embodiment, after step 308, the following steps are further included:

[0126] Step 310, forward propagating the sample UI design draft and the corresponding generated new UI design draft to the fully connected layer.

[0127] It can be understood that in this embodiment, the high-level content and structural information of the UI design draft are captured through the features of the fully connected layer. After forward propagation to the fully connected layer, the feature map of the sample UI design draft on the fully connected layer and the feature map of the generated new UI design draft are obtained.

[0128] Step 312, calculating the content loss based on the feature graph of the sample UI design draft and the corresponding generated new UI design draft at the fully connected layer.

[0129] Specifically, in this embodiment, the following formula is used to calculate the content loss:

[0130]

[0131] Among them, F ij represents the value of the feature map of the sample UI design draft at position (i, j) in the fully connected layer, P ij Represents the value of the feature map of the fully connected layer at position (i, j) of the corresponding generated new UI design.

[0132] Step 314, using the Gram matrix algorithm to calculate the style loss of the sample UI design draft and the generated new UI design draft in the visual feature extraction layer and the fully connected layer.

[0133] The Gram Matrix is ​​a correlation matrix between each channel of the feature map, which can be obtained by calculating the inner product of the feature map. Specifically, the following formula is used to calculate the style loss in this embodiment:

[0134]

[0135] Among them, the layers where the sample UI design draft and the generated new UI design draft are located include the visual feature extraction layer and the fully connected layer. l and N l is the dimension of the feature map of layer l, Represents the elements of the Gram matrix of the sample UI design draft at level l, Represents the element of the Gram matrix corresponding to the generated new UI design draft at layer l.

[0136] Step 316: Generate a total loss function based on the content loss and the style loss.

[0137] Optionally, a weighted combination of content loss and style loss can be used to obtain the total loss function, that is, the total loss function can be expressed as:

[0138] L total =αL content +βL style

[0139] Among them, α and β are preset weight coefficients used to adjust the importance of content loss and style loss terms.

[0140] Step 318: Use the total loss function to optimize the user interface generation model.

[0141] Specifically, the gradient descent method can be used to minimize the total loss, and the user interface generation model can be optimized by minimizing the total loss. The method of optimizing model parameters by minimizing the total loss in this embodiment is similar to the method of minimizing the loss to optimize model parameters in the prior art, and will not be repeated in this embodiment.

[0142] By calculating the content loss of the feature maps of the sample UI design draft and the corresponding generated new UI design draft in the fully connected layer, the user interface generation model can retain the core content information of the original UI design draft (such as important content and page layout structure), ensuring that the information conveyed by the design is not lost. At the same time, the Gram matrix algorithm is used to calculate the style loss of the sample UI design draft and the generated new UI design draft in the visual feature extraction layer and the fully connected layer, ensuring that the new UI design draft inherits the style features of the sample design draft, such as color, texture, etc., and optimizes the user interface generation model based on the total loss function combining content loss and style loss. The user interface generation model can be trained to generate a UI design that not only retains the content information and style information of the sample UI design draft content image, but also incorporates the innovative visual features in the design specification library, ensuring that the UI design draft generated in the subsequent model application process is consistent with the original UI design draft in content and style. At the same time, it can also combine new design elements to create a UI design draft that is both in line with the original UI design intention and innovative.

[0143] Step 110, pre-process the original UI design draft and input it into the trained user interface generation model to output a new UI design draft.

[0144] Among them, the method of pre-processing the original UI design draft in this embodiment can refer to steps 202 to 204, and this embodiment will not be repeated here.

[0145] The following example verifies the visual effect of a new UI design draft generated by the neural network-based user interface generation method proposed in this embodiment.

[0146] Please refer to the attached Figure 2 , Figure 2 This is an example diagram of an original UI design draft for inputting a user interface generation model according to an embodiment of the present invention. Figure 2 The original UI design draft is pre-processed and then input into the trained user interface generation model. The output new UI design draft is shown in the attached Figure 3 , Figure 3 This is an example diagram of a new UI design draft output by the user interface generation model in an embodiment of the present invention.

[0147] Understandably, for the protection of information privacy, Figure 2 and Figure 3 The specific content values ​​in the UI design draft have been mosaiced to a certain extent.

[0148] from Figure 2 and Figure 3It can be seen from the comparison that the new UI design draft generated by the neural network-based user interface generation method proposed in this embodiment can be consistent with the original UI design draft in content and style, and at the same time combines the new design elements in the design specification library to create a UI design draft that is consistent with the original UI design intention and is innovative and has more stunning visual effects.

[0149] Please see attached Figure 4 , Figure 4 A schematic diagram of the structure of a user interface generation system based on a neural network provided in accordance with an embodiment of the present specification.

[0150] like Figure 4 As shown, the short-term photovoltaic power prediction system based on the similar day algorithm may at least include: a data acquisition module 1, a preprocessing module 2, a model building module 3, a model training module 4 and a user interface generation module 5, wherein:

[0151] Data acquisition module 1, used to obtain a large number of sample UI design drafts;

[0152] Preprocessing module 2 is used to preprocess the sample UI design draft and use the preprocessed data as a training set;

[0153] A model building module 3 is used to build a user interface generation model based on a convolutional neural network framework and a generative adversarial network framework, wherein the user interface generation model includes a visual feature extraction layer, a fully connected layer, a feature recombination layer, and a decision output layer connected in sequence;

[0154] Model training module 4, used to train the user interface generation model using the training set to obtain a trained user interface generation model;

[0155] The user interface generation module 5 pre-processes the original UI design draft, inputs the trained user interface generation model, and outputs a new UI design draft.

[0156] It is understandable that the technical concept of the user interface generation system based on a neural network provided in this embodiment is similar to the technical concept of the user interface generation method based on a neural network described above, and this embodiment will not be repeated here.

[0157] Another embodiment of the present specification provides a computer-readable storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps in the above-mentioned neural network-based user interface generation method embodiment. If the component modules of the above-mentioned electronic device are implemented in the form of software functional units and predicted or used as independent downstream tasks, they can be stored in a computer-readable storage medium.

[0158] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of this specification is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. Available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, digital versatile discs (DVD)), or semiconductor media (eg, solid state disks (SSD)).

[0159] The above descriptions are only preferred embodiments disclosed in this application and descriptions of the technical principles used. Those skilled in the art should understand that the scope of protection involved in this disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this disclosure (but not limited to) to form a technical solution.

[0160] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

Claims

1. A method for generating a user interface based on a neural network, characterized in that: The following steps are involved: Step 102, obtaining a large number of sample UI design drafts; Step 104, preprocessing the sample UI design draft, and using the preprocessed data as a training set; Step 106, constructing a user interface generation model based on a convolutional neural network framework and a generative adversarial network framework, wherein the user interface generation model includes a visual feature extraction layer, a fully connected layer, a feature recombination layer, and a decision output layer connected in sequence; Step 108, using the training set to train the user interface generation model to obtain a trained user interface generation model; Step 110, pre-process the original UI design draft and input it into the trained user interface generation model to output a new UI design draft.

2. The method for generating a user interface based on a neural network as claimed in claim 1, characterized in that: In step 104, the sample UI design draft is preprocessed, and the preprocessed data is used as a training set, including: Step 202, converting the sample UI design draft into image data, wherein the image data includes static images and dynamic images; Step 204, digitizing the image data to obtain a pixel matrix corresponding to the image data; Step 206: Use the pixel matrix corresponding to all the image data as a training set.

3. The method for generating a user interface based on a neural network as claimed in claim 2, characterized in that: In step 204, the image data is digitized to obtain a pixel matrix corresponding to the image data, including: When the image data is a static image, each pixel in the static image is converted into a corresponding pixel value, and a pixel matrix corresponding to the static image is obtained based on the pixel values ​​of all the pixels in the static image; When the image data is a dynamic image, several frames in the dynamic image are extracted to obtain the frame sequence of the dynamic image, and then each pixel in each frame is converted into a corresponding pixel value. Based on the frame sequence number of the dynamic image and the pixel values ​​of all pixels in each frame, the pixel matrix corresponding to the dynamic image is obtained.

4. The method for generating a user interface based on a neural network as claimed in claim 1, characterized in that: In step 106, a user interface generation model is constructed based on a convolutional neural network framework and a generative adversarial network framework, including: Construct visual feature extraction layer and fully connected layer based on convolutional neural network framework; The generator part based on the generative adversarial network framework constructs a feature recombination layer and a decision output layer, and the decision output layer is used to output a new UI design draft.

5. The method for generating a user interface based on a neural network as claimed in claim 1, characterized in that: In step 108, the user interface generation model is trained using the training set to obtain a trained user interface generation model, including: Step 302, using the visual feature extraction layer and the fully connected layer to learn the visual features of each sample UI design draft in the training set; Step 304, generating a design specification library based on the visual features of each sample UI design draft learned by the visual feature extraction layer and the fully connected layer; Step 306, the feature reorganization layer reorganizes the visual features of the sample UI design draft based on the design specification library; Step 308: The decision output layer generates a new UI design draft based on the reorganized visual features.

6. The method for generating a user interface based on a neural network as claimed in claim 5, characterized in that: In step 302, the visual feature extraction layer and the fully connected layer are used to learn the visual features of each sample UI design draft in the training set, including: The visual feature extraction layer is used to extract the component features, color features, font features, and icon features of each sample UI design draft in the training set; Use the fully connected layer to learn the page layout features of each sample UI design draft; The set of component features, color features, font features, icon features and page layout features of the sample UI design draft constitutes the visual features of the sample UI design draft.

7. The method for generating a user interface based on a neural network as claimed in claim 5, characterized in that: In step 306, the feature reorganization layer reorganizes the visual features of the sample UI design draft based on the design specification library, including: The similarity measurement method is used to calculate the similarity between the visual features of the input sample UI design draft and the visual features in the design specification library; Based on the similarity value, the visual features of the sample UI design draft are matched with the visual features in the design specification library; Based on the matching results, the visual features of the sample UI design draft are reorganized.

8. The method for generating a user interface based on a neural network as claimed in claim 5, characterized in that: After step 308, the method further includes: Forward propagate the sample UI design draft and the corresponding generated new UI design draft to the fully connected layer; The content loss is calculated based on the feature graph of the sample UI design draft and the corresponding generated new UI design draft in the fully connected layer; The Gram matrix algorithm is used to calculate the style loss of the sample UI design draft and the generated new UI design draft in the visual feature extraction layer and the fully connected layer; Generate a total loss function based on content loss and style loss; The total loss function is used to optimize the user interface generation model.

9. A user interface generation system based on a neural network, characterized in that: include: Data acquisition module, used to obtain a large number of sample UI design drafts; The preprocessing module is used to preprocess the sample UI design draft and use the preprocessed data as the training set; A model building module, used to build a user interface generation model based on a convolutional neural network framework and a generative adversarial network framework, wherein the user interface generation model includes a visual feature extraction layer, a fully connected layer, a feature recombination layer, and a decision output layer connected in sequence; A model training module, used to train the user interface generation model using a training set to obtain a trained user interface generation model; The user interface generation module pre-processes the original UI design draft, inputs the trained user interface generation model, and outputs a new UI design draft.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for generating a user interface based on a neural network as described in any one of claims 1 to 8 is implemented.

Images