Method and system for automatic generation of front-end code based on design interface screenshots
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2023-07-04
- Publication Date
- 2026-06-02
Smart Images
Figure CN116841545B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and more specifically, to a method and system for automatically generating front-end code based on screenshots of the design interface. Background Technology
[0002] With the continuous development of computer languages, deep learning models have been widely applied to code generation. Automatic code generation can greatly benefit programmers, reduce labor costs for companies, and is also an important technical means for software intelligence.
[0003] The paper Beltramelli T. pix2code: Generating code from a graphical user interface screenshot [C] / / Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems. 2018: 1-6. discloses a method for generating code using a deep learning model. This deep learning model uses CNN and LSTM networks for feature extraction and decoding, respectively. However, this deep learning model is too rudimentary, and its code generation capability has room for improvement. Furthermore, the dataset used is too simple to meet the needs of generating code for some more complex design interfaces.
[0004] In existing technologies, imgcook generates front-end code by using sketch or psd files as input. However, the input interface design is complex, and the tool has a relatively high learning curve. Sketch2Code can generate code for some simple interfaces, but this project cannot modify the dataset according to different needs, resulting in poor scalability.
[0005] Therefore, there is an urgent need in the market for an automated front-end code generation method and system based on design interface screenshots that can directly generate descriptive text from corresponding descriptive languages while ensuring accuracy. Summary of the Invention
[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for automatically generating front-end code based on screenshots of the design interface.
[0007] A method for automatically generating front-end code based on a screenshot of a design interface, provided by the present invention, includes:
[0008] Step S1: Obtain a screenshot of the design interface;
[0009] Step S2: Construct a domain-specific language mapping table to describe the front-end design interface;
[0010] Step S3: Generate an image-description dataset based on the domain-specific language mapping table;
[0011] Step S4: Construct a deep learning model and train the deep learning model using the image-description dataset as input;
[0012] Step S5: Input the screenshot of the design interface into the trained deep learning model to generate the corresponding descriptive text.
[0013] Preferably, step S2 includes the following sub-steps:
[0014] Step S2.1: Determine the interface components included in the design interface based on the requirements document, and provide a preliminary description of the interface components;
[0015] Step S2.2: Construct the interface components described in the initial step using a front-end language;
[0016] Step S2.3: Adjust the image features corresponding to the interface components after the initial description;
[0017] Step S2.4: Establish the mapping relationship between the description text of each interface component in the domain-specific language and the corresponding front-end code to obtain the domain-specific language mapping table.
[0018] Preferably, step S3 includes the following sub-steps:
[0019] Step S3.1: Generate a descriptive text instance using the descriptive text generator;
[0020] Step S3.2: Based on the domain-specific language mapping table, compile the description text instance into the corresponding HTML code file using the description text compiler;
[0021] Step S3.3: Convert the HTML code file into a corresponding design interface screenshot to obtain data pairs of images and descriptive text;
[0022] Repeat steps S3.1 to S3.3 to generate the image-description dataset for this paper.
[0023] Preferably, the encoder in the deep learning model extracts image features from the design interface screenshot through a CNN network, extracts text features from the descriptive text through a Transformer network, decodes the image features and text features through a BiLSTM network, and finally outputs the predicted elements through a Softmax layer.
[0024] Preferably, step S5 includes the following sub-steps:
[0025] Step S5.1: Load the parameters corresponding to the trained deep learning model;
[0026] Step S5.2: Input the corresponding design interface screenshot and the description text file that currently only contains the start mark into the trained deep learning model for reasoning to obtain the corresponding elements;
[0027] Step S5.3: Determine whether the output of the current deep learning model is an end marker or whether the number of accumulated output elements exceeds the threshold. If yes, the inference ends and the description text file corresponding to the final design interface screenshot is obtained; otherwise, proceed to step S5.4.
[0028] Step S5.4: Input the current element into the description text file that currently only contains the start marker, to obtain the updated description text file that only contains the start marker, and return to step S5.2.
[0029] According to the present invention, a front-end code automated generation system based on design interface screenshots includes:
[0030] Module M1: Get screenshots of the design interface;
[0031] Module M2: Constructs a domain-specific language mapping table to describe the front-end design interface;
[0032] Module M3: Generates an image-description dataset based on the domain-specific language mapping table;
[0033] Module M4: Constructs a deep learning model and trains it using the image-description dataset as input;
[0034] Module M5: Input the screenshot of the design interface into the trained deep learning model to generate the corresponding descriptive text.
[0035] Preferably, module M2 includes the following sub-modules:
[0036] Module M2.1: Determine the interface components included in the design interface based on the requirements document, and provide a preliminary description of the interface components;
[0037] Module M2.2: Constructs the initial UI components using a front-end language;
[0038] Module M2.3: Adjust the image features corresponding to the interface components described in the preliminary description;
[0039] Module M2.4: Establishes a mapping relationship between the description text of each interface component in the domain-specific language and the corresponding front-end code, and obtains a domain-specific language mapping table.
[0040] Preferably, module M3 includes the following sub-modules:
[0041] Module M3.1: Generates descriptive text instances using a descriptive text generator;
[0042] Module M3.2: Based on the domain-specific language mapping table, the description text instance is compiled into the corresponding HTML code file by the description text compiler;
[0043] Module M3.3: Converts the HTML code file into a corresponding design interface screenshot, obtaining data pairs of images and descriptive text;
[0044] Repeatedly trigger modules M3.1 through M3.3 to generate the image-description dataset for this paper.
[0045] Preferably, the encoder in the deep learning model extracts image features from the design interface screenshot through a CNN network, extracts text features from the descriptive text through a Transformer network, decodes the image features and text features through a BiLSTM network, and finally outputs the predicted elements through a Softmax layer.
[0046] Preferably, module M5 includes the following sub-modules:
[0047] Module M5.1: Loads the parameters corresponding to the trained deep learning model;
[0048] Module M5.2: Input the corresponding design interface screenshot and the description text file containing only the start marker into the trained deep learning model for inference to obtain the corresponding elements;
[0049] Module M5.3: Determines whether the output of the current deep learning model is an end marker or whether the number of accumulated output elements exceeds a threshold. If yes, the inference ends and the description text file corresponding to the final design interface screenshot is obtained; otherwise, module M5.4 is triggered.
[0050] Module M5.4: Input the current element into the description text file that currently only contains the start marker, obtain the updated description text file that only contains the start marker, and return to module M5.2.
[0051] Compared with the prior art, the present invention has the following beneficial effects:
[0052] 1. This invention solves the problems of insufficient and mismatched datasets in the task of generating code from screenshots of the design interface by designing a dataset generation algorithm and a domain-specific language design method, and can generate corresponding datasets according to different needs.
[0053] 2. This invention solves the problem of poor model training performance caused by poor component image features during dataset design and generation by utilizing the class-activated image method.
[0054] 3. This invention improves the model performance and increases the accuracy of generating descriptive text from design interface screenshots by using a Transformer-based text feature encoder and a BiLSTM decoder.
[0055] 4. This invention simplifies the difficulty of code implementation in software development and reduces the cost required for front-end developers by generating the corresponding description language from the design interface screenshot. Attached Figure Description
[0056] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0057] Figure 1 This is a schematic diagram of the deep learning model workflow in this invention.
[0058] Figure 2 This is a schematic diagram illustrating the process of constructing a domain-specific language in this invention.
[0059] Figure 3 This is a schematic diagram of the image-description text dataset generation process in this invention.
[0060] Figure 4 This is a schematic diagram of the model reasoning process in this invention. Detailed Implementation
[0061] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.
[0062] According to the present invention, a method for automatically generating front-end code based on screenshots of a design interface is provided, such as... Figures 1 to 4 As shown, it includes:
[0063] Step S1: Obtain a screenshot of the design interface.
[0064] Step S2: Construct a domain-specific language mapping table to describe the front-end design interface. Step S2 includes the following sub-steps:
[0065] Step S2.1: Determine the interface components included in the design interface based on the requirements document, and provide a preliminary description of the interface components. That is, use brief descriptive text to describe the interface components, such as search, text, switch, etc.
[0066] Step S2.2: Construct the initially described interface components using front-end languages. Front-end languages include HTML and CSS; that is, the designed components are roughly implemented using HTML and CSS.
[0067] Step S2.3: Adjust the image features corresponding to the interface components after the initial description. The adjustment method includes using class-activated images to adjust the image features of the implemented components. By using class-activated images, the degree of influence of each region of the input image on the final model output can be determined. Based on the determined degree of influence, the corresponding components in the regions with a small influence on the final output are selected, that is, the image features of these components are poor. Then, the front-end code of the components with poor image features is adjusted, while other components with better image features are retained. In other words, by using class-activated images to retain some components with better image features and filtering out some poor components for front-end code adjustment, the effect of dataset optimization is achieved after each interface component obtains better image features, thereby improving the subsequent training effect of the model.
[0068] Step S2.4: Establish the mapping relationship between elements in the domain-specific language and the corresponding front-end code to obtain the domain-specific language mapping table.
[0069] Step S3: Generate an image-description text dataset based on the domain-specific language mapping table. The description text is a text file describing the corresponding interface in the domain-specific language. Step S3 includes the following sub-steps: Step S3.1: Generate description text instances using a description text generator. Step S3.2: Based on the domain-specific language mapping table, compile the description text instances into corresponding HTML code files using a description text compiler. Step S3.3: Convert the HTML code files into corresponding design interface screenshots and save them, obtaining image and description text data pairs. The conversion method includes simulating a browser. Repeat steps S3.1 to S3.3 to generate the image-description text dataset. This image-description text dataset generation method can generate corresponding datasets according to different needs, demonstrating its scalability.
[0070] Step S4: Construct a deep learning model and train it using the image-description text dataset as input. The encoder in the deep learning model extracts image features from the design interface screenshot using a CNN network, extracts text features from the description text using a Transformer network, decodes the image and text features using a BiLSTM network, and finally outputs the predicted elements using a Softmax layer. Using superior feature extraction networks and decoders such as Transformer and BiLSTM can effectively improve the predictive ability of the deep learning model. Compared to the original LSTM network, the Transformer can calculate the degree of mutual influence between elements in the sequence through a multi-head self-attention mechanism, thereby determining the weights of different elements in the sequence, which helps the network extract more effective features and ignore some invalid features. Compared to the original LSTM decoder, the BiLSTM network can handle not only the preceding and following information of long sequences, but also the following information, which can better help the model process the interface description text file data in this task.
[0071] Step S5: Input the screenshot of the design interface into the trained deep learning model to generate the corresponding descriptive text. Step S5 includes the following sub-steps:
[0072] Step S5.1: Load the parameters corresponding to the trained deep learning model.
[0073] Step S5.2: Input the corresponding design interface screenshot and the description text file that currently only contains the start marker into the trained deep learning model for reasoning to obtain the corresponding elements.
[0074] Step S5.3: Determine whether the output of the current deep learning model is an end marker or whether the number of accumulated output elements exceeds the threshold. If yes, the inference ends and the description text file corresponding to the final design interface screenshot is obtained; otherwise, proceed to step S5.4.
[0075] Step S5.4: Input the current element into the description text file that currently only contains the start marker, to obtain the updated description text file that only contains the start marker, and return to step S5.2.
[0076] This invention can generate corresponding image-description text datasets according to different needs, and use these datasets to train corresponding deep learning models. Finally, by using a simplified screenshot of the design interface as input for the project, the corresponding interface description text can be obtained. This description text can help front-end developers carry out subsequent development work without requiring very complex input, making it easy to use and training a deep learning model that better meets the needs.
[0077] Furthermore, in the application, some component description texts are first designed according to the requirements document of the software design interface. Then, the dataset generation method in this invention is used to generate an image-description text dataset. That is, the image-description text dataset is generated through steps S1 to S3. Then, the dataset is used to train the deep learning model constructed in step S4 of this invention. After training is completed, the corresponding deep learning model is obtained.
[0078] Before designing the final interface based on the requirements document, designers first create a simplified version. This simplified version contains only basic information such as the number, types, and layout of components, but not complex details like styles and designs. By using screenshots of this simplified version as model input, we can obtain corresponding descriptive text, effectively extracting the basic information contained in the final interface. Front-end developers can then use this descriptive text to assist in the subsequent development of the actual interface.
[0079] Example 2
[0080] The present invention also provides a front-end code automated generation system based on design interface screenshots. Those skilled in the art can implement the front-end code automated generation system based on design interface screenshots by executing the steps of the method. That is, the method for automated generation of front-end code based on design interface screenshots can be understood as a preferred embodiment of the front-end code automated generation system based on design interface screenshots.
[0081] According to the present invention, a front-end code automated generation system based on design interface screenshots includes:
[0082] Module M1: Get screenshots of the design interface.
[0083] Module M2: Constructs a domain-specific language (DNS) mapping table to describe the front-end design interface. Module M2 includes the following sub-modules: Module M2.1: Determines the interface components included in the design interface based on the requirements document and provides a preliminary description of the interface components. Module M2.2: Constructs the preliminarily described interface components using the front-end language. Module M2.3: Adjusts the image features corresponding to the preliminarily described interface components. Module M2.4: Establishes a mapping relationship between elements in the domain-specific language and their corresponding front-end code, obtaining the domain-specific language mapping table.
[0084] Module M3: Generates an image-description text dataset based on the domain-specific language mapping table. Module M3 includes the following sub-modules: Module M3.1: Generates descriptive text instances using a descriptive text generator. Module M3.2: Compiles the descriptive text instances into corresponding HTML code files using a descriptive text compiler based on the domain-specific language mapping table. Module M3.3: Converts the HTML code files into corresponding design interface screenshots, obtaining data pairs of images and descriptive text. Modules M3.1 to M3.3 are repeatedly triggered to generate the image-description text dataset.
[0085] Module M4: Constructs a deep learning model and trains it using the image-description dataset as input. The encoder in the deep learning model extracts image features from the design interface screenshot using a CNN network, extracts text features from the descriptive text using a Transformer network, decodes the image and text features using a BiLSTM network, and finally outputs the predicted elements using a Softmax layer.
[0086] Module M5: Inputs the screenshot of the design interface into the trained deep learning model to generate the corresponding descriptive text. Module M5 includes the following sub-modules: Module M5.1: Loads the parameters corresponding to the trained deep learning model. Module M5.2: Inputs the corresponding screenshot of the design interface and the current descriptive text file containing only the start marker into the trained deep learning model for inference to obtain the corresponding elements. Module M5.3: Determines whether the current output result of the deep learning model is an end marker or whether the number of accumulated output elements exceeds a threshold. If yes, the inference ends and the descriptive text file corresponding to the final design interface screenshot is obtained; otherwise, module M5.4 is triggered. Module M5.4: Inputs the current element into the current descriptive text file containing only the start marker to obtain an updated descriptive text file containing only the start marker, and returns to module M5.2.
[0087] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.
[0088] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. A method for automatically generating front-end code based on screenshots of a design interface, characterized in that, include: Step S1: Obtain a screenshot of the design interface; Step S2: Construct a domain-specific language mapping table to describe the front-end design interface; Step S3: Generate an image-description dataset based on the domain-specific language mapping table; Step S4: Construct a deep learning model and train the deep learning model using the image-description dataset as input; Step S5: Input the screenshot of the design interface into the trained deep learning model to generate the corresponding descriptive text; Step S2 includes the following sub-steps: Step S2.1: Determine the interface components included in the design interface based on the requirements document, and provide a preliminary description of the interface components; Step S2.2: Construct the interface components described in the initial step using a front-end language; Step S2.3: Adjust the image features corresponding to the interface components after the initial description; Step S2.4: Establish the mapping relationship between the description text of each interface component in the domain-specific language and the corresponding front-end code to obtain the domain-specific language mapping table; Step S3 includes the following sub-steps: Step S3.1: Generate a descriptive text instance using the descriptive text generator; Step S3.2: Based on the domain-specific language mapping table, compile the description text instance into the corresponding HTML code file using the description text compiler; Step S3.3: Convert the HTML code file into a corresponding design interface screenshot to obtain data pairs of images and descriptive text; Repeat steps S3.1 to S3.3 to generate the image-description dataset for this paper; The encoder in the deep learning model extracts image features from the screenshot of the design interface through a CNN network, extracts text features from the descriptive text through a Transformer network, decodes the image features and text features through a BiLSTM network, and finally outputs the predicted elements through a Softmax layer.
2. The method for automatically generating front-end code based on design interface screenshots according to claim 1, characterized in that, Step S5 includes the following sub-steps: Step S5.1: Load the parameters corresponding to the trained deep learning model; Step S5.2: Input the corresponding design interface screenshot and the description text file that currently only contains the start mark into the trained deep learning model for reasoning to obtain the corresponding elements; Step S5.3: Determine whether the output of the current deep learning model is an end marker or whether the number of accumulated output elements exceeds the threshold. If yes, the inference ends and the description text file corresponding to the final design interface screenshot is obtained; otherwise, proceed to step S5.
4. Step S5.4: Input the current element into the description text file that currently only contains the start marker, to obtain the updated description text file that only contains the start marker, and return to step S5.
2.
3. A front-end code automated generation system based on design interface screenshots, characterized in that, The front-end code automated generation method based on design interface screenshots as described in any one of claims 1-2, wherein the generation system comprises: Module M1: Get screenshots of the design interface; Module M2: Constructs a domain-specific language mapping table to describe the front-end design interface; Module M3: Generates an image-description dataset based on the domain-specific language mapping table; Module M4: Constructs a deep learning model and trains it using the image-description dataset as input; Module M5: Input the screenshot of the design interface into the trained deep learning model to generate the corresponding descriptive text.
4. The front-end code automated generation system based on design interface screenshots according to claim 3, characterized in that, Module M2 includes the following sub-modules: Module M2.1: Determine the interface components included in the design interface based on the requirements document, and provide a preliminary description of the interface components; Module M2.2: Constructs the initial UI components using a front-end language; Module M2.3: Adjust the image features corresponding to the interface components described in the preliminary description; Module M2.4: Establishes a mapping relationship between the description text of each interface component in the domain-specific language and the corresponding front-end code, and obtains a domain-specific language mapping table.
5. The front-end code automated generation system based on design interface screenshots according to claim 3, characterized in that, Module M3 includes the following sub-modules: Module M3.1: Generates descriptive text instances using a descriptive text generator; Module M3.2: Based on the domain-specific language mapping table, the description text instance is compiled into the corresponding HTML code file by the description text compiler; Module M3.3: Converts the HTML code file into a corresponding design interface screenshot, obtaining data pairs of images and descriptive text; Repeatedly trigger modules M3.1 through M3.3 to generate the image-description dataset for this paper.
6. The front-end code automated generation system based on design interface screenshots according to claim 3, characterized in that, The encoder in the deep learning model extracts image features from the screenshot of the design interface through a CNN network, extracts text features from the descriptive text through a Transformer network, decodes the image features and text features through a BiLSTM network, and finally outputs the predicted elements through a Softmax layer.
7. The front-end code automated generation system based on design interface screenshots according to claim 3, characterized in that, Module M5 includes the following sub-modules: Module M5.1: Loads the parameters corresponding to the trained deep learning model; Module M5.2: Input the corresponding design interface screenshot and the description text file containing only the start marker into the trained deep learning model for inference to obtain the corresponding elements; Module M5.3: Determines whether the output of the current deep learning model is an end marker or whether the number of accumulated output elements exceeds a threshold. If yes, the inference ends and the description text file corresponding to the final design interface screenshot is obtained; otherwise, module M5.4 is triggered. Module M5.4: Input the current element into the description text file that currently only contains the start marker, obtain the updated description text file that only contains the start marker, and return to module M5.2.