Coarse-grained and fine-grained progressive front-end code generation system and method based on diffusion model and knowledge distillation

Abstract

The application provides a coarse and fine granularity progressive front-end code generation system and method based on a diffusion model and knowledge distillation, which comprises an interface skeleton diffusion generation module, an interface skeleton code distillation module and a front-end code progressive fusion module; the interface skeleton diffusion generation module is used for generating a conditional discrete diffusion model for a UI skeleton, understanding the layout of UI components in an input pixel graph and completing the extraction of the layout relationship of the components; the interface skeleton code distillation module is used for completing the lightweight implementation of layout code generation based on the knowledge distillation technology by learning the layout structure and layout code relationship labeled by a teacher model; and the front-end code progressive fusion module is used for establishing the relationship among the pixel graph coordinates, the layout structure and the code document object model by using the divide-and-conquer idea, and realizing the fusion of the final code through the component coordinates and the layout code index. The application improves the front-end low-code generation efficiency and quality.

Description

Technical Field

[0001] This invention relates to the field of front-end code generation technology, and in particular to a coarse-grained progressive front-end code generation system and method based on diffusion model and knowledge distillation. Background Technology

[0002] In the field of software engineering, front-end development, as a core component of building user interfaces, has long faced the challenges of high repetition and low efficiency. Traditional development models require engineers to manually write large amounts of structurally similar code (such as UI components, state management, and routing configurations), which not only consumes a significant amount of time but is also prone to errors due to human negligence. With the increasing complexity and number of web applications, this inefficient development approach has become a major factor restricting project iteration speed. Against this backdrop, front-end code generation technology has emerged. Its core objective is to automate the transformation of design specifications and business logic into executable front-end code, thereby significantly improving development efficiency and ensuring code quality.

[0003] In recent years, the emergence of multimodal large models has provided new possibilities for automating the process from interface to front-end code. These models combine the capabilities of computer vision and natural language processing, enabling a deep understanding of UI design drafts. By analyzing visual elements in design images, the model can accurately identify various UI components (such as buttons, input boxes, cards, etc.) and their layout structure, including relative positions, hierarchical relationships, and style features. Based on this, the model can transform this visual information into a code structure that conforms to front-end framework specifications, achieving end-to-end conversion from design drafts to runnable code. This technological breakthrough allows developers to simply provide design drafts, and the system can automatically generate a basic front-end code framework, significantly reducing the workload of manual coding.

[0004] Front-end code generation involves understanding user interface design diagrams, while using multimodal large models to solve front-end development automation problems involves the visual understanding and code generation capabilities of multimodal large models.

[0005] In front-end code generation tasks, visual understanding capabilities are primarily reflected in the comprehension of elements, layout relationships, and style features of UI images. While mainstream models (such as GPT-4 and the Claude series) have achieved significant progress in understanding natural images and the relationships between objects, they still face numerous challenges when processing UI user interface designs containing elements of various forms and complex nested layouts. These challenges include element omissions, errors in layout information extraction, and background image interference, making it difficult to accurately and effectively determine the nesting and parallel relationships between different components. Therefore, multimodal large models still face key challenges in understanding UI layouts.

[0006] In front-end code generation tasks, the ability to generate code primarily manifests in the accurate reproduction of extracted layout information. Currently, mainstream models, even with sufficient layout information provided by text, cannot accurately reproduce the original UI user interface design. Furthermore, current research mainly focuses on text-based code generation tasks, with less attention paid to visual modal code generation. Additionally, utilizing the model's inherent generation capabilities limits the output code length, making it impossible to generate long codes with complex layouts. Therefore, accurately reproducing UI layouts during code generation remains a key challenge for multimodal large-scale models. Summary of the Invention

[0007] The main objective of this invention is to provide a coarse-grained progressive front-end code generation system and method based on diffusion model and knowledge distillation, aiming to improve the efficiency and quality of front-end low-code generation.

[0008] To achieve the above objectives, this invention proposes a coarse-fine granular progressive front-end code generation system based on a diffusion model and knowledge distillation. The system includes an interface skeleton diffusion generation module, an interface skeleton code distillation module, and a front-end code progressive fusion module.

[0009] The interface skeleton diffusion generation module is used to generate a conditional discrete diffusion model for the UI skeleton, understand the layout of UI components in the input pixel image, and extract the layout relationship of the components.

[0010] The interface skeleton code distillation module is used to achieve a lightweight implementation of layout code generation by learning the layout structure and layout code relationships annotated by the teacher model based on knowledge distillation technology.

[0011] The progressive fusion module for the front-end code is used to establish the relationship between pixel coordinates, layout structure, and code document object model by using the divide-and-conquer approach. The model is used to generate code for fine-grained local blocks, and the final code is fused through component coordinates and layout code indexes.

[0012] A further technical solution of the present invention is characterized in that the interface skeleton diffusion generation module is further used for:

[0013] A UI-to-skeleton adapter module is added to learn the matching relationship between pixel images and UI skeletons. The UI-to-skeleton adapter module consists of three downsampling modules of different scales, which are used to capture components and layout details of different granularities in UI pixel images. The feature vectors of each layer are extracted through the residual network backbone, projected, and the feature information of each layer is injected into the corresponding layer of the UNet of the noise diffusion model through cross attention.

[0014] Conditional generation training process: Freeze the model parameters of unconditional skeleton generation, train only the interface to skeleton adapter part, and coordinate the ratio of unconditional generation and conditional generation through the guide.

[0015] Conditional generation inference process: Initialize a random skeleton noise, use a pre-trained noise predictor to predict the skeleton noise to be removed at each step by combining the feature information of the UI pixel map input, and remove the noise. After denoising for several consecutive time steps, the decoder finally parses a web page skeleton that conforms to the rules and corresponds to the UI pixel map from the noise vector.

[0016] A further technical solution of the present invention is that the interface skeleton diffusion generation module is also used to: generate an unconditional discrete diffusion model oriented towards the UI skeleton.

[0017] A further technical solution of the present invention is that the interface skeleton diffusion generation module is also used for:

[0018] Conditional training data generation design: Collect web page data, and then... The webpage is processed, rendered, and a screenshot is taken to obtain the image. The position and size of outer components are obtained through manual annotation. Each webpage contains a variable number of outer components. The first page The coordinates of each component are denoted as... A data pair consists of a screenshot of a webpage and the location set of the outer components of that webpage. A training data set consisting of all data pairs is denoted as data. ;

[0019] Unconditionally generated training data design: This involves generating training data sets... Each data pair Let's create a separate unconditional skeleton generation dataset, denoted as . ;

[0020] Diffusion process: Generating datasets from unconditional skeletons Coordinates of each component in the skeleton of each webpage Random noise is added after embedding. The latent vector representing the initial webpage skeleton is transformed into a completely random discrete noise vector by adding noise over T time steps. Correspondingly, a regular webpage skeleton will gradually break down over multiple time steps, becoming a random, unstructured skeleton. Each webpage skeleton generates a data sample containing a noisy skeleton across multiple time steps. The set of all samples is denoted as the dataset. ;

[0021] Training process: The data generated during the diffusion process is used to train the model parameters of the discrete skeleton noise predictor. The input to the discrete noise predictor is the time step. and the Noise skeleton at each time step The output label is Ultimately, a discrete skeleton noise predictor that can generate webpage skeletons according to rules is obtained; wherein, the entire training process randomly selects one time step. By learning the relationship between the changes in the noise skeleton at different time steps through a discrete diffusion model, we can learn how to generate a UI webpage skeleton that conforms to rules from random noise.

[0022] Inference process: Initialize a random skeleton noise, predict the skeleton noise to be removed at each step through the noise predictor, and remove the noise. After denoising for several consecutive time steps, finally use the decoder to parse a skeleton that conforms to the rules from the noise vector.

[0023] A further technical solution of the present invention is that the interface skeleton code distillation module is also used for:

[0024] Model initialization: A small-parameter open-source visual language model was selected as the student model, and a closed-source, high-quality commercial model was selected as the teacher model.

[0025] Multimodal distillation instruction design: The distillation instructions include image understanding instructions and code generation instructions. The image understanding instructions require the teacher model to describe the layout structure of the skeleton diagram of the input UI image using text. The code generation instructions require the teacher model to generate HTML code that restores the UI layout based on the skeleton diagram of the input UI image.

[0026] Automated pseudo-label annotation: Using the image understanding instructions, code generation instructions, and skeleton diagram, layout text descriptions and layout codes corresponding to the skeleton diagram are generated in the classroom model and used as training pseudo-labels;

[0027] Invalid sample filtering: Design an automated evaluation framework for UI scenarios to evaluate code execution correctness, image-code matching, and sample diversity, and remove sample data that does not meet the conditions;

[0028] Distillation data instruction fine-tuning: The ability to guide student models to learn from teacher model images to skeleton code through instruction fine-tuning.

[0029] A further technical solution of the present invention is that the progressive fusion module for front-end code is also used for:

[0030] Region association index: Based on the relationship between the region position in the skeleton diagram and the region's place space in the skeleton code, index the region pixel fragments in the original UI image, and recursively segment the pixel block fragments of each region.

[0031] Fine-grained code generation: Using fine-grained region code generation instructions, Claude or GPT series models convert region pixel patch fragments into region code fragments;

[0032] Coarse-grained progressive code fusion: Leveraging the decoupling characteristics between element nodes in the Document Object Model (DOM), fine-grained region code snippets are directly mounted onto the region container nodes of coarse-grained skeleton code based on region association indexes. Through top-down recursive DOM tree checks, it is ensured that the nodes are mounted correctly, without omissions, redundancies, or incorrect mounting. Ultimately, coarse-grained to fine-grained code fusion generation is completed.

[0033] To achieve the above objectives, this invention also proposes a coarse-to-fine granular progressive front-end code generation method based on a diffusion model and knowledge distillation, the method comprising the following steps:

[0034] Step S10: Generate a conditional discrete diffusion model for the UI skeleton, understand the layout of UI components in the input pixel image, and extract the layout relationship of the components.

[0035] Step S20: Based on knowledge distillation technology, a lightweight implementation of layout code generation is achieved by learning the layout structure and layout code relationships annotated by the teacher model.

[0036] Step S30: Using the divide-and-conquer approach, establish the relationship between pixel map coordinates, layout structure, and code document object model. The model is used to generate code for fine-grained local map tiles. The final code is then integrated using component coordinates and layout code indexes.

[0037] A further technical solution of the present invention is that step S10 includes:

[0038] Step S101: Add an interface to skeleton adapter module to learn the matching relationship between pixel map and UI skeleton; the interface to skeleton adapter module consists of three downsampling modules of different scales, which are used to capture components and layout details of different granularities in UI pixel map, respectively. The feature vectors of each layer are extracted through the residual network backbone, projected, and the feature information of each layer is injected into the corresponding layer of the UNet of the noise diffusion model through cross attention.

[0039] Step S102, Conditional generation training process: Freeze the model parameters of unconditional skeleton generation, train only the interface to skeleton adapter part, and coordinate the ratio of unconditional generation and conditional generation through the guide.

[0040] Step S103, conditional generation inference process: Initialize a random skeleton noise, predict the skeleton noise to be removed at each step by using a pre-trained noise predictor and combining the feature information of the UI pixel map input, and remove the noise. After denoising for several consecutive time steps, the decoder finally parses a web page skeleton that conforms to the rules and corresponds to the UI pixel map from the noise vector.

[0041] A further technical solution of the present invention is that step S10 further includes:

[0042] Conditional training data generation design: Collect web page data, and then... The webpage is processed, rendered, and a screenshot is taken to obtain the image. The position and size of outer components are obtained through manual annotation. Each webpage contains a variable number of outer components. The first page The coordinates of each component are denoted as... A data pair consists of a screenshot of a webpage and the location set of the outer components of that webpage. A training data set consisting of all data pairs is denoted as data. ;

[0043] Unconditionally generated training data design: This involves generating training data sets... Each data pair individually forms an unconditional skeleton generation dataset, denoted as . ;

[0044] Diffusion process: Generating datasets from unconditional skeletons Coordinates of each component in the skeleton of each webpage Random noise is added after embedding. The latent vector representing the initial webpage skeleton is transformed into a completely random discrete noise vector by adding noise over T time steps. Correspondingly, a regular webpage skeleton will gradually break down over multiple time steps, becoming a random, unstructured skeleton. Each webpage skeleton generates a data sample containing a noisy skeleton across multiple time steps. The set of all samples is denoted as the dataset. ;

[0045] Training process: The data generated during the diffusion process is used to train the model parameters of the discrete skeleton noise predictor. The input to the discrete noise predictor is the time step. and the Noise skeleton at each time step The output label is Ultimately, a discrete skeleton noise predictor that can generate webpage skeletons according to rules is obtained; wherein, the entire training process randomly selects one time step. By learning the relationship between the changes in the noise skeleton at different time steps through a discrete diffusion model, we can learn how to generate a UI webpage skeleton that conforms to rules from random noise.

[0046] Inference process: Initialize a random skeleton noise, predict the skeleton noise to be removed at each step through the noise predictor, and remove the noise. After denoising for several consecutive time steps, finally use the decoder to parse a skeleton that conforms to the rules from the noise vector.

[0047] A further technical solution of the present invention is that step S20 includes:

[0048] Step S201, Model initialization: Select an open-source visual language model with a small parameter scale as the student model, and select a closed-source, high-quality commercial model as the teacher model.

[0049] Step S202, Multimodal distillation instruction design: The distillation instruction includes an image understanding instruction and a code generation instruction. The image understanding instruction requires the teacher model to describe the layout structure of the skeleton diagram of the input UI image using text. The code generation instruction requires the teacher model to generate HTML code that restores the UI layout based on the skeleton diagram of the input UI image.

[0050] Step S203, Automated pseudo-label annotation: Using the image understanding instructions, code generation instructions, and skeleton diagram, the layout text description and layout code corresponding to the skeleton diagram are generated in the classroom model as training pseudo-labels;

[0051] Step S204, Invalid Sample Filtering: Design an automated evaluation framework for UI scenarios to evaluate code execution correctness, image-code matching, and sample diversity, and remove sample data that does not meet the conditions;

[0052] Step S205, Distillation Data Instruction Fine-tuning: Guide the student model to learn the teacher's model image to skeleton code through instruction fine-tuning;

[0053] Step S30 includes:

[0054] Step S301, Region Association Index: Based on the relationship between the region position in the skeleton diagram and the region space occupied in the skeleton code, index the region pixel fragments in the original UI image, and recursively segment the pixel block fragments of each region.

[0055] Step S302, Fine-grained code generation: Through fine-grained region code generation instructions, the region pixel patch fragments are converted into region code fragments by the Claude or GPT series models;

[0056] Step S303, coarse-grained progressive code fusion: Utilizing the decoupling characteristics between element nodes of the Document Object Model, fine-grained region code snippets are directly mounted onto the region container nodes of the coarse-grained skeleton code based on the region association index. Through top-down recursive document object model tree checks, it is ensured that the nodes are mounted correctly, without omissions, redundancies, or incorrect mounting. Finally, the coarse-grained to fine-grained code fusion generation is completed.

[0057] The beneficial effects of the coarse-grained and fine-grained progressive front-end code generation system and method based on diffusion model and knowledge distillation of this invention are:

[0058] This invention, through the aforementioned technical solution, generates a conditional discrete diffusion model oriented towards the UI skeleton, understands the layout of UI components in the input pixel image, and extracts the layout relationships of the components. Based on knowledge distillation technology, it learns the layout structure and layout code relationships annotated by the teacher's model to achieve a lightweight implementation of layout code generation. Utilizing the divide-and-conquer approach, it establishes the relationship between pixel image coordinates, layout structure, and code document object model. The model is used to generate code for fine-grained local blocks. Through component coordinates and layout code indexes, the final code is integrated, improving the efficiency and quality of front-end low-code generation. Attached Figure Description

[0059] Figure 1 This is a system framework diagram of a preferred embodiment of the coarse-grained progressive front-end code generation system based on diffusion model and knowledge distillation of the present invention;

[0060] Figure 2 This is a schematic diagram of the diffusion process;

[0061] Figure 3 This is a diagram illustrating the training process;

[0062] Figure 4 This is a schematic diagram of a conditional discrete diffusion model for UI skeleton.

[0063] Figure 5 This is a schematic diagram of automated pseudo-label labeling and fine-tuning of distillation data instructions;

[0064] Figure 6 This is a diagram illustrating the gradual integration of front-end code;

[0065] Figure 7 This is a flowchart illustrating a preferred embodiment of the coarse-grained progressive front-end code generation method based on diffusion model and knowledge distillation of the present invention.

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Detailed Implementation

[0067] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0068] This invention proposes a coarse-to-fine granular progressive front-end code generation system based on a diffusion model and knowledge distillation.

[0069] like Figure 1 As shown, a preferred embodiment of the coarse-to-fine granular progressive front-end code generation system based on diffusion model and knowledge distillation of the present invention includes three core modules: an interface skeleton diffusion generation module, an interface skeleton code distillation module, and a front-end code progressive fusion module.

[0070] The interface skeleton diffusion generation module is used to generate a conditional discrete diffusion model for the UI skeleton. It is responsible for understanding the layout of UI components in the input pixel image and extracting the layout relationship of the components, thus eliminating the gap between the pixel image and the structure.

[0071] The interface skeleton code distillation module is used to achieve lightweight implementation of layout code generation by learning the layout structure and layout code relationships marked by the teacher model based on knowledge distillation technology.

[0072] The progressive fusion module for the front-end code is used to establish the relationship between pixel coordinates, layout structure, and code document object model by using the divide-and-conquer approach. The model is used to generate code for fine-grained local blocks, and the final code is fused through component coordinates and layout code indexes.

[0073] In this embodiment, the UI skeleton generation is based on the discrete diffusion model.

[0074] The surface skeleton diffusion generation module includes a design unit for an unconditional discrete diffusion model oriented towards the UI skeleton and a design unit for a conditional discrete diffusion model oriented towards the UI skeleton.

[0075] The design unit of the UI skeleton-oriented unconditional discrete diffusion model is used for:

[0076] (1) Conditional training data generation design: Collect web page data, and then... The webpage is processed, rendered, and a screenshot is taken to obtain the image. The position and size of outer components are obtained through manual annotation. Each webpage contains a variable number of outer components. The first page The coordinates of a component can be denoted as: A data pair consists of a screenshot of a webpage and the location set of the outer components of that webpage. A training data set consisting of all data pairs is denoted as data. .

[0077] (2) Design of unconditionally generating training data: the training data set Each data pair Let's create a separate unconditional skeleton generation dataset, denoted as . .

[0078] (3) Noise addition process (diffusion process): such as Figure 2 For unconditional skeleton generation datasets Coordinates of each component in the skeleton of each webpage Random noise is added after embedding. The latent vector representing the initial webpage skeleton, after adding noise over T time steps, can eventually be transformed into a completely random discrete noise vector. Correspondingly, a regular webpage skeleton will gradually break down over multiple time steps, becoming a random, unstructured skeleton. Each webpage skeleton generates a data sample containing a noisy skeleton across multiple time steps. The set of all samples is denoted as the dataset. .

[0079] (4) Denoising process (training): The data generated during the noise addition process is used to train the model parameters of the discrete skeleton noise predictor. The input of the discrete noise predictor is the time step. and the Noise skeleton at each time step The output label is The final result is a discrete skeleton noise predictor that can generate webpage skeletons according to rules. The entire training process is as follows: Figure 3 As shown, a time step is randomly selected. By learning the relationship between noise skeletons at different time steps through a discrete diffusion model, we can learn how to generate a UI webpage skeleton that conforms to rules from random noise.

[0080] (5) Denoising process (inference): Initialize a random skeleton noise, predict the skeleton noise to be removed at each step using a noise predictor, and remove the noise. Through continuous... The model performs denoising at each time step, and finally uses a decoder to extract a rule-compliant skeleton from the noise vector. This process continues until the model learns how to generate a regular webpage skeleton.

[0081] The design unit of the conditional discrete diffusion model oriented towards the UI skeleton is used for:

[0082] (1) such as Figure 4 As shown, an additional UI-to-skeleton adapter module is added to learn the matching relationship between the pixel image and the UI skeleton. The UI-to-skeleton adapter module consists of three downsampling modules at different scales, which are used to capture components and layout details of different granularities in the UI pixel image. The feature vectors of each layer are extracted through the residual network backbone, projected, and the feature information of each layer is injected into the corresponding layer of the UNet of the noise diffusion model through cross attention.

[0083] (2) Conditional generation training process. Freeze the model parameters of unconditional skeleton generation, train only the interface to skeleton adapter part, and coordinate the ratio of unconditional generation and conditional generation through the guide.

[0084] (3) Conditional Generative Inference Process. Initialize a random skeleton noise. Using an unconditional skeleton noise predictor, combined with the feature information from the UI pixel map input, predict the skeleton noise to be removed at each step, and remove the noise. This process is repeated continuously. The noise is denoised at each time step, and finally, a decoder extracts a regular webpage skeleton that corresponds to the UI pixel image from the noise vector. Until the model learns how to generate a regular webpage skeleton based on the UI pixel image.

[0085] In this embodiment, the interface skeleton code distillation is based on the skeleton code generation of knowledge distillation.

[0086] The interface skeleton code distillation module is used for:

[0087] (1) Model initialization. Open-source visual language models with small parameter scales were selected as student models. Candidate models include Qwen2.5-VL-3B or 7B, deepseek-vl2-tiny, LLama3.2-Vision-11B, InternVL2.5-3B or 7B, etc. Closed-source, high-quality commercial models were selected as teacher models. Candidate models include Claude Sonnet 4, GPT-4.5, etc.

[0088] (2) Multimodal distillation instruction design. Distillation instructions are mainly divided into image understanding instructions and code generation instructions. Image understanding instructions require the teacher model to describe the layout structure of the skeleton diagram of the input UI image using text. Code generation instructions require the teacher model to generate HTML code that restores the UI layout based on the skeleton diagram of the input UI image.

[0089] (3) Automated pseudo-label annotation. This utilizes image understanding instructions, code generation instructions, and skeleton diagrams, such as... Figure 5In the teacher model, layout text descriptions and layout code corresponding to the skeleton diagram are generated and used as training pseudo-labels. To fully learn the diversity of the teacher model's output, different temperature coefficients are set to obtain the code generation and text description output results under different temperature values.

[0090] (4) Invalid Sample Filtering. An automated evaluation framework for UI scenarios is designed, mainly to evaluate code execution correctness, image-code matching, and sample diversity, and remove sample data that does not meet the conditions. Code execution correctness is verified by static code analysis tools to verify whether the code can run in the actual environment. Invalid or erroneous code will be removed. Image-code matching adopts a rendering-then-evaluation approach. The generated code is first rendered into a webpage and a screenshot is taken. The overall image similarity is compared using the CLIP index. The position and size of the region are extracted by object detection. Based on the position information and size of the detection box, the positional similarity and size similarity of the two images in the regional structure are compared. Diversity is measured by the distribution of the detection box center points. Specifically, the variance and entropy of the distribution of the detection box center points are judged. If the distribution of the detection box center points in the skeleton image is wide and the variance is large, it indicates that the spatial distribution of the skeleton image is more diverse.

[0091] (5) Fine-tuning of distillation data instructions. For example... Figure 5 This approach guides student models to learn the teacher model's image-to-skeleton code capability through fine-tuning of instructions. The core steps are as follows: Combining multimodal tasks, joint training—simultaneously training image understanding and code generation—reduces the task difficulty for student models in code generation. Underlying image understanding assists student models in better learning the teacher model's image-to-code capability. KL divergence is used to minimize the difference between the student model's output and the teacher model's output code. Based on different temperature coefficients, the student model learns the diverse functional characteristics of the teacher model's output.

[0092] In this embodiment, the progressive fusion of front-end code adopts a coarse-grained to fine-grained progressive code fusion framework.

[0093] The progressive fusion module for the front-end code is also used for:

[0094] (1) Region-related index: The progressive fusion module of the front-end code completes the coarse-grained skeleton code generation, such as Figure 6 As shown, based on the relationship between the region position in the skeleton diagram and the region space occupied in the skeleton code, the region pixel fragments in the original UI image are indexed out, and the pixel block fragments of each region are recursively segmented.

[0095] (2) Fine-grained code generation: Through fine-grained region code generation instructions, the Claude or GPT series models convert region pixel patch fragments into region code fragments.

[0096] (3) Gradual code fusion: Utilizing the decoupling characteristics between element nodes of the Document Object Model, fine-grained region code fragments are directly mounted onto the region container nodes of the coarse-grained skeleton code based on the region association index. Through top-down recursive document object model tree checks, it is ensured that the nodes are mounted correctly, without omissions, redundancies, or incorrect mounting. Finally, the coarse-grained to fine-grained code fusion generation is completed.

[0097] The key points of this invention, a coarse- and fine-grained progressive front-end code generation system based on diffusion models and knowledge distillation, are:

[0098] 1. This invention proposes a conditional generation of UI skeletons using a discrete diffusion model. Pixel images are used as conditional inputs, and through progressive denoising using the discrete diffusion model, a structured skeleton output that conforms to design specifications is finally obtained.

[0099] 2. This invention proposes a skeleton code generation method based on knowledge distillation, which reduces resource consumption and transfers skeleton code generation capabilities from the teacher model to the student model.

[0100] 3. This invention constructs a granular decoupling paradigm of visual-code dual space, and designs a top-down understanding from fine-grained to coarse-grained visual elements, and a bottom-up generation and restoration strategy from coarse-grained to fine-grained code elements.

[0101] The beneficial effects of this invention's coarse-grained and fine-grained progressive front-end code generation system based on diffusion model and knowledge distillation are:

[0102] This invention, through the above technical solutions, provides a UI skeleton diffusion generation module that generates a conditional discrete diffusion model oriented towards the UI skeleton, understands the layout of UI components in the input pixel image, and extracts the layout relationships of the components. The UI skeleton code distillation module, based on knowledge distillation technology, learns the layout structure and layout code relationships annotated by the teacher's model to achieve lightweight implementation of layout code generation. The front-end code progressive fusion module utilizes a divide-and-conquer approach to establish the relationship between pixel image coordinates, layout structure, and code document object model. The model is used to generate code for fine-grained local blocks. Through component coordinates and layout code indexes, the final code is fused, improving the efficiency and quality of front-end low-code generation.

[0103] To achieve the above objectives, this invention also proposes a coarse-to-fine granular progressive front-end code generation method based on a diffusion model and knowledge distillation, such as... Figure 7 As shown, a preferred embodiment of the method of the present invention includes the following steps:

[0104] Step S10: Generate a conditional discrete diffusion model for the UI skeleton, understand the layout of UI components in the input pixel image, and extract the layout relationship of the components.

[0105] Step S20: Based on knowledge distillation technology, a lightweight implementation of layout code generation is achieved by learning the layout structure and layout code relationships annotated by the teacher model.

[0106] Step S30: Using the divide-and-conquer approach, establish the relationship between pixel map coordinates, layout structure, and code document object model. The model is used to generate code for fine-grained local map tiles. The final code is then integrated using component coordinates and layout code indexes.

[0107] Step S10 includes:

[0108] Step S101: Add an interface to skeleton adapter module to learn the matching relationship between pixel map and UI skeleton; the interface to skeleton adapter module consists of three downsampling modules of different scales, which are used to capture components and layout details of different granularities in UI pixel map, respectively. The feature vectors of each layer are extracted through the residual network backbone, projected, and the feature information of each layer is injected into the corresponding layer of the UNet of the noise diffusion model through cross attention.

[0109] Step S102, Conditional generation training process: Freeze the model parameters of unconditional skeleton generation, train only the interface to skeleton adapter part, and coordinate the ratio of unconditional generation and conditional generation through the guide.

[0110] Step S103, conditional generation inference process: Initialize a random skeleton noise, predict the skeleton noise to be removed at each step by using a pre-trained noise predictor and combining the feature information of the UI pixel map input, and remove the noise. After denoising for several consecutive time steps, the decoder finally parses a web page skeleton that conforms to the rules and corresponds to the UI pixel map from the noise vector.

[0111] Step S10 further includes:

[0112] Conditional training data generation design: Collect web page data, and then... The webpage is processed, rendered, and a screenshot is taken to obtain the image. The position and size of outer components are obtained through manual annotation. Each webpage contains a variable number of outer components. The first page The coordinates of each component are denoted as... A data pair consists of a screenshot of a webpage and the location set of the outer components of that webpage. A training data set consisting of all data pairs is denoted as data. ;

[0113] Unconditionally generated training data design: This involves generating training data sets... Each data pair Let's create a separate unconditional skeleton generation dataset, denoted as . ;

[0114] Diffusion process: Generating datasets from unconditional skeletons Coordinates of each component in the skeleton of each webpage Random noise is added after embedding. The latent vector representing the initial webpage skeleton is transformed into a completely random discrete noise vector by adding noise over T time steps. Correspondingly, a regular webpage skeleton will gradually break down over multiple time steps, becoming a random, unstructured skeleton. Each webpage skeleton generates a data sample containing a noisy skeleton across multiple time steps. The set of all samples is denoted as the dataset. ;

[0115] Training process: The data generated during the diffusion process is used to train the model parameters of the discrete skeleton noise predictor. The input to the discrete noise predictor is the time step. and the Noise skeleton at each time step The output label is Ultimately, a discrete skeleton noise predictor that can generate webpage skeletons according to rules is obtained; wherein, the entire training process randomly selects one time step. By learning the relationship between the changes in the noise skeleton at different time steps through a discrete diffusion model, we can learn how to generate a UI webpage skeleton that conforms to rules from random noise.

[0116] Inference process: Initialize a random skeleton noise, predict the skeleton noise to be removed at each step using the noise predictor, and remove the noise. This process is repeated continuously. The noise is denoised at each time step, and finally a skeleton conforming to the rules is extracted from the noise vector by the decoder.

[0117] Step S20 includes:

[0118] Step S201, Model initialization: Select an open-source visual language model with a small parameter scale as the student model, and select a closed-source, high-quality commercial model as the teacher model.

[0119] Step S202, Multimodal distillation instruction design: The distillation instruction includes an image understanding instruction and a code generation instruction. The image understanding instruction requires the teacher model to describe the layout structure of the skeleton diagram of the input UI image using text. The code generation instruction requires the teacher model to generate HTML code that restores the UI layout based on the skeleton diagram of the input UI image.

[0120] Step S203, Automated pseudo-label annotation: Using the image understanding instructions, code generation instructions, and skeleton diagram, the layout text description and layout code corresponding to the skeleton diagram are generated in the classroom model as training pseudo-labels;

[0121] Step S204, Invalid Sample Filtering: Design an automated evaluation framework for UI scenarios to evaluate code execution correctness, image-code matching, and sample diversity, and remove sample data that does not meet the conditions;

[0122] Step S205, Distillation Data Instruction Fine-tuning: Guide the student model to learn the teacher's model image to skeleton code through instruction fine-tuning;

[0123] Step S30 includes:

[0124] Step S301, Region Association Index: Based on the relationship between the region position in the skeleton diagram and the region space occupied in the skeleton code, index the region pixel fragments in the original UI image, and recursively segment the pixel block fragments of each region.

[0125] Step S302, Fine-grained code generation: Using fine-grained region code generation instructions, the Claude or GPT series models convert region pixel patch fragments into region code fragments.

[0126] Step S303, coarse-grained progressive code fusion: Utilizing the decoupling characteristics between element nodes of the Document Object Model, fine-grained region code snippets are directly mounted onto the region container nodes of the coarse-grained skeleton code based on the region association index. Through top-down recursive document object model tree checks, it is ensured that the nodes are mounted correctly, without omissions, redundancies, or incorrect mounting. Finally, the coarse-grained to fine-grained code fusion generation is completed.

[0127] The beneficial effects of the coarse-to-fine granular progressive front-end code generation method based on the diffusion model and knowledge distillation in this invention are:

[0128] This invention, through the aforementioned technical solution, generates a conditional discrete diffusion model oriented towards the UI skeleton, understands the layout of UI components in the input pixel image, and extracts the layout relationships of the components. Based on knowledge distillation technology, it learns the layout structure and layout code relationships annotated by the teacher's model to achieve a lightweight implementation of layout code generation. Utilizing the divide-and-conquer approach, it establishes the relationship between pixel image coordinates, layout structure, and code document object model. The model is used to generate code for fine-grained local blocks. Through component coordinates and layout code indexes, the final code is integrated, improving the efficiency and quality of front-end low-code generation.

[0129] The above description is only a preferred embodiment of the present invention and does not limit the scope of the invention. Any equivalent structural or procedural changes made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of protection of the present invention.

Claims

1. A coarse-fine-grained progressive front-end code generation system based on diffusion model and knowledge distillation, characterized in that, The system comprises an interface skeleton diffusion generation module, an interface skeleton code distillation module and a front-end code progressive fusion module; The interface skeleton diffusion generation module is configured to generate a conditional discrete diffusion model for a UI skeleton, understand the layout of a UI component in an input pixel graph, and complete extraction of the layout relationship of the component; The interface skeleton code distillation module is configured to learn the layout structure and layout code relationship labeled by a teacher model based on a knowledge distillation technique, and complete lightweight implementation of layout code generation; The front-end code progressive fusion module is configured to utilize a divide-and-conquer idea, establish a relationship among a pixel graph coordinate, a layout structure and a code document object model, complete code generation of a fine-grained local block by a model, and realize fusion of a final code through component coordinates and a layout code index; The interface skeleton diffusion generation module is further configured to: add an interface-to-skeleton adapter module to learn a matching relationship between a pixel graph and a UI skeleton; the interface-to-skeleton adapter module comprises three down-sampling modules of different scales, which are configured to capture components and layout details of different granularities in a UI pixel graph, extract feature vectors of each layer through a residual network backbone, project the feature information of each layer, and inject the feature information of each layer into a corresponding layer of a UNet of a diffusion model of predicted noise through cross-attention; a conditional generation training process: freeze model parameters of unconditional skeleton generation, only train the interface-to-skeleton adapter part, and coordinate the proportion relationship between unconditional generation and conditional generation through a guide; a conditional generation inference process: initialize a random skeleton noise, predict the skeleton noise to be removed at each step through a pre-trained noise predictor combined with feature information of a UI pixel graph input, remove the noise, denoise for T consecutive time steps, and finally parse a regular skeleton corresponding to the UI pixel graph from the noise vector through a decoder; The interface skeleton diffusion generation module is further configured to: generate an unconditional discrete diffusion model for a UI skeleton; Conditional generation training data design: collect webpage data, process the ith webpage, render and screenshot to get the picture s i , the position and size of the outer component are obtained by manual annotation, each webpage contains an indefinite number of outer components, the coordinates of the jth component in the ith webpage are recorded as B i,j =(l i,j ,t i,j ,r i,j ,b i,j ), a data pair d i =(s i ,B i ) is composed of a webpage screenshot and the position set of the outer component of the webpage, all data pairs form a training data set, recorded as data D UI,Skeleton ; Unconditional generation training data design: A training data set D UI,Skeleton B of each data pair i Individual constitutes an unconditional skeleton generation data set, recorded as D Skeleton ; Diffusion process: for each skeleton generated data set D Skeleton Coordinates B of each component of each skeleton of each webpage i,j Embedding and random noise addition, z0 represents the initial hidden vector of the webpage skeleton, through T time steps of noise addition, a completely random discrete noise vector z is finally obtained T ; Correspondingly, a regular webpage skeleton will be gradually destroyed and become an irregular random skeleton graph N after multiple time steps i,T Each webpage skeleton generates a data sample N containing multiple time steps of noise skeletons i The whole sample is denoted as data set D Skeleton ; Training process: the data generated in the diffusion process is used to train the model parameters of the discrete skeleton noise predictor, the input of the discrete noise predictor is the time step t and the noise skeleton N of the t time step i,t , the output label is N i,(t-1) , and finally a discrete skeleton noise predictor that can generate regular web page skeletons is obtained; wherein the entire training process randomly selects a time step t, learns the change relationship between the noise skeletons of different time steps through the discrete diffusion model, and learns how to generate a regular UI web page skeleton from random noise; The interface skeleton diffusion generation module is further configured to: an inference process: initialize a random skeleton noise, predict the skeleton noise to be removed at each step through the noise predictor, remove the noise, denoise for T consecutive time steps, and finally parse a regular skeleton through a decoder from the noise vector; The interface skeleton code distillation module is further configured to: model initialization: select a small-parameter-scale open-source visual language model as a student model, and select a closed-source high-quality commercial model as a teacher model; multi-modal distillation instruction design: the distillation instruction comprises an image understanding instruction and a code generation instruction, the image understanding instruction requires the teacher model to input a layout structure of a UI image skeleton graph with a text description, and the code generation instruction requires the teacher model to generate HTML code restoring a UI layout according to the skeleton graph of the input UI image; pseudo-label automatic labeling: the image understanding instruction, the code generation instruction and the skeleton graph are used to generate a layout text description and a layout code corresponding to the skeleton graph in the teacher model as training pseudo-labels; Invalid sample filtering: design an automated evaluation framework for UI scenarios, evaluate code correctness, image-code matching, and sample diversity, and remove samples that do not meet the conditions; Distillation data instruction fine-tuning: guide student model to learn teacher model image to skeleton code ability through instruction fine-tuning; The front-end code progressive fusion module is further used for: Region association index: according to the association relationship between the region position in the skeleton graph and the region placeholder space in the skeleton code, index the region pixel segment in the original UI image, and recursively segment the pixel block segment of each region; Fine-grained code generation: through fine-grained region code generation instructions, Claude or GPT series models are used to convert region pixel block segments into region code segments; Coarse and fine-grained progressive code fusion: using the decoupling characteristics between element nodes of the document object model, according to the region association index, the fine-grained region code segment is directly mounted to the region container node of the coarse-grained skeleton code, and through the top-down recursive document object model tree check, it is ensured that the node mounting is correct, not missing, not redundant, not wrong, and finally, the coarse-grained to fine-grained code fusion generation is completed.

2. A coarse-fine-grained progressive front-end code generation method based on a diffusion model and knowledge distillation, characterized in that, The method comprises the following steps: Step S10, generating a conditional discrete diffusion model for a UI skeleton, understanding the layout of the UI components in the input pixel graph, and completing the extraction of the layout relationship of the components; Step S20, based on the knowledge distillation technology, through learning the layout structure and layout code relationship labeled by the teacher model, completing the lightweight implementation of layout code generation; Step S30, using the divide-and-conquer idea, establishing the relationship among pixel graph coordinates, layout structure, and code document object model, and completing fine-grained local block code generation by the model, and realizing the fusion of the final code through component coordinates and layout code index; The step S10 comprises: Step S101, adding an interface-to-skeleton adapter module to learn the matching relationship between the pixel graph and the UI skeleton; the interface-to-skeleton adapter module is composed of three different scale down-sampling modules, which are respectively used to capture components and layout details of different granularities in the UI pixel graph, extract feature vectors of each layer through a residual network backbone, project and inject feature information of each layer into the corresponding layer of the UNet of the diffusion model of the prediction noise through the cross-attention method; Step S102, conditional generation training process: freeze the model parameters of the unconditional skeleton generation, only train the interface-to-skeleton adapter part, and coordinate the proportion relationship between unconditional generation and conditional generation through the guide; Step S103, conditional generation inference process: initialize a random skeleton noise, through the pre-trained noise predictor, combine the feature information input by the UI pixel graph, predict the skeleton noise to be removed at each step, remove the noise, through continuous T time step denoising, finally through the decoder, parse a regular web skeleton corresponding to the UI pixel graph from the noise vector; The step S10 further comprises: Conditional generation training data design: collect webpage data, process the ith webpage, render and screenshot to get the picture s i , the position and size of the outer component are obtained by artificial labeling, each webpage contains an indefinite number of outer components, the coordinates of the jth component in the ith webpage are recorded as B i,j =(l i,j ,t i,j ,r i,j ,b i,j ), a data pair d i =(s i ,B i ) is composed of a webpage screenshot and the position set of the outer component of the webpage, all data pairs form a training data set, recorded as data D UI,Skeleton ; Unconditional generation training data design: A training data set D UI,Skeleton B of each data pair i Individual constitutes an unconditional skeleton generation data set, denoted as D Skeleton ; Diffusion process: for the unconditional skeleton generation dataset D Skeleton Coordinates B of each component of each web page skeleton i,j Embedding and random noise addition, z0 represents the initial hidden vector of the web page skeleton, through T time steps of noise addition, a completely random discrete noise vector z is finally obtained T ; Correspondingly, a regular web page skeleton will be gradually destroyed and become a random skeleton graph N after multiple time steps i,T Each web page skeleton generates a data sample N containing multiple time step noise skeletons i The whole sample is denoted as dataset D Skeleton ; Training process: the data generated in the diffusion process is used to train the model parameters of the discrete skeleton noise predictor, the input of the discrete noise predictor is the time step t and the noise skeleton N of the t time step i,t , the output label is N i,(t-1) , and finally a discrete skeleton noise predictor that can generate regular web page skeletons is obtained; wherein the entire training process randomly selects a time step t, learns the change relationship between the noise skeletons of different time steps through the discrete diffusion model, and learns how to generate a regular UI web page skeleton from random noise; The inference process includes: initializing a random skeleton noise, predicting the skeleton noise to be removed at each step through the noise predictor, removing the noise, denoising for T consecutive time steps, and finally parsing a regular skeleton from the noise vector through the decoder; The step S20 includes: Step S201, model initialization: selecting a small parameter scale open source visual language model as a student model, and selecting a closed source high quality commercial model as a teacher model; Step S202, multi-modal distillation instruction design: the distillation instruction includes an image understanding instruction and a code generation instruction, the image understanding instruction requires the teacher model to input the layout structure of the UI image skeleton graph with a text description, and the code generation instruction requires the teacher model to generate HTML code restoring the UI layout according to the skeleton graph of the input UI image; Step S203, pseudo-label automatic annotation: using the image understanding instruction and the code generation instruction, the skeleton graph, and the teacher model, generating the layout text description and the layout code corresponding to the skeleton graph as training pseudo-labels; Step S204, invalid sample filtering: designing an automatic evaluation framework for UI scenes, evaluating the code running correctness, image-code matching, and sample diversity, and removing the sample data that does not meet the conditions; Step S205, distillation data instruction fine-tuning: guiding the student model to learn the ability of the teacher model from image to skeleton code through instruction fine-tuning; The step S30 includes: Step S301, region association index: according to the association relationship between the region position in the skeleton graph and the region placeholder space in the skeleton code, the region pixel segment in the original UI image is indexed, and the pixel block segment of each region is recursively divided; Step S302, fine-grained code generation: through the fine-grained region code generation instruction, the Claude or GPT series model is used to convert the region pixel block segment into a region code segment; Step S303, coarse and fine-grained progressive code fusion: using the decoupling characteristics between element nodes of the document object model, according to the region association index, the fine-grained region code segment is directly mounted to the region container node of the coarse-grained skeleton code, through the top-down recursive document object model tree check, the node mounting is ensured to be correct, not missing, not redundant, not wrong, and finally, the coarse-grained to fine-grained code fusion generation is completed.

Images