User interface (UI) code generation methods, devices, and electronic devices

Abstract

This disclosure provides a method, apparatus, and electronic device for generating user interface (UI) code, relating to the field of computer technology, particularly to the fields of artificial intelligence, natural language processing, large-scale modeling, and deep learning. The specific implementation scheme is as follows: Obtaining design requirement information for the target interface to be designed; responding to the design requirement information, including the UI design diagram, using an element detection model to perform element detection on the UI design diagram, obtaining element detection results; wherein, the UI design diagram displays the UI components required for the target interface; using a Visual Language Model (VLM), based on the element detection results and the UI design diagram, obtaining the first page layout information of the target interface, the first component attribute information of each UI component, and the first logical relationship information between each UI component; generating the interface code of the target interface based on the first page layout information, the first component attribute information, and the first logical relationship information.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence, natural language, large models, deep learning, etc., and especially to a method, apparatus and electronic device for generating user interface (UI) code. Background Technology

[0002] With the rapid development of front-end development technologies and visualization tools, low-code platforms have emerged, providing convenience for the rapid construction of applications. However, complex pages still require developers to manually add and arrange a large number of components and manually configure properties and events. Essentially, this transforms coding work into graphical operations, which is not flexible enough in supporting complex business logic. Summary of the Invention

[0003] This disclosure provides a method, apparatus, and electronic device for generating user interface (UI) code.

[0004] According to one aspect of this disclosure, a method for generating user interface (UI) code is provided. The method includes: obtaining design requirement information of a target interface to be designed; responding to the design requirement information including a UI design drawing, performing element detection on the UI design drawing using an element detection model to obtain element detection results; wherein the UI design drawing displays UI components required by the target interface; using a Visual Language Model (VLM), based on the element detection results and the UI design drawing, obtaining first page layout information of the target interface, first component attribute information of each UI component, and first logical relationship information between each UI component; and generating interface code of the target interface according to the first page layout information, the first component attribute information, and the first logical relationship information.

[0005] According to another aspect of this disclosure, a user interface (UI) code generation apparatus is provided. The apparatus includes: a first acquisition module, configured to acquire design requirement information of a target interface to be designed; a detection module, configured to, in response to the design requirement information including a UI design drawing, perform element detection on the UI design drawing using an element detection model to obtain element detection results; wherein the UI design drawing displays UI components required by the target interface; a second acquisition module, configured to, using a visual language model (VLM), acquire first page layout information of the target interface, first component attribute information of each UI component, and first logical relationship information between each UI component based on the element detection results and the UI design drawing; and a first generation module, configured to generate interface code for the target interface based on the first page layout information, the first component attribute information, and the first logical relationship information.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the user interface (UI) code generation method proposed above in this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to execute the user interface (UI) code generation method proposed above in this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the user interface (UI) code generation method described above.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure; Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure; Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure; Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure; Figure 5 This is a schematic diagram of the architecture of the artificial intelligence coding platform provided in this disclosure; Figure 6 This is a schematic diagram according to the fifth embodiment of the present disclosure; Figure 7 This is a block diagram of an electronic device used to implement the user interface (UI) code generation method of the embodiments of this disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] This disclosure presents a method, apparatus, and electronic device for generating user interface (UI) code.

[0014] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure. It should be noted that the user interface (UI) code generation method of the present disclosure can be applied to a user interface (UI) code generation device, which can be configured in an electronic device so that the electronic device can perform the user interface (UI) code generation function.

[0015] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, smart speakers, servers, server clusters, and other hardware devices with various operating systems, touch screens and / or displays.

[0016] The user interface (UI) code generation device can also be software within an electronic device, such as UI code generation software. In the following embodiments, an electronic device is used as an example for illustration.

[0017] like Figure 1 As shown, the UI code generation method may include the following steps: Step 101: Obtain the design requirements information for the target interface to be designed.

[0018] The target interface to be designed refers to the user interface that needs to be implemented and has specific functions and visual effects in the project development or product iteration process.

[0019] It should be noted that this disclosure does not restrict the application scenarios, target users, or core functions of the target interface. For example, the target interface may include e-commerce detail pages, backend management forms, APP homepages, etc., the target users may be ordinary users or relevant staff, etc., and the core functions may include displaying data, data interaction, data submission, etc.

[0020] It should also be noted that this disclosure does not limit the number of target interfaces to be designed; there may be one or more.

[0021] The design requirements information may include text information describing the target interface, images displaying text information describing the target interface, or images that visually represent the target interface, etc. This disclosure does not limit this.

[0022] In this embodiment of the disclosure, design requirement information for the target interface to be designed can be obtained. As an example, the design requirement information for the target interface to be designed can be obtained in response to user input.

[0023] The forms of input operations include, but are not limited to: voice input, keyboard input, mouse input, scanning input, etc., and this disclosure does not impose any restrictions on them.

[0024] Step 102: In response to the design requirement information, including the UI design drawing, an element detection model is used to perform element detection on the UI design drawing to obtain the element detection results.

[0025] Among them, the UI design diagram can show the UI components required for the target interface.

[0026] Among them, UI design diagrams can be visual renderings drawn by relevant users based on design requirements, which can intuitively present the visual effects of a single interface, or they can be flowcharts that intuitively present the interaction flow and operation path of the user interface in a graphical way.

[0027] Among them, the element detection model can be a pre-trained target detection model (also known as an object detection model), such as YOLOv8 (You Only Look Once version 8), Faster R-CNN (Faster Region-based Convolutional Neural Network), DETR (Detection Transformer), etc., which can be used to detect or recognize elements in UI design drawings.

[0028] The element detection results may include, but are not limited to: the type of the element, the position information of the element in the UI design drawing, the text corresponding to the element, etc., and this disclosure does not impose any restrictions on them.

[0029] As an example, when the design requirements information includes UI design sketches, the UI design sketches can be input into an element detection model to perform element detection on the UI design sketches, and the element detection results can be obtained in response to the output of the element detection model.

[0030] Optionally, in some embodiments, the element detection model includes an RPN and a detection head network.

[0031] As one possible approach, the UI design draft can be preprocessed before element detection.

[0032] The preprocessing may include, but is not limited to, image enhancement, image normalization, format conversion, image noise reduction, size adjustment, etc., and this disclosure does not limit it.

[0033] As one possible implementation, a quality score for the UI design draft can be determined before element detection is performed; if the quality score is less than a set threshold, a prompt message is generated and sent to remind the user to optimize the UI design draft.

[0034] To obtain a quality score for a UI design draft, for example, the UI design draft can be evaluated in multiple dimensions; the evaluation scores in multiple dimensions are then weighted and summed to obtain the quality score of the UI design draft.

[0035] These dimensions can include image clarity, contrast effectiveness, effective area ratio, size adaptability, noise interference, and so on.

[0036] For example, UI design drafts can be input into a quality assessment model, and the model's output can be used to obtain a quality score for the UI design drafts. The quality assessment model can be pre-trained and has the function of evaluating image quality.

[0037] Step 103: Using VLM (Vision-Language Model), based on element detection results and UI design diagrams, obtain the first page layout information of the target interface, the first component attribute information of each UI component, and the first logical relationship information between each UI component.

[0038] The layout information of the first page may include, but is not limited to: page type, layout type, component hierarchy, spacing rules, responsive logic, etc.

[0039] The page types can include, but are not limited to, shopping pages, dashboard pages, etc., and can be set as needed. This disclosure does not impose any restrictions on them.

[0040] The layout types may include flow layout, grid layout, flexible layout, absolute positioning layout, etc., and this disclosure does not limit them.

[0041] Spacing rules can be used to indicate the spacing between components, or the distance between a component and the four borders of the page, etc.

[0042] The first component's attribute information may include, but is not limited to, visual attributes, functional attributes, and layout attributes. Visual attributes may include, for example, color, size, font, border, rounded corners, and shadow. Functional attributes may include different states of the component (e.g., the default state of a button, mouse hover state, and click state), pre-set placeholder text within the input box (e.g., "Please enter your email address"), and the source path corresponding to image loading. Layout attributes may include the component's width and height, and positioning method (e.g., an input box width of 300px and center alignment).

[0043] The first logical relationship information can be used to indicate the interaction logic and data association between various UI components.

[0044] As an example, element detection results and UI design diagrams can be input into VLM, and in response to the output of VLM, the first page layout information of the target interface, the first component attribute information of each UI component, and the first logical relationship information between each UI component can be obtained. The first page layout information can be described in a structured form, the first component attribute information can be in JSON (JavaScript Object Notation) format, and the first logical relationship information can be described in the form of a flowchart.

[0045] Step 104: Generate the interface code of the target interface based on the first page layout information, the first component attribute information, and the first logical relationship information.

[0046] As an example, a code generation tool maps the first page layout information to HTML (Hyper Text Markup Language) code or CSS (Cascading Style Sheets) code, the first component attribute information to CSS code, and the first logical relationship information to JavaScript code. The above code is then merged to obtain modular code, such as Vue or React components. A series of syntax and integrity checks are then performed on the merged code to obtain the interface code of the target interface.

[0047] Syntax checks may include verifying the closure and nesting of HTML tags, the validity of HTML attribute values, the validity of CSS attribute values ​​and the validity of units, the syntactic correctness and logical rigor of JavaScript statements, etc., and this disclosure does not impose any restrictions on them.

[0048] Integrity checks may include, but are not limited to, verifying the data integrity, functional integrity, and structural integrity of the code.

[0049] The UI code generation method in this embodiment of the disclosure obtains design requirement information of the target interface to be designed; in response to the design requirement information including a UI design diagram, an element detection model is used to perform element detection on the UI design diagram to obtain element detection results; wherein, the UI design diagram shows the UI components required by the target interface; a Visual Language Model (VLM) is used to obtain the first page layout information of the target interface, the first component attribute information of each UI component, and the first logical relationship information between each UI component based on the element detection results and the UI design diagram; and the interface code of the target interface is generated according to the first page layout information, the first component attribute information, and the first logical relationship information. Therefore, it can automatically parse the component composition, layout rules, and logical relationships in UI design drawings, avoiding the tedious process of manually identifying design drawings and writing interface code, effectively shortening the cycle from design finalization to code implementation, and reducing labor costs. With the powerful understanding and data processing capabilities of AI models, page layout information, component attribute information, and logical relationship information can be quickly extracted, effectively avoiding problems such as human interpretation bias, incorrect component parameters, and errors in logical relationship sorting, improving the high degree of fidelity between the generated interface code and the design drawings, and improving the development quality of the target interface. At the same time, it can realize the automated conversion of UI design requirements information into code, reducing the dependence on the design interpretation ability of front-end developers, and improving the flexibility and feasibility of application development processes.

[0050] like Figure 2 As shown, this disclosure also proposes a method for generating user interface (UI) code. Figure 2 The diagram is based on a second embodiment of this disclosure. The user interface (UI) code generation method may include the following steps: Step 201: Obtain the design requirements information for the target interface to be designed.

[0051] It should be noted that the execution process of step 201 can refer to the execution process of any embodiment of this disclosure, and will not be repeated here.

[0052] Step 202: In response to the design requirement information including the UI design drawing, the RPN in the element detection model is used to perform a first detection on the UI design drawing to obtain multiple candidate image regions; wherein, any candidate image region displays UI components.

[0053] It should be noted that the explanation of the UI design diagram and element detection model in step 102 also applies to this embodiment, and will not be repeated here.

[0054] In this embodiment of the disclosure, the element detection model may include RPN (Region Proposal Network).

[0055] As an example, the UI design sketch can be input into the RPN in the element detection model to perform the first detection on the UI design sketch through the RPN, identify the image region containing UI components, and obtain multiple candidate image regions in response to the output of the RPN.

[0056] Optionally, in some embodiments, FPN (Feature Pyramid Network) is used to extract features from the UI design drawing to obtain multi-scale visual features; RPN is used to perform a first detection on the multi-scale visual features to obtain multiple candidate image regions.

[0057] As an example, a convolutional network of the backbone network in an element detection model can be used to extract features from the UI design image based on different receptive fields, obtaining multi-scale initial features. Then, along the bottom-up path, top-down path, and lateral path of the FPN, the multi-scale initial features are fused and enhanced to obtain multi-scale visual features. Finally, the RPN can be used to perform the first detection on the visual features at each scale, generating multiple location information. Any location information can be used to indicate the corresponding candidate image region.

[0058] Therefore, the multi-scale feature extraction and detection method based on FPN can fully adapt to the detection needs of UI components of different sizes and levels in UI design drawings, and improve the accuracy and comprehensiveness of candidate image region generation.

[0059] Step 203: Use the detection head network in the element detection model to perform a second detection on any candidate image region to obtain the corresponding region detection result.

[0060] Among them, the region detection results can be used to indicate the atomic visual category to which the UI components displayed in the corresponding candidate image region belong.

[0061] Optionally, in some embodiments, the region detection results can also be used to indicate at least one of the following: the confidence level of the candidate image region and the location information of the candidate image region. This allows for the automatic extraction of diverse information about the candidate image region, improving the usability and accuracy of the subsequently determined element detection results and providing data support for subsequent VLM input.

[0062] Among them, the atomic vision category can be a predefined atomic category.

[0063] Optionally, in some embodiments, the atomic vision categories may include: shape categories, text categories, interaction categories, container categories, and data display categories. It should be noted that the atomic vision categories can be set as needed, and this disclosure does not impose any limitations on this.

[0064] The categories include: shape (e.g., rectangles, circles, lines, composite shapes, icons); text (e.g., title text, body text, label text, interactive text); interaction (e.g., clickable areas, input areas, selection controls, slider controls, toggle controls, navigation elements, action triggers, gesture areas); container (e.g., content containers, list containers, grid containers, modal containers); and data display (e.g., data visualization, media display, progress indicators, status indicators).

[0065] Therefore, classifying atomic vision categories into shape categories, text categories, interaction categories, container categories, data display categories, etc., helps to accurately determine the category of UI components, improves the structure and refinement of element detection results, provides data support for subsequent visual language model parsing and interface code, and helps to improve the accuracy and effectiveness of visual language model output.

[0066] As an example, for any candidate image region, the candidate image region can be aligned with the ROI (Region of Interest) to obtain the corresponding feature tensor; the feature tensor is encoded using a fully connected layer in the element detection model to obtain the target feature; the target feature is input into the detection head network in the element detection model to perform a second detection on the target feature, including classification prediction and regression prediction. The classification prediction can output the atomic visual category and confidence level of the UI component displayed in the candidate image region, and the regression prediction can output the location information of the candidate image region.

[0067] Step 204: Determine the element detection results based on the region detection results.

[0068] As an example, region detection results can be used as element detection results.

[0069] As another example, when the region detection results include the confidence levels of candidate image regions, the candidate image regions can be filtered based on the confidence levels, and the filtered region detection results can be used as element detection results. For instance, the region detection results of candidate image regions with confidence levels greater than a set confidence threshold can be determined as element detection results.

[0070] Step 205: Using VLM, based on the element detection results and UI design diagram, obtain the first page layout information of the target interface, the first component attribute information of each UI component, and the first logical relationship information between each UI component.

[0071] It should be noted that the execution process of step 205 can refer to the execution process of any embodiment of this disclosure, and will not be described in detail here.

[0072] Optionally, in some embodiments, a first prompt word is determined based on the element detection results and the UI design diagram; the first prompt word is input into the VLM to obtain configuration code; wherein, the configuration code is used to describe the page layout of the target interface, the component attributes of each UI component, and the logical relationship between each UI component; the first page layout information, the first component attribute information, and the first logical relationship information are read from the configuration code.

[0073] As an example, the element detection results and UI design diagrams can be filled into the corresponding positions in the first prompt word template to obtain the first prompt word; then, the first prompt word is input into VLM and the output of VLM is responded to to obtain the configuration code; finally, the first page layout information, the first component attribute information and the first logical relationship information can be read from the configuration code.

[0074] Optionally, in some embodiments, during the process of obtaining the first prompt word, code generation rules can also be filled into the corresponding position in the first prompt word template.

[0075] Among them, code generation rules can be used to indicate the rules that the generated configuration code must meet, such as the syntax rules, format rules, etc. that the generated configuration code must meet.

[0076] It should be noted that the code generation rules corresponding to different information can be the same or different.

[0077] To clearly illustrate how the first prompt word is input into the VLM to obtain the configuration code, optionally, in some embodiments, the first prompt word is input into the VLM model to obtain the initial analysis result; wherein, the initial analysis result may include the initial page layout information of the target interface, the initial component attribute information of each UI component, and the initial logical relationship information between each UI component; and the corresponding configuration code is generated based on the initial page layout information, the initial component attribute information, and the initial logical relationship information, respectively.

[0078] The initial page layout information can be used to describe the page layout of the target interface; the initial component attribute information can be used to describe the component attributes of the corresponding UI components in the target interface; and the initial logical relationship information can be used to describe the logical relationships between the UI components.

[0079] As an example, design a structured first cue word, requiring the model to output initial analysis results in JSON format. The first cue word could be, for example:

[0080] The first prompt word is input into the VLM, and in response to the output of the VLM, the structured initial analysis results are obtained.

[0081] One possible implementation is to interact with the user to obtain supplementary information about the initial analysis results, and then use LLM to parse the supplementary information to obtain parameter information for supplementing the initial analysis results. The parameter information is then added to the initial analysis results to obtain the configuration code.

[0082] In order to obtain the configuration code corresponding to the initial page layout information, optionally, in some embodiments, a first target layout template is determined from a plurality of preset first layout templates based on the initial page layout information; and the configuration code corresponding to the initial page layout information is generated based on the layout parameters of the first target layout template.

[0083] The first layout template can be a pre-set or pre-defined reusable page layout baseline, which may include standard layout parameters and element combination features adapted to the layout template. For example, if the layout template is a login page template, the tuple combination features that the login page template needs to adapt to should include a username input box, a password input box, a login button, etc. It should be noted that the example of this layout template is only exemplary, and in actual applications, it can be other types, and this disclosure does not limit it.

[0084] It should be noted that this disclosure does not restrict the layout method adopted by the first layout template, including but not limited to: absolute positioning or relative positioning, flexible layout or grid layout, etc.

[0085] As an example, if any first layout template has a corresponding code template or code framework, the first layout template that matches the initial page information from a set of preset first layout templates can be selected based on the initial page layout information, and this first layout template can be used as the first target layout template. Then, the layout parameters corresponding to the first target layout template are filled into the corresponding positions in the code template to obtain the configuration code. The code template can be a JSON format code template, a CSS code template, etc., and this disclosure does not impose any restrictions on it.

[0086] In other embodiments, after determining the first target layout template, the position information of the UI components in the element detection results can be mapped to CSS parameters under the first target layout template; then, the CSS parameters are extracted to obtain the target CSS parameters, and the target CSS parameters are assembled to obtain the corresponding configuration code.

[0087] Among them, CSS parameters may include, but are not limited to: column template properties, row template properties, column properties of grid items, row properties of grid items in grid layout mode, margin properties, basic size properties of flexible items, alignment properties of items within flexible containers in flexible layout mode, top distance properties, left distance properties, width properties, height properties in positioning layout mode, etc.

[0088] It should be noted that target CSS parameters may include display type, spacing properties, positioning method, grid template properties, flexible item arrangement direction properties, etc.

[0089] Therefore, by matching the first target layout template from multiple preset first layout templates, proven and mature layout frameworks can be directly reused, replacing the process of coding layout logic from scratch. This effectively shortens the generation cycle of layout configuration code and improves the generation efficiency and standardization of layout configuration code. On the other hand, generating configuration code based on the layout parameters of the first target layout template ensures that the coding process of layout information strictly follows the preset parameter specifications of the template, avoiding problems such as layout parameter confusion and poor adaptability caused by custom coding, and ensuring that the generated layout configuration code has good compatibility and maintainability.

[0090] In order to obtain the configuration code corresponding to the initial component attribute information, optionally, in some embodiments, when the initial component attribute information includes component type description information, a first target component template is determined from a plurality of predefined first component templates based on the component type description information; based on the initial component attribute information, the code structure of the first target component template is filled with information to obtain the configuration code corresponding to the initial component attribute information.

[0091] It should be noted that the initial component attribute information may include component type description information, which can be used to describe the component type of the corresponding UI component. For example, the component type description information may be "an input box with a search icon".

[0092] Each of the first component templates can be defined using an instantiable code structure, such as JSON format.

[0093] To clearly illustrate how the first target component template is determined from a plurality of predefined first component templates based on component type description information, optionally, in some embodiments, a first semantic vector of the component type description information and a second semantic vector of the preset component description information corresponding to each first component template are obtained; for any first component template, the similarity between the component type description information and the first component template is determined according to the second semantic vector and the first semantic vector corresponding to the first component template; and the first target component template is determined from a plurality of first component templates based on the similarity.

[0094] Each of the first component templates may have corresponding preset component description information. It should be noted that the preset component description information can be pre-defined and is used to describe the component-related information (such as attribute information, functional information, etc.) corresponding to the first component template.

[0095] To obtain the first semantic vector, for example, keyword extraction can be performed on the component type description information to obtain multiple keywords corresponding to the component type description information; then, the feature values ​​corresponding to each keyword are determined, and the first semantic vector is generated based on the feature values ​​corresponding to each keyword; alternatively, the component type description information can be input into a feature extraction model, and the first semantic vector can be obtained in response to the output of the model. It should be noted that the feature extraction model has the function of feature extraction, and this disclosure does not limit the feature extraction model.

[0096] It should also be noted that the method for obtaining the second semantic vector of the preset component description information corresponding to any first component template can be similar to the method for obtaining the first semantic vector, and will not be elaborated here.

[0097] In addition, as a possible implementation, the second semantic vector of the preset component description information corresponding to any first component template can be saved to the relevant database for subsequent data processing.

[0098] To determine the similarity between component type description information and the first component template, for example, for any first component template, a similarity algorithm, such as cosine similarity algorithm or Euclidean distance algorithm, can be used to determine the similarity between the second semantic vector corresponding to the first component template and the first semantic vector, and the similarity between the second semantic vector corresponding to the first component template and the first semantic vector is determined as the similarity between component type description information and the first component template.

[0099] Therefore, in this embodiment of the disclosure, the first component template corresponding to the maximum similarity can be used as the first target component template.

[0100] Therefore, a semantic vector similarity matching mode can be adopted to automatically and intelligently filter out component templates that match the initial component attribute information from multiple first component templates, thereby improving the accuracy and effectiveness of the first target component templates obtained through filtering.

[0101] In this embodiment of the disclosure, for example, initial component attribute information can be filled into the corresponding positions in the code structure of the first target component template to obtain the configuration code corresponding to the initial component attribute information. Thus, on the one hand, by matching the first target component template from predefined templates based on component type description information, standardized code structures preset for different types of UI components can be directly reused, replacing the process of building a code framework from scratch for each component, thereby improving the generation efficiency of component configuration code. On the other hand, by filling the code structure of the target template with information based on the initial component attribute information, the visual attributes and functional attributes of the UI components in the design drawing can be mapped to the corresponding parameter positions in the template, ensuring that the generated component configuration code can completely and accurately restore the design attributes of each UI component. Furthermore, the template-based filling method also makes component attribute modifications more targeted. If only the attributes of a certain component need to be adjusted, only the filled attribute values ​​need to be modified, without changing the core code structure of the component, which can effectively reduce the maintenance cost of configuration code and improve the response efficiency of attribute information changes.

[0102] Therefore, the VLM model is first used to semantically organize the relevant information of the UI design draft, and output the initial analysis results including the initial page layout, component attributes, and logical relationships. This transforms visual information into understandable design logic information, avoiding coding chaos caused by the mixing of information dimensions when directly generating configuration code. On the other hand, based on the initial analysis results, corresponding configuration codes are generated for different dimensions. Adaptive coding rules can be used for the three different dimensions of information: page layout, component attributes, and logical relationships, to obtain the corresponding configuration codes, thereby improving the effectiveness, flexibility, and accuracy of configuration code acquisition.

[0103] In summary, the first prompt words constructed based on element detection results and UI design diagrams can transform fragmented visual detection data into text instructions that VLM can understand. This guides VLM to focus on core design information, avoids misunderstandings caused by fragmented information, and ensures that the configuration code generated by VLM can fully cover page layout, component attributes, and logical relationships. It enables structured storage and parsing of page layout information, component attribute information, and logical relationship information. Consequently, page layout information, component attribute information, and logical relationship information can be automatically obtained from the configuration code, improving the flexibility, efficiency, and intelligence of information acquisition.

[0104] Step 206: Generate the interface code of the target interface based on the first page layout information, the first component attribute information, and the first logical relationship information.

[0105] It should be noted that the execution process of step 206 can refer to the execution process of any embodiment of this disclosure, and will not be described in detail here.

[0106] Optionally, in some embodiments, page rendering is performed based on the interface code of the target interface to obtain a preview page; in response to the confirmation operation of the preview page, a web application of the target interface is generated and deployed based on the interface code.

[0107] In one example, an independent and controlled preview environment, or sandbox environment, can be created. In this environment, the DOM mechanism of the front-end framework, such as Vue's render function or React's createElement, is utilized. This leverages the runtime compilation and rendering capabilities of the front-end framework to dynamically create in-memory component instances from the target interface's code. Simulated data or real-world states are injected into the components via data binding. Finally, an interactive preview interface is mounted and rendered in an isolated container (such as a DOM container). When the user confirms the preview page, the interface code is deemed error-free. The necessary dependencies are then installed in the build environment, followed by a quality scan and security check. If the security check passes, the code is built and packaged, resulting in a directly deployable static web application. Subsequently, a Docker image is generated using an image file template and the built artifacts. Based on the preset deployment configuration information, the web application is deployed to the target deployment environment indicated by the deployment configuration information. It should be noted that after successful deployment, the execution entity of this disclosure can generate and send an accessible URL (Uniform Resource Locator) to the user.

[0108] The target deployment environment can be a Kubernetes cluster or a cloud server, etc.

[0109] Therefore, generating preview pages based on interface code rendering transforms automatically generated code into visual page effects. This allows relevant personnel to intuitively verify whether the generated interface fully matches the visual effects and interaction logic of the UI design. It also enables timely detection and correction of layout deviations, component attribute errors, and logical relationship anomalies that may occur during the code generation stage. This avoids online failures or rework costs caused by discovering problems only after direct deployment. By using visualization, the barrier to entry for non-technical personnel to participate in acceptance is lowered. They can intuitively judge whether the interface meets expectations without viewing the code. The web application is only generated and deployed after the preview page is confirmed. This effectively reduces the delivery risks caused by the uncertainty of automatically generated code and improves the accuracy and usability of the target interface delivery.

[0110] Optionally, in some embodiments, in response to an editing operation performed on the second component in the preview page, the interface code is updated according to the edited second component.

[0111] Editing operations include, but are not limited to: deletion, addition, component replacement, changing component position or size by dragging and dropping, text modification, style modification, component attribute migration, etc.

[0112] As an example, in response to an editing operation performed on the second component in the preview page, the execution subject of this disclosure can call an editor, determine the code snippet corresponding to the second component from the interface code based on the second component, parse the editing operation through the editor to determine the new component parameters, and update the corresponding code snippet with the new component parameters to update the interface code.

[0113] Therefore, it supports direct editing of the second component on the preview page, eliminating the need for developers to manually locate and modify the underlying code. This transforms professional code modification into an intuitive visual operation, lowering the technical barrier to interface fine-tuning. Even non-front-end developers can directly participate in interface optimization, improving tuning efficiency. Furthermore, after editing, the interface code can be automatically updated, ensuring that the visual adjustments are completely consistent with the underlying code logic. This avoids issues such as mismatch between visual effects and code logic, and omissions in parameter modifications that may occur when manually modifying code, thus maintaining the accuracy and integrity of the code.

[0114] Optionally, in some embodiments, in response to a task instruction to update the preview page, an LLM (Large Language Model) is used to determine page update requirement information based on the task instruction; according to the page update requirement information, the target code block to be updated is determined from the interface code, and the target code block is updated.

[0115] The update requirement information may include the component to be updated, the operation type, the operation parameters, etc., and this disclosure does not impose any restrictions on this. The operation type may include, for example, modifying attributes, adding components, deleting components, etc.; the operation parameters may include, for example, the attribute values ​​to be updated (such as color values, size, etc.), text content, etc.

[0116] For example, in response to a user input task instruction to update the preview page, an LLM can be used to perform semantic understanding of the task instruction to obtain page update requirement information. Then, based on the component to be updated indicated by the page update requirement information, the target code block to be updated is determined from the interface code. Finally, the target code block is updated according to the operation type and operation parameters.

[0117] Therefore, by leveraging the powerful understanding and data processing capabilities of large models, adjustments and updates to the interface code can be made, thereby improving the efficiency and intelligence of page updates.

[0118] The UI code generation method of this disclosure uses the RPN in the element detection model to perform a first detection on the UI design drawing to obtain multiple candidate image regions; wherein any candidate image region displays UI components; the detection head network in the element detection model is used to perform a second detection on any candidate image region to obtain the corresponding region detection result; wherein the region detection result is used to indicate the atomic visual category to which the UI component displayed in the corresponding candidate image region belongs; and the element detection result is determined based on the region detection result. Therefore, on the one hand, by first generating candidate image regions containing UI components through RPN, meaningless blank or background areas in the design can be quickly filtered out, focusing the detection attention on areas that may contain the target UI components. This helps reduce the computational load of the subsequent detection head network and improves the overall detection efficiency. On the other hand, by using the detection head network to perform refined detection on the filtered candidate regions and identify the atomic visual category to which the UI components belong, the problems of component omission, category confusion, or positioning deviation caused by directly detecting individual design drafts are avoided, ensuring that each UI component can be accurately classified and located. At the same time, the detection results based on atomic visual categories can provide standardized and structured basic data for the subsequent VLM model to parse the layout, attributes, and logical relationships, maintaining the accuracy of information transmission throughout the entire process from design drafts to code generation, and providing data support for the final generation of highly realistic interface code.

[0119] like Figure 3 As shown, this disclosure also proposes a method for generating user interface (UI) code. Figure 3 The diagram is based on a third embodiment of this disclosure. The user interface (UI) code generation method may include the following steps: Step 301: Obtain the design requirements information for the target interface to be designed.

[0120] It should be noted that the execution process of step 301 can refer to the execution process of any embodiment of this disclosure, and will not be repeated here.

[0121] Step 302: In response to the design requirement information, including the UI description text, LLM is used to perform intent recognition on the UI description text to obtain semantic intent information; wherein, the semantic intent information includes the element attribute information of the page elements in the target interface and the page layout information of the target interface.

[0122] The UI description text can be used to describe the target interface to be designed. As an example, the UI description text could be: "Create a dashboard homepage with a data overview card at the top, a sales trend line chart in the middle, and a list of recent transactions at the bottom, supporting filtering by date." In one example, the UI description text can be text obtained by recognizing a speech signal, text obtained by recognizing an image, raw text, etc., and this disclosure does not limit it.

[0123] The semantic intent information may include element attribute information of page elements in the target interface and page layout information of the target interface.

[0124] Among them, element attribute information can be used to describe the corresponding page elements, such as element type, quantity, logical interface information related to the page element, etc.

[0125] The element types can include data card rows, line charts, data tables, etc.

[0126] The logical interface information can be used to indicate the interface associated with the corresponding page element and the corresponding logical relationship between the interface and the page element.

[0127] The page layout information describes the layout of page elements on the target interface. For example, the page layout information is: the overall layout is a vertical linear layout, with "cards at the top, line charts in the middle, and data tables at the bottom".

[0128] As an example, suppose the UI description text is "Create a dashboard homepage with a data overview card at the top, a sales trend line chart in the middle, and a recent transaction list at the bottom, supporting filtering by date." When the design requirements include UI description text, the UI description text can be input into an LLM (Local Management Library) for semantic analysis and intent recognition. The LLM output will then yield application blueprint information in JSON format, referred to in this disclosure as semantic intent information. For example, the application blueprint information might look like this: { "pageTitle": "Data Dashboard", "sections": [ { "type": "StatsCardRow", "count": 4, "dataSource": "Overview API"}, { "type": "LineChart", "title": "Sales Trend", "dataSource": "Trend API"}, { "type": "DataTable", "title": "Recent Transactions", "dataSource": "Transaction API", "hasFilter": true, "filterField": "date"} ] } Here, type indicates the element type of the corresponding page element, count indicates the number of corresponding page elements, and dataSource indicates that the API (Application Programming Interface) and logical relationship corresponding to the page element are the data source.

[0129] It should be noted that this disclosure does not limit the number of page elements on the target interface; it can be, but is not limited to, one.

[0130] Step 303: Generate the component code corresponding to the page element based on the element attribute information of the page element.

[0131] Optionally, in some embodiments, when there is at least one page element and the element attribute information includes the element type, for any page element, a second target component template is determined from a plurality of predefined second component templates according to the element type of the page element; in response to the element attribute information including logical interface information related to the page element, a corresponding first target interface is determined from a plurality of preset first interfaces according to the logical interface information; based on the interface configuration information of the first target interface and the element attribute information of the page element, the second target component template is populated with information to obtain the component code corresponding to the page element.

[0132] The second component template can be defined using an instantiable code structure. Optionally, any second component template can have corresponding preset component description information.

[0133] Each of the first interfaces has corresponding interface configuration information, which can be predefined.

[0134] To obtain the second target component template corresponding to a page element, as an example, for any page element, the third semantic vector of the component type corresponding to the page element and the fourth semantic vector of the preset component description information corresponding to any second component template can be obtained; a similarity algorithm is used to determine the similarity between the third semantic vector and any fourth semantic vector; the second component template with the highest similarity among multiple second component templates is determined as the second target component template corresponding to the page element.

[0135] To obtain the first target interface corresponding to a page element, as an example, we can obtain the fifth semantic vector of the logical interface information corresponding to the page element and the sixth semantic vector of the interface description information of any interface; use a similarity algorithm to determine the similarity between the fifth semantic vector and any sixth semantic vector; and determine the first interface with the highest similarity among multiple first interfaces as the first target interface corresponding to the page element.

[0136] Furthermore, based on the second target component template, semantic matching and rule mapping can be used to identify target interface information strongly related to the component function corresponding to the page element from the interface configuration information of the first target interface, and determine the correspondence between the element attribute information of the page element and the placeholders in the second target component template. Then, according to the preset placeholder mapping rules in the component template, the identified target interface information and the element attribute information of the page element are respectively filled into the specified positions of the second target component template. Finally, the filled template is subjected to syntax validation and standardization processing to generate directly runnable component code that corresponds one-to-one with the current page element.

[0137] Therefore, mature code frameworks pre-set for different types of elements can be directly reused, ensuring that the component code conforms to the general specifications of front-end development and avoiding the inefficiency of building the code structure from scratch for each element. When the element attribute information contains logical interface information, the corresponding first target interface can be matched from the pre-set first interface and the interface configuration information can be extracted. This allows for deep binding of the component's visual attributes with the business logic interface, so that the generated component code not only has a visual style but also has directly callable business logic capabilities, eliminating the need for subsequent manual development of interface adaptation code and improving the functional completeness of the component. At the same time, information is populated into the second target component template based on the interface configuration information and element attribute information, which can map the element's visual attributes, functional attributes, and other information to the corresponding positions in the template. This allows the generated component code to accurately reproduce the design intent described in natural language, improving the efficiency, accuracy, and completeness of component code generation.

[0138] Step 304: Generate layout code based on page layout information and element attribute information.

[0139] Optionally, in some embodiments, a target layout rule is determined from a plurality of preset layout rules based on page layout information and element type; the UI components corresponding to the page elements are laid out according to the target layout rule to obtain the layout parameters of the target interface; and layout code is generated according to the layout parameters of the target interface.

[0140] The layout rules can be pre-defined; for example, the layout rules can be stored in JSON format.

[0141] As an example, page features can be extracted from page layout information and element types. A rule-matching method can then be used to obtain a target deployment rule that matches the page features from multiple preset layout rules. Next, based on this target deployment rule, and combined with the component attribute information of the UI components corresponding to the page elements (such as size and scalability) and the constraint information of the target interface, the layout parameters of the UI components corresponding to each page element can be calculated, such as coordinates, width, height, spacing, and alignment. Finally, the layout parameters are filled into the corresponding positions of a predefined layout syntax template, and the filled layout syntax template is subjected to syntax validation and format standardization to generate layout code.

[0142] Therefore, by matching target layout rules from a preset rule library based on page layout information and element type, validated layout logic that conforms to best practices in front-end development can be directly reused, avoiding compatibility issues and visual inconsistencies caused by custom layout logic, and ensuring the basic rationality of the layout. Standardizing the layout of UI components according to the target layout rules and generating layout parameters enables the layout parameter values ​​of different pages and different elements to be consistent, forming a standardized layout parameter system. This not only improves the visual consistency of the interface, but also facilitates batch adjustments and maintenance of the layout in the future.

[0143] Optionally, in other embodiments, a layout prediction model is used to obtain predicted layout parameters based on page layout information and element types; layout code is then generated based on the predicted layout parameters.

[0144] The layout prediction model, which generates layout strategies, can be pre-trained. For example, it can be trained using sample data, allowing the model to learn layout strategies. This sample data could include page layout information and element types from a large number of high-quality management backend pages.

[0145] As an example, page layout information and the element types of each page element in the target interface can be input into the layout prediction model, and the predicted layout parameters can be obtained in response to the model's output. Then, the layout parameters are filled into the corresponding positions of the predefined layout syntax template, and the filled layout syntax template is subjected to syntax validation and format standardization to generate layout code.

[0146] Therefore, by leveraging the predictive capabilities of the layout prediction model, combined with page layout information and element types, predicted layout parameters can be generated, avoiding issues such as spacing confusion, poor adaptability, and non-compliance with user visual habits that occur when manually setting layout parameters. Furthermore, based on the predicted layout parameters output by the model, layout code can be generated automatically and intelligently, transforming abstract layout intentions into structured and standardized layout code without the need for manual adjustment of parameter format or code structure, thus improving the efficiency and accuracy of layout code generation.

[0147] Step 305: Merge the component code and layout code to obtain the interface code of the target interface.

[0148] As an example, component code and layout code can be merged to obtain merged code; then, the merged code can be syntax-checked and formatted, and necessary logic code can be added to generate the interface code of the target interface.

[0149] The UI code generation method of this disclosure, in response to design requirement information including UI description text, employs a Large Language Model (LLM) to perform intent recognition on the UI description text, obtaining semantic intent information. This semantic intent information includes element attribute information of page elements in the target interface and page layout information of the target interface. Based on the element attribute information of the page elements, component code corresponding to the page elements is generated. Based on the page layout information and element attribute information, layout code is generated. The component code and layout code are then fused to obtain the interface code of the target interface. Therefore, by using a large model to perform intent recognition on the UI description text, colloquial and business-oriented text description information can be transformed into structured semantic intent information. Element attribute information and page layout information of page elements are automatically and intelligently extracted, avoiding information omissions or misunderstandings during manual requirement breakdown. Furthermore, generating component code and layout code based on semantic intent information makes the code generation process more targeted. Finally, interface code can be obtained based on the component code and layout code, realizing automatic development of the target interface and end-to-end automated conversion from natural language description to interface code, lowering the threshold for interface development and improving the efficiency and accuracy of interface development.

[0150] like Figure 4 As shown, this disclosure also proposes a method for generating user interface (UI) code. Figure 4This is a schematic diagram according to the fourth embodiment of the present disclosure. The user interface (UI) code generation method may include the following steps: Step 401: Obtain the design requirements information for the target interface to be designed.

[0151] It should be noted that the execution process of step 401 can refer to the execution process of any embodiment of this disclosure, and will not be repeated here.

[0152] Step 402: In response to design requirement information, including user-inputted target component description information for the first component, determine a second target interface adapted to the first component from a plurality of preset second interfaces.

[0153] The first component can be obtained in response to a selection operation performed on multiple components displayed by the low-code platform, and the first component is deployed in the editing canvas of the low-code platform. It should be noted that the low-code platform can maintain a component template corresponding to any one of the multiple components.

[0154] The selection operation can take the form of, but is not limited to, mouse click, drag, pull, and slack.

[0155] The target interface's UI code is generated in response to operations on the low-code platform.

[0156] The target component description information can be used to describe the corresponding first component, and this disclosure does not limit the number of first components.

[0157] As an example, a low-code platform can have a component library panel, which displays multiple components. Users can use drag-and-drop, click to add, or other methods to deploy the selected component (referred to as the first component in this disclosure) to the editing canvas. Then, users can input the corresponding component description information for the component, which is referred to as the target component description information in this disclosure.

[0158] It should be noted that this disclosure does not restrict the input method for the target component description information, and may include, but is not limited to, voice input, keyboard input, mouse input, etc.

[0159] In order to obtain the second target interface, optionally, in some embodiments, information extraction is performed on the target component description information to obtain keywords related to the first component; based on the keywords, a second target interface adapted to the first component is determined from multiple second interfaces.

[0160] For example, the second interface whose interface description information includes keywords can be determined as the second target interface adapted to the first component; or, a seventh semantic vector corresponding to the first component can be generated based on keywords related to the first component, and an eighth semantic vector corresponding to the interface description information of any second interface can be obtained; a similarity algorithm is used to determine the similarity between the seventh semantic vector and each eighth semantic vector, and the second interface corresponding to the eighth semantic vector with the highest similarity is determined as the second target interface adapted to the first component.

[0161] Optionally, in other embodiments, a prediction model can be used to predict the logical interface corresponding to the first component based on the component type of the first component, so as to obtain the second target interface corresponding to the first component.

[0162] The prediction model has the function of predicting the logical interface of the corresponding component. It should be noted that this disclosure does not impose any restrictions on the prediction model.

[0163] For example, the component type of the first component and the interface description information of multiple second interfaces can be input into the prediction model, and in response to the model's output, the second target interface corresponding to the first component can be obtained. For instance, assuming a user drags in a "form" component and a "submit button" component, the AI ​​model (referred to as the prediction model in this disclosure) can automatically infer and suggest binding the button's click event to the form's submit action, and recommend a "create resource" API.

[0164] Therefore, by extracting keywords and removing invalid information from the target component description, we can avoid interface matching deviations caused by the complexity of the target component description. Furthermore, by selecting suitable second target interfaces from multiple second interfaces based on the extracted keywords, we can transform the user's vague business description into a precise basis for interface matching, thereby improving the accuracy and efficiency of obtaining business interfaces that match the component's functions.

[0165] Step 403: Generate the component code corresponding to the first component based on the first component and the second target interface.

[0166] As an example, the component attribute information of the first component can be extracted from the target component description information of the first component, and the component template corresponding to the first component can be populated with information based on the component attribute information and the interface configuration parameters of the second target interface, thereby obtaining the component code corresponding to the first component.

[0167] Step 404: Obtain the layout parameters of the first component and generate layout code based on the layout parameters.

[0168] As an example, in response to a user's click on a target control button in a low-code platform, the system can automatically parse and maintain the component attribute information of the first component to extract initial layout parameters, such as spacing, alignment, and responsive breakpoints. Based on the component type of the first component, it determines the target layout rule that matches the component type from preset layout rules, and lays out the first component according to the target layout rule to correct the initial layout parameters, resulting in corrected layout parameters. Based on the corrected layout parameters, it uses an aesthetic model to adjust the page layout of the target interface to obtain the target layout parameters. Finally, based on the target layout parameters, it generates layout code.

[0169] Among them, target control buttons, such as automatic layout buttons, optimized layout buttons, etc.

[0170] It should be noted that a correspondence between component types and layout rules can be established in advance and saved. Then, after determining the component type, the above correspondence can be queried based on the component type to determine the corresponding target layout rule.

[0171] Among them, the aesthetic model can be a machine learning or deep learning model trained on the visual aesthetics, interactive experience and layout specifications of UI design, which can adjust the layout parameters based on visual effects, space utilization and interactive comfort.

[0172] As an example, after deploying multiple first components in the editing canvas, in response to the user's click on the "AI Layout Optimization" button in the low-code platform, the execution entity of this disclosure automatically parses and maintains the component attribute information of the first components, extracting initial layout parameters, such as default values ​​for spacing, alignment, and responsive breakpoints. Then, based on the component type, such as a form or a button group, it matches the corresponding target layout rules from the preset layout rule library, such as vertical arrangement of forms and baseline alignment of button groups, and performs compliance corrections on the initial layout parameters according to the rules, such as correcting the horizontal spacing of forms to the minimum threshold required by the rules, or changing the left alignment of button groups to the right alignment specified by the rules. Afterwards, the corrected layout parameters are input into the aesthetic model, and the model performs fine-tuning of the overall layout of the target interface from dimensions such as visual balance, space utilization, and interactive experience, such as optimizing the gradient distribution of spacing, adjusting the visual center of alignment, and optimizing the values ​​and switching logic of responsive breakpoints. Finally, based on the target layout parameters adjusted by the aesthetic model, layout code is generated.

[0173] Step 405: Generate the UI code for the target interface based on the component code and layout code.

[0174] As an example, component code and layout code can be merged to obtain merged code; then, the merged code can be syntax-checked and formatted, and necessary logic code can be added to generate the interface code of the target interface.

[0175] The user interface (UI) code generation method of this disclosure, in response to design requirement information including component description information for a first component input by the user, determines a second target interface adapted to the first component from a plurality of preset second interfaces based on the component description information; wherein the first component is obtained in response to performing a selection operation on a plurality of components displayed on a low-code platform, and the first component is deployed in the editing canvas of the low-code platform; component code corresponding to the first component is generated based on the first component and the second target interface; layout parameters of the first component are obtained, and layout code is generated based on the layout parameters; and UI code of the target interface is generated based on the component code and the layout code. Therefore, by matching the component description information input by the user with the second target interface, the user's business intent can be directly transformed into interface adaptation logic. This avoids the tedious operation of manually filtering interfaces one by one in the low-code platform, and solves the problem of component-interface mismatch. It ensures that the component can accurately connect to the required business interface, so that the visually built component has the functions that fit the actual business scenario. The component code is generated based on the selected first component and the matched second target interface. It can reuse the component code framework preset by the low-code platform and incorporate interface logic that fits the business. This makes the generated component code have both standardized structure and personalized business capabilities. At the same time, the layout code is generated by combining the layout parameters automatically extracted after visual deployment and merging them into complete interface code. This lowers the threshold for using the low-code platform to generate interface code and improves the efficiency and effectiveness of generating interface code.

[0176] To clearly illustrate the UI code generation method disclosed herein, the following explanation is provided with examples.

[0177] As an example, the user interface (UI) code generation method of this disclosure is applied to an artificial intelligence coding platform, which is a low-code platform. Figure 5 Here is a platform architecture diagram for this AI coding platform, such as... Figure 5 As shown, this artificial intelligence coding platform has an input layer, an AI processing layer, a component library layer, a generation layer, and an output layer, wherein: I. Input Layer The input layer can receive the following information to provide input data for the subsequent AI processing layer: User requirements (natural language description): Functional requirements described by developers using natural language; Design draft: UI design draft, including visual layout, component styles and other information; Data structures, such as API definitions, request or response data structures of backend interfaces, and interface addresses, provide a basis for generating front-end and back-end interaction logic; II. AI Processing Layer The AI ​​processing layer has the following functions: 1. NLP Understanding: Employing natural language understanding technology to analyze user needs.

[0178] 2. Design Identification: Layout analysis of UI design drafts. 3. Component matching: Matches the most suitable predefined component from the component library; 4. AI Orchestration: Based on requirements and layout information, orchestrate the logical relationships between components, including interaction logic, data flow, page state, etc.

[0179] 5. Code optimization: Perform syntax checks, performance optimizations, and code standardization alignment on the generated code.

[0180] III. Component Library Layer The component library layer includes the UI component library, layout component library, business component library, and logic component library. For example, the UI component library includes basic interactive components, the layout component library includes page layout containers, the business component library encapsulates components with specific business logic, and the logic component library is a logical unit that handles state management and API calls.

[0181] IV. Generation Layer The generation layer can have the following functions: Code generation feature: Generates framework-level logic code, supporting mainstream front-end frameworks such as React, Vue, and Angular.

[0182] Style generation function: Generates style code for the page, supporting CSS, Tailwind, SCSS and other solutions.

[0183] Logic generation function: Generates business logic code such as state management and API calls, and connects front-end components with back-end interfaces.

[0184] V. Output Layer The output layer can be used to output complete front-end pages that can be directly deployed and run, without the need for extensive manual secondary development.

[0185] As an example, a user interface (UI) code generation method may include the following steps: 1. Obtain the generation mode selected by the user.

[0186] The production modes can include image generation mode, drag-and-drop component mode, and prompt description mode.

[0187] 2. Responding to the image generation mode: 2.1 Obtain UI flowcharts, wireframes, or design draft images input by the user.

[0188] It should be noted that in this disclosure, UI flowcharts, wireframes, or design draft images are all referred to as UI design drawings.

[0189] 2.2. Preprocess the images input by the user.

[0190] Preprocessing includes noise reduction, size normalization, and element segmentation.

[0191] Element segmentation can be achieved by using contour detection models, deep learning segmentation networks, etc., to initially separate possible component regions.

[0192] 2.3 Design prompts and input the prompts and the aforementioned images into the visual language model. Responding to the model's output, a structured page analysis result is obtained (referred to as the initial analysis result in this disclosure). For example, the prompts might be:

[0193] 2.4 Component Mapping and Generation Using a keyword matching algorithm or a vector similarity algorithm, the component type description information in the page analysis results is semantically matched with the predefined components in the component library (referred to as the first component template in this disclosure) to obtain the component that matches the component type description information (referred to as the first target component template in this disclosure); the component is instantiated, and the text content parsed by AI is filled into the corresponding position in the component template to obtain the corresponding configuration code.

[0194] 2.5 Layout Inference and Generation Based on the layout structure from the page analysis results, basic layout templates are selected, such as top-bottom layouts, header-side-content layouts, and left-right column layouts. Then, using these selected templates, a fusion of computer vision and deep learning technologies is employed to achieve high-precision layout reconstruction. For example, a convolutional neural network-based object detection model is constructed and trained on labeled datasets to recognize common UI components (such as buttons, input boxes, and navigation bars) and accurately locate the spatial coordinates and category attributes of each component within the user interface. The component bounding box coordinates output by the model are mapped to CSS layout parameters using a coordinate transformation algorithm. For fixed positioning scenarios, absolute or relative positioning is used; for adaptive layout requirements, the parameters are converted into Flexbox or Grid layout configurations, ultimately achieving pixel-level layout reconstruction accuracy from the visual layer to the code layer. Finally, based on the layout parameters, corresponding configuration code is generated.

[0195] 3. Responds to the drag-and-drop component mode in the build mode: 3.1 In response to user drag-and-drop or click-triggered operations on multiple components displayed in the component library panel, the first component selected by the user is deployed to the editing canvas; It's important to note that users are adding "components," not the final form. For example, if a user drags in a "data list," it can initially be displayed as a placeholder.

[0196] 3.2 AI-assisted attribute population and logical connections: In the interaction process, after the user completes the operation of adding the first component to the editing canvas, the system's built-in intelligent mechanism will automatically trigger the AI ​​assistant and activate the intelligent assisted interaction mode. For example, when the user places a "table" component into the designated area by dragging and dropping, the AI ​​assistant will immediately interact with the user in a pop-up window, displaying the question: "What data do you want this table to display?" At this time, the user does not need to use professional code or specific format, but can directly enter natural language to describe the requirements, such as "Display orders from the past week, including three columns: order number, amount, and status".

[0197] In response to the user's input of a target component description for the first component, the AI ​​assistant determines a second target interface that matches the first component from a set of pre-defined second interfaces. For example, after receiving natural language input from the user, the AI ​​assistant uses its semantic understanding and data analysis capabilities to parse the request. After understanding the user's intent, the AI ​​assistant automatically filters multiple second interfaces in a pre-defined API library and recommends the "Get Order List" API, which has the highest match to the user's needs. Simultaneously, the AI ​​assistant automatically generates corresponding data retrieval code and field mapping code based on the data columns specified by the user, and accurately fills these codes into the configuration parameters of the table component to complete the table's data configuration.

[0198] When a user drags a "Form" component and a "Submit Button" component onto the interface, the AI ​​assistant intelligently infers based on the components' attributes and common use cases. It automatically identifies the logical relationship between the two components and suggests binding the "Submit Button's" click event to the "Form's" submit action. Furthermore, based on the form's potential data submission requirements, the AI ​​assistant recommends a suitable "Create Resource" API from its API library, facilitating subsequent user operations.

[0199] Then, based on the first component and the second target interface, the component code corresponding to the first component is generated; 3.4 Layout Optimization: After deploying multiple first components in the editing canvas, in response to the user's click on the "AI Layout Optimization" button in the low-code platform, the execution entity of this disclosure automatically parses and maintains the component attribute information of the first components, extracting initial layout parameters such as spacing, alignment, and default values ​​for responsive breakpoints. Then, based on the component type, such as a form or button group, it matches the corresponding target layout rules from the preset layout rule library, such as vertical arrangement of forms and baseline alignment of button groups, and performs compliance corrections on the initial layout parameters according to the rules, such as correcting the horizontal spacing of forms to the minimum threshold required by the rules, or changing the left alignment of button groups to the right alignment specified by the rules. Afterwards, the corrected layout parameters are input into the aesthetic model, which performs fine-tuning of the overall layout of the target interface from dimensions such as visual balance, space utilization, and interactive experience, such as optimizing the gradient distribution of spacing, adjusting the visual center of alignment, and optimizing the values ​​and switching logic of responsive breakpoints. Finally, based on the target layout parameters adjusted by the aesthetic model, layout code is generated.

[0200] 4. Responding to the generation mode as a prompt description mode: 4.1 Obtain the page description text input by the user (referred to as UI description text in this disclosure) For example, the page description text could be: "Create a dashboard homepage with a data overview card at the top, a sales trend line chart in the middle, and a list of recent transactions at the bottom, which supports filtering by date."

[0201] 4.2 Intent parsing and component planning LLM is used to perform intent recognition on the page text to obtain application blueprint information; { "pageTitle": "Data Dashboard", "sections": [ { "type": "StatsCardRow", "count": 4, "dataSource": "Overview API"}, { "type": "LineChart", "title": "Sales Trend", "dataSource": "Trend API"}, { "type": "DataTable", "title": "Recent Transactions", "dataSource": "Transaction API", "hasFilter": true, "filterField": "date"} ] } 4.3 Component Retrieval and Assembly Based on application blueprint information, a search-enhanced generation technology is used to retrieve and assemble components.

[0202] For example, each section.type in the application blueprint information is compared with the component description information in the component library using vector similarity search to find the most matching component. For instance, if a part of the blueprint needs to display a statistical card with numerical animation, the Antd Card component is matched and paired with numerical animation effects to build StatsCard; if a line chart is to be displayed, the Echarts LineChart component is selected to display it.

[0203] At the same time, based on the intent analysis of the dataSource in the application blueprint information, a comprehensive search of the API library can be conducted to find matching APIs, providing strong support for subsequent data acquisition and interaction.

[0204] 4.4 Adaptive Layout Generation Based on the section order and corresponding type determined in the application blueprint information, a preset layout strategy is applied: For rule-driven layout strategies, the layout is implemented according to established rules. For example, when encountering a card row type section, a Flexbox layout is used to ensure that the cards are arranged neatly and orderly; for a chart type section, a full-width container is used to ensure that the chart can be displayed completely and clearly. Alternatively, a layout strategy based on a machine learning model can be used. This model has been trained on a large number of management backend page layout patterns and can intelligently predict and generate the most suitable layout scheme based on the application blueprint information; then, the page layout is performed according to the layout rules and / or the predicted layout method to obtain layout parameters, and layout code is generated based on the layout parameters.

[0205] 5. Code generation Based on the code generated by combining at least one or more of the above patterns, page layout information, component attribute information, and logical relationship information are extracted. Then, based on this information, corresponding code snippets are generated, the code snippets are integrated to obtain integrated code, and then a series of syntax checks and integrity checks are performed on the integrated code to obtain the interface code of the target interface.

[0206] 6. Code optimization Understandably, the target interface can be visually rearranged a second time. Render the page based on the target interface's code to obtain a preview page; then, use the enhanced design-code hybrid editor to edit and update the target interface: 1. In response to an editing operation performed on the second component in the preview page, update the interface code based on the edited second component.

[0207] For example, users can directly drag and drop AI-generated components on the preview page to adjust their position and size. The editor backend will respond to these drag-and-drop operations, converting them into corresponding layout code in real time. This includes modifying the `order` property in the Flexbox layout to adjust the component arrangement order, modifying the `flex` property to control the component's scaling ratio, or modifying the `grid-area` property in the Grid layout to specify the area where the component is located.

[0208] For example, a user can click on a component on the preview page, and a menu of operation options will pop up, including the "Replace Component" option. After clicking "Replace Component," the system will open the component library interface, where the user can freely browse and select another suitable component, such as replacing the currently used Echarts chart with an Antv G2 chart. After the user completes the component replacement operation, the system will automatically and intelligently handle the attribute mapping of the new component, ensuring that the new component inherits the key attribute settings of the original component, while also adapting it appropriately according to the characteristics of the new component; and simultaneously performing API data adaptation, allowing the new component to seamlessly connect with the original data source, ensuring normal data display and interaction.

[0209] 2. In response to the task instruction to update the preview page, the LLM is used to determine the page update requirement information based on the task instruction; based on the page update requirement information, the target code block to be updated is determined from the interface code, and the target code block is updated.

[0210] For example, when a user interacts with the interface, a simple click on a specific button will trigger a quick response from the system, opening a convenient AI dialogue sidebar. The user can then input natural language commands in this sidebar, such as "Change this filter to multiple selection." Leveraging its powerful contextual understanding capabilities, the AI ​​will identify the current page environment and component states, automatically modifying the corresponding component configurations and updating the relevant logic code to fulfill the user's request.

[0211] 7. Automated Deployment After obtaining the UI code of the target interface to be designed, a complete Vue / React project folder can be obtained. Then, the project can be installed for dependencies, scanned for code quality, built and packaged to generate a Docker image, and deployed to a Kubernetes cluster or cloud server according to the user's preset configuration. Finally, an accessible URL link address is provided to the user.

[0212] In summary, the method disclosed herein has at least the following advantages: This disclosed method for generating user interface (UI) code uses UI business process diagrams as a "requirements specification" that machines can understand, leveraging a powerful AI model with multimodal processing capabilities to act as a "super development assistant." This AI model can deeply analyze various information in the business process diagram, automatically and efficiently completing a series of tasks such as requirements analysis, UI decomposition, component selection, appropriate API calls, complex logic orchestration, reasonable page layout, and complete code writing. In some scenarios, it can even directly complete the deployment and launch phase, greatly reducing manual intervention.

[0213] The user interface (UI) code generation method disclosed herein does not require relevant personnel to possess complex technical knowledge. They only need to draw a standard and compliant business process diagram or provide user interface description text, and the system can automatically generate a fully functional, scientifically and rationally laid out, and directly deployable application based on the input information, thereby improving the efficiency of user interface code generation and thus improving the deployment efficiency of web applications.

[0214] To implement the above embodiments, this disclosure also provides a user interface (UI) code generation apparatus. For example... Figure 6 As shown, Figure 6 This is a schematic diagram according to the fifth embodiment of the present disclosure. The user interface (UI) code generation apparatus 600 may include: a first acquisition module 601, a detection module 602, a second acquisition module 603, and a first generation module 604.

[0215] The first acquisition module 601 is used to acquire the design requirements information of the target interface to be designed.

[0216] The detection module 602 is used to respond to design requirement information, including UI design drawings, by using an element detection model to perform element detection on the UI design drawings and obtain element detection results; wherein, the UI design drawings display the UI components required for the target interface.

[0217] The second acquisition module 603 is used to acquire the first page layout information of the target interface, the first component attribute information of each UI component, and the first logical relationship information between each UI component by using the Visual Language Model (VLM) based on the element detection results and the UI design diagram.

[0218] The first generation module 604 is used to generate the interface code of the target interface based on the first page layout information, the first component attribute information and the first logical relationship information.

[0219] In one possible implementation of this disclosure, the element detection model includes a Region Proposal Network (RPN) and a detection head network; the detection module 602 is configured to: use the RPN to perform a first detection on the UI design drawing to obtain multiple candidate image regions; wherein any candidate image region displays UI components; use the detection head network to perform a second detection on any candidate image region to obtain a corresponding region detection result; wherein the region detection result is used to indicate the atomic visual category to which the corresponding candidate image region belongs; and determine the element detection result based on the region detection result.

[0220] In one possible implementation of this disclosure, the detection module 602 is used to: use a Feature Pyramid Network (FPN) to extract features from the UI design drawing to obtain multi-scale visual features; and use an Resource Pyramid Network (RPN) to perform a first detection on the multi-scale visual features to obtain multiple candidate image regions.

[0221] In one possible implementation of the embodiments of this disclosure, the atomic vision category includes at least one of the following: shape category, text category, interaction category, container category, and data display category.

[0222] In one possible implementation of this disclosure, the region detection result is further used to indicate at least one of the following: the location information of the candidate image region; the confidence level of the candidate image region.

[0223] In one possible implementation of this disclosure, the second acquisition module 603 is configured to: determine a first prompt word based on the element detection result and the UI design drawing; input the first prompt word into the VLM to obtain configuration code; wherein the configuration code is used to describe the page layout of the target interface, the component attributes of each UI component, and the logical relationship between each UI component; and read the first page layout information, the first component attribute information, and the first logical relationship information from the configuration code.

[0224] In one possible implementation of this embodiment, the second acquisition module 603 is configured to: input the first prompt word into the VLM to obtain an initial analysis result; wherein the initial analysis result includes the initial page layout information of the target interface, the initial component attribute information of each UI component, and the initial logical relationship information between each UI component; and generate corresponding configuration code based on the initial page layout information, the initial component attribute information, and the initial logical relationship information, respectively.

[0225] In one possible implementation of this disclosure, the second acquisition module 603 is configured to: determine a first target layout template from a plurality of preset first layout templates based on the initial page layout information; and generate configuration code corresponding to the initial page layout information based on the layout parameters and element detection results of the first target layout template.

[0226] In one possible implementation of this disclosure, the initial component attribute information includes component type description information; the second acquisition module 603 is used to: determine a first target component template from a plurality of predefined first component templates based on the component type description information; wherein any first component template is defined using an instantiable code structure; and fill the code structure of the first target component template with information based on the initial component attribute information to obtain the configuration code corresponding to the initial component attribute information.

[0227] In one possible implementation of this disclosure, the second acquisition module 603 is configured to: acquire a first semantic vector of component type description information and a second semantic vector of preset component description information corresponding to each first component template; for any first component template, determine the similarity between the component type description information and the first component template based on the second semantic vector and the first semantic vector of the preset component description information corresponding to the first component template; and determine a first target component template from multiple first component templates based on the similarity.

[0228] In one possible implementation of this disclosure, the user interface (UI) code generation device 600 may further include: The recognition module is used to respond to design requirement information, including UI description text, and to perform intent recognition on the UI description text using a Large Language Model (LLM) to obtain semantic intent information. The semantic intent information includes element attribute information of page elements in the target interface and page layout information of the target interface.

[0229] The second generation module is used to generate the component code corresponding to the page elements based on the element attribute information of the page elements.

[0230] The third generation module is used to generate layout code based on page layout information and / or element attribute information.

[0231] The first fusion module is used to merge component code and layout code to obtain the interface code of the target interface.

[0232] In one possible implementation of this disclosure, there is at least one page element, and the element attribute information includes the element type; the second generation module is configured to: for any page element, determine a second target component template from a plurality of predefined second component templates according to the element type of the page element; in response to the element attribute information including logical interface information related to the page element, determine a corresponding first target interface from a plurality of preset first interfaces according to the logical interface information; and fill information into the second target component template based on the interface configuration information of the first target interface and the element attribute information of the page element to obtain the component code corresponding to the page element.

[0233] In one possible implementation of this disclosure, the element attribute information includes the element type; the third generation module is configured to: determine a target layout rule from a plurality of preset layout rules based on the page layout information and the element type; lay out the UI components corresponding to the page elements according to the target layout rule to obtain the layout parameters of the target interface; and generate layout code according to the layout parameters of the target interface.

[0234] In one possible implementation of this disclosure, the third generation module is configured to: use a layout prediction model to obtain predicted layout parameters based on page layout information and element types; and generate layout code based on the predicted layout parameters.

[0235] In one possible implementation of this disclosure, the interface code of the target interface is generated in response to operations on a low-code platform. The user interface (UI) code generation device 600 may further include: The first determining module is configured to, in response to the design requirement information including target component description information input by the user for the first component, determine a second target interface adapted to the first component from a plurality of preset second interfaces based on the target component description information; wherein the first component is obtained in response to performing a selection operation on a plurality of components displayed by the low-code platform, and the first component is deployed in the editing canvas of the low-code platform.

[0236] The fourth generation module is used to generate the component code corresponding to the first component based on the first component and the second target interface.

[0237] The fifth generation module is used to obtain the layout parameters of the first component and generate layout code based on the layout parameters.

[0238] The sixth generation module is used to generate the interface code of the target interface based on the component code and layout code.

[0239] In one possible implementation of this disclosure, the first determining module is configured to: extract information from the target component description information to obtain keywords related to the first component; and, based on the keywords, determine a second target interface adapted to the first component from a plurality of second interfaces.

[0240] In one possible implementation of this disclosure, the user interface (UI) code generation device 600 may further include: The rendering module is used to render the page based on the interface code of the target interface to obtain a preview page.

[0241] The seventh generation module is used to generate the web application corresponding to the target interface based on the interface code in response to the confirmation operation of the preview page.

[0242] In one possible implementation of this disclosure, the user interface (UI) code generation device 600 may further include: The first update module is used to respond to the editing operation performed on the second component in the preview page and update the interface code according to the edited second component.

[0243] In one possible implementation of this disclosure, the user interface (UI) code generation device 600 may further include: The second determination module is used to respond to the task instruction to update the preview page, and uses LLM based on the task instruction to determine the page update requirement information; The third module is used to determine the target code block to be updated from the interface code based on the page update requirement information.

[0244] The second update module is used to update the target code block.

[0245] It should be noted that the UI code generation apparatus provided in this embodiment can implement all the method steps implemented in any of the above-described UI code generation method embodiments and can achieve the same technical effect. Therefore, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail here.

[0246] The user interface (UI) code generation apparatus of this disclosure acquires design requirement information of a target interface to be designed; in response to the design requirement information including a UI design drawing, it uses an element detection model to perform element detection on the UI design drawing to obtain element detection results; wherein the UI design drawing displays the UI components required by the target interface; it uses a visual language model (VLM) to acquire first page layout information of the target interface, first component attribute information of each UI component, and first logical relationship information between each UI component based on the element detection results and the UI design drawing; and it generates interface code for the target interface based on the first page layout information, first component attribute information, and first logical relationship information. Therefore, it can automatically parse the component composition, layout rules, and logical relationships in UI design drawings, avoiding the tedious process of manually identifying design drawings and writing interface code, effectively shortening the cycle from design finalization to code implementation, and reducing labor costs. With the powerful understanding and data processing capabilities of AI models, page layout information, component attribute information, and logical relationship information can be quickly extracted, effectively avoiding problems such as human interpretation bias, incorrect component parameters, and errors in logical relationship sorting, improving the high degree of fidelity between the generated interface code and the design drawings, and improving the development quality of the target interface. At the same time, it can realize the automated conversion of UI design requirements information into code, reducing the dependence on the design interpretation ability of front-end developers, and improving the flexibility and feasibility of application development processes.

[0247] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, are all carried out with the user's consent and comply with relevant laws and regulations, and do not violate public order and good morals.

[0248] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0249] Figure 7 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0250] like Figure 7 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0251] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0252] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the user interface (UI) code generation method. For example, in some embodiments, the UI code generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the UI code generation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the UI code generation method by any other suitable means (e.g., by means of firmware).

[0253] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0254] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0255] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0256] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0257] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0258] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0259] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0260] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating user interface (UI) code, wherein, The method includes: Obtain the design requirements information for the target interface to be designed; In response to the design requirement information including the UI design diagram, an element detection model is used to perform element detection on the UI design diagram to obtain the element detection result; wherein, the UI design diagram displays the UI components required for the target interface; Using the Visual Language Model (VLM), based on the element detection results and the UI design diagram, the first page layout information of the target interface, the first component attribute information of each UI component, and the first logical relationship information between each UI component are obtained. Based on the first page layout information, the first component attribute information, and the first logical relationship information, the interface code of the target interface is generated.

2. The method according to claim 1, wherein, The element detection model includes a Region Proposal Network (RPN) and a detection head network; the element detection model is used to perform element detection on the UI design drawing to obtain element detection results, including: The RPN is used to perform a first detection on the UI design drawing to obtain multiple candidate image regions; wherein any one of the candidate image regions displays the UI component; The detection head network is used to perform a second detection on any candidate image region to obtain a corresponding region detection result; wherein, the region detection result is used to indicate the atomic vision category to which the corresponding candidate image region belongs; Based on the region detection results, the element detection results are determined.

3. The method according to claim 2, wherein, The RPN is used to perform a first detection on the UI design drawing to obtain multiple candidate image regions, including: The Feature Pyramid Network (FPN) is used to extract features from the UI design diagram to obtain multi-scale visual features. The RPN is used to perform a first detection on the multi-scale visual features to obtain the multiple candidate image regions.

4. The method according to claim 2, wherein, The atomic vision category includes at least one of the following: Shape category, text category, interaction category, container category, data display category.

5. The method according to claim 2, wherein, The regional detection results are also used to indicate at least one of the following: The location information of the candidate image region; The confidence level of the candidate image region.

6. The method according to claim 1, wherein, The method employs a Visual Language Model (VLM) to obtain, based on the element detection results and the UI design diagram, the first page layout information of the target interface, the first component attribute information of each UI component, and the first logical relationship information between each UI component, including: Based on the element detection results and the UI design diagram, the first prompt word is determined; The first prompt word is input into the VLM to obtain configuration code; wherein, the configuration code is used to describe the page layout of the target interface, the component attributes of each UI component, and the logical relationship between each UI component; Read the first page layout information, the first component attribute information, and the first logical relationship information from the configuration code.

7. The method according to claim 6, wherein, The step of inputting the first prompt word into the VLM to obtain the configuration code includes: The first prompt word is input into the VLM to obtain the initial analysis result; wherein, the initial analysis result includes the initial page layout information of the target interface, the initial component attribute information of each UI component, and the initial logical relationship information between each UI component; Based on the initial page layout information, the initial component attribute information, and the initial logical relationship information, the corresponding configuration code is generated.

8. The method according to claim 7, wherein, Based on the initial page layout information, the corresponding configuration code is generated, including: Based on the initial page layout information, a first target layout template is determined from a plurality of preset first layout templates; Based on the layout parameters of the first target layout template, the configuration code corresponding to the initial page layout information is generated.

9. The method according to claim 7, wherein, The initial component attribute information includes component type description information; based on the initial component attribute information, the corresponding configuration code is generated, including: Based on the component type description information, a first target component template is determined from a plurality of predefined first component templates; wherein any of the first component templates is defined using an instantiable code structure; Based on the initial component attribute information, the code structure of the first target component template is populated with information to obtain the configuration code corresponding to the initial component attribute information.

10. The method according to claim 8, wherein, The step of determining the first target component template from a plurality of predefined first component templates based on the component type description information includes: Obtain the first semantic vector of the component type description information, and the second semantic vector of the preset component description information corresponding to each of the first component templates; For any of the first component templates, the similarity between the component type description information and the first component template is determined based on the second semantic vector corresponding to the first component template and the first semantic vector. Based on the similarity, the first target component template is determined from the plurality of first component templates.

11. The method according to claim 1, wherein, The method further includes: In response to the design requirement information including UI description text, a Large Language Model (LLM) is used to perform intent recognition on the UI description text to obtain semantic intent information; wherein, the semantic intent information includes element attribute information of page elements in the target interface and page layout information of the target interface; Based on the element attribute information of the page element, generate the component code corresponding to the page element; Based on the page layout information and the element attribute information, generate layout code; The component code and the layout code are merged to obtain the interface code of the target interface.

12. The method according to claim 11, wherein, The page element is at least one, and the element attribute information includes the element type; generating the component code corresponding to the page element based on the element attribute information includes: For any of the page elements, a second target component template is determined from a plurality of predefined second component templates based on the element type of the page element; In response to the element attribute information including logical interface information related to the page element, a corresponding first target interface is determined from a plurality of preset first interfaces based on the logical interface information. Based on the interface configuration information of the first target interface and the element attribute information of the page element, the second target component template is populated with information to obtain the component code corresponding to the page element.

13. The method according to claim 11, wherein, The element attribute information includes the element type; the step of generating layout code based on the page layout information and the element attribute information includes: Based on the page layout information and the element type, a target layout rule is determined from a plurality of preset layout rules; According to the target layout rules, the UI components corresponding to the page elements are laid out to obtain the layout parameters of the target interface; The layout code is generated based on the layout parameters of the target interface.

14. The method according to claim 11, wherein, The step of generating layout code based on the page layout information and the element attribute information includes: A layout prediction model is used to obtain predicted layout parameters based on the page layout information and the element type; The layout code is generated based on the predicted layout parameters.

15. The method according to any one of claims 1-14, wherein, The interface code of the target interface is generated in response to operations on a low-code platform, and the method further includes: In response to the design requirement information including the target component description information for the first component input by the user, a second target interface adapted to the first component is determined from a plurality of preset second interfaces based on the target component description information; wherein, the first component is obtained in response to performing a selection operation on a plurality of components displayed by the low-code platform, and the first component is deployed in the editing canvas of the low-code platform; Based on the first component and the second target interface, generate the component code corresponding to the first component; Obtain the layout parameters of the first component, and generate layout code based on the layout parameters; Based on the component code and the layout code, the interface code of the target interface is generated.

16. The method according to claim 15, wherein, The step of determining a second target interface adapted to the first component from a plurality of preset second interfaces based on the target component description information includes: Information extraction is performed on the description information of the target component to obtain keywords related to the first component; Based on the keywords, a second target interface that is compatible with the first component is determined from the plurality of second interfaces.

17. The method according to any one of claims 1-14, wherein, The method further includes: The page is rendered based on the interface code of the target interface to obtain a preview page; In response to the confirmation operation of the preview page, a web application corresponding to the target interface is generated and deployed based on the interface code.

18. The method according to claim 17, wherein, The method further includes: In response to an editing operation performed on the second component in the preview page, the interface code is updated according to the edited second component.

19. The method of claim 17, wherein, The method further includes: In response to the task instruction to update the preview page, LLM is used to determine the page update requirement information based on the task instruction; Based on the page update requirement information, the target code block to be updated is determined from the interface code, and the target code block is updated.

20. A user interface (UI) code generation device, wherein, The device includes: The first acquisition module is used to acquire the design requirements information of the target interface to be designed. The detection module is used to respond to the design requirement information, including the UI design drawing, by using an element detection model to perform element detection on the UI design drawing and obtain element detection results; wherein, the UI design drawing displays the UI components required by the target interface; The second acquisition module is used to acquire, using the Visual Language Model (VLM), the first page layout information of the target interface, the first component attribute information of each UI component, and the first logical relationship information between each UI component, based on the element detection results and the UI design diagram. The first generation module is used to generate the interface code of the target interface based on the first page layout information, the first component attribute information and the first logical relationship information.

21. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 19.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 19.

23. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 19.

Images