Front-end code generation method and device, storage medium and electronic equipment
By extracting node encoding metadata and design images from UI design files, and combining machine learning and heuristic rules, the system automatically identifies and updates node component data to generate DSL structure files. This solves the problem of inconsistent understanding between designers and developers and achieves efficient and accurate front-end code generation.
Patent Information
- Application Number
- CN202310511875.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-08
- Publication Date
- 2024-11-08
AI Technical Summary
In the current technology, the inconsistent understanding between designers and developers during the front-end code generation process leads to time-consuming and inefficient communication, uneven programming quality, and difficulty in ensuring code consistency and standardization.
By obtaining node encoding metadata and design images from UI design files, determining the node metadata tree based on the node encoding metadata, performing node component identification processing, generating DSL structure files, and finally generating front-end code files in the target encoding language, the system automatically identifies and updates node component data using a component identification method that combines machine learning and heuristic rules.
It achieves more efficient front-end code generation, improves code style accuracy and usability, simplifies the development process, enhances the complementary advantages of design draft images and metadata, and improves the efficiency and usability of code generation.
Smart Images

Figure CN118916028A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a front-end code generation method, device, storage medium and electronic device. Background Art
[0002] A clear, beautiful and unified front-end interface design (UI design) can bring a good user experience. In the actual project process, the development of application software usually requires a multi-person team to complete. First, the UI designer uses professional design software (such as Figma) to edit the UI design draft. Then, the project developer implements the project engineering (such as HTML 5 code conversion, Jetpack Compose code conversion, etc.) based on the UI designer's UI design draft, so as to realize the conversion into machine-executable front-end code. Summary of the invention
[0003] The present application provides a front-end code generation method, device, storage medium and electronic device, and the technical solution is as follows:
[0004] In a first aspect, an embodiment of the present application provides a front-end code generation method, the method comprising:
[0005] Acquire node encoding metadata and a design draft image from the UI design draft file, and determine a node metadata tree based on the node encoding metadata;
[0006] Perform node component identification processing based on the node encoding metadata and the design draft image to obtain at least one target node component;
[0007] Based on the at least one target node component, the node metadata tree is updated with node component data to generate a DSL structure file;
[0008] A front-end code file in a target coding language is generated based on the DSL structure file.
[0009] In a second aspect, an embodiment of the present application provides a front-end code generation device, the device comprising:
[0010] A data processing module, used to obtain node coding metadata and a design draft image from a UI design draft file, and determine a node metadata tree based on the node coding metadata;
[0011] A component identification module, used to perform node component identification processing based on the node encoding metadata and the design draft image to obtain at least one target node component;
[0012] A structure generation module, used to update the node component data of the node metadata tree based on the at least one target node component, and generate a DSL structure file;
[0013] A code generation module is used to generate a front-end code file in a target coding language based on the DSL structure file.
[0014] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.
[0015] In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.
[0016] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:
[0017] In one or more embodiments of the present application, by obtaining node coding metadata and design draft images from a UI design draft file, node component identification processing is performed based on the node coding metadata and design draft images to obtain at least one target node component, and the node component data is updated on the node metadata tree based on the at least one target node component to generate a DSL structure file and a front-end code file of the target coding language; more fully utilizing the metadata and design draft images in the UI design draft file to automatically identify node components, and update the DSL structure file through the metadata tree; based on the DSL structure file, front-end code with accurate style and high usability can be generated, thereby overcoming the limitations of front-end code generation in related technologies, simplifying the front-end development process, and greatly improving the efficiency of the front-end development process, as well as fully combining the design draft images and metadata to complement each other's advantages, thereby improving the usability of generating front-end code. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 It is a flowchart of a front-end code generation method provided by an embodiment of the present application;
[0020] Figure 2 This is a schematic diagram of an operation of a UI design draft file provided in an embodiment of the present application;
[0021] Figure 3 It is a schematic diagram of a front-end code conversion provided by an embodiment of the present application;
[0022] Figure 4 It is a schematic diagram of a node metadata tree determination process involved in a front-end code generation method provided in an embodiment of the present application;
[0023] Figure 5 It is an interpretation diagram of data preprocessing provided by an embodiment of the present application;
[0024] Figure 6 This is a schematic diagram of a data preprocessing scenario provided by an embodiment of the present application;
[0025] Figure 7 It is a schematic diagram of the component identification process involved in the front-end code generation method provided in an embodiment of the present application;
[0026] Figure 8 It is a schematic diagram of a DSL structure file generation process involved in the front-end code generation method provided in an embodiment of the present application;
[0027] Fig. 9 It is an interpretation diagram of an intelligent recognition scene provided by an embodiment of the present application;
[0028] Fig.10 is a schematic diagram of component properties provided by an embodiment of the present application;
[0029] Fig.11 It is an interpretation diagram of a cross-level attribute extraction method provided in an embodiment of the present application;
[0030] Fig.12 is a schematic diagram of a front-end code generation device provided in an embodiment of the present application;
[0031] Fig.13 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;
[0032] Fig.14 It is a schematic diagram of the structure of the operating system and user space provided in the embodiment of the present application;
[0033] Fig.15 yes Fig.14 The architecture diagram of the Android operating system;
[0034] Fig.16 yes Fig.14 Architecture diagram of the IOS operating system. DETAILED DESCRIPTION
[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0036] In the description of the present application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood in specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are an "or" relationship.
[0037] In the related art, many problems may occur in the front-end code generation process involved in the background technology part. First, the developers and designers have inconsistent understandings of the UI design draft, and there are deviations in the actual development. Therefore, a lot of time is needed for communication and docking between designers and developers to clarify the specific requirements and possible ambiguities in the design drawings; secondly, the programming capabilities of developers are different, the quality of coding varies, and uniformity is difficult to guarantee, and the specifications are inconsistent. Therefore, developers also need to communicate with each other to coordinate the implementation methods and logic of the code to ensure that unified code specifications are followed. These processes are time-consuming and inefficient, and are often accompanied by errors. It can be seen that there are certain limitations in the front-end code generation in the related art.
[0038] The present application is described in detail below with reference to specific embodiments.
[0039] In one embodiment, Figure 1As shown, a front-end code generation method is proposed, which can be implemented by a computer program and can be run on a front-end code generation device based on the von Neumann system. The computer program (such as a conversion plug-in, a conversion program) can be integrated in an application or run as an independent tool application. The front-end code generation device can be a terminal device or a service platform, including but not limited to: a personal computer, a tablet computer, a handheld device, a vehicle-mounted device, a wearable device, a computing device or other processing device connected to a wireless modem. In different networks, terminal devices can be called different names, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device, cellular phone, cordless phone, 5G network or terminal device in future evolution network, etc.
[0040] Specifically, the front-end code generation method includes:
[0041] S102: Acquire node coding metadata and a design draft image from the UI design draft file, and determine a node metadata tree based on the node coding metadata;
[0042] The UI design draft file may be a draft of the front-end UI edited by the developer using a UI design tool. For example, the UI design tool may be Figma design software, and the UI design draft file designed by Figma design software may be referred to as a Figma design draft file;
[0043] In principle, node encoding metadata can represent the hierarchical relationship, node attributes and other detailed information of the UI design draft, which can help achieve more accurate UI style and position reproduction in subsequent processing.
[0044] like Figure 2 As shown, Figure 2 This is a UI design file operation diagram involved in this manual. Figure 2 In the UI design file, after the user finishes editing the UI design file, the node encoding metadata figma-meta and the design image Image in the UI design file can be obtained;
[0045] Optionally, the design draft image can be an interface screenshot for the front-end interface in the UI design draft file. The design draft image can provide better semantic understanding of UI components and component function classification (button, switch, etc.) in subsequent processing.
[0046] Furthermore, a node metadata tree can be preliminarily determined based on the node hierarchical relationship in the node encoding metadata;
[0047] The node metadata tree is the node data structure of this UI design draft that is preliminarily determined based on the node encoding metadata.
[0048] Optionally, in some embodiments, data preprocessing may be performed on the node metadata tree corresponding to the node encoding metadata to achieve a streamlined data structure tree and reduce unnecessary nodes and layer structures as much as possible;
[0049] Optionally, after determining the node metadata tree based on the node coding metadata, the subsequent step of performing node component identification processing based on the node coding metadata and the design draft image to obtain at least one target node component may also be directly executed.
[0050] S104: Perform node component identification processing based on the node encoding metadata and the design draft image to obtain at least one target node component;
[0051] Schematically, the node component identification process can also be referred to as intelligent component identification based on node encoding metadata and design draft images;
[0052] Node component identification and processing:
[0053] In this specification, it is considered that when developers implement UI design draft files, they not only need to ensure the accuracy of the style (such as components and elements), but also need to use the correct components to support subsequent interface interaction events. In this specification, node component identification can be performed based on node encoding metadata and design draft images to determine several target node components for the node metadata tree.
[0054] In one or more embodiments of the present specification, in order to identify the component type intended to be used in the UI design draft file, at least one of the following means may be used:
[0055] (1) The component identification method based on heuristic rules is to identify node components by taking node encoding metadata as the processing object and obtain one or more target node components.
[0056] (2) Image classification method based on machine learning: The image classification method based on machine learning can be combined with node coding metadata to perform component recognition processing on the local image of the component in the design draft image and complete component category labeling.
[0057] It should be noted that the component identification and processing means mentioned above are only for a better understanding of the inventive concept of this specification, and certainly cannot be used to limit the scope of rights of this application. Therefore, equivalent changes to other means of component identification and processing according to the claims of this application are still within the scope covered by this application.
[0058] Optionally, in some embodiments, after obtaining at least one target node component, label correction may be used to further improve the accuracy of component identification results.
[0059] S106: updating node component data of the node metadata tree based on the at least one target node component to generate a DSL structure file;
[0060] The DSL structure file is also known as the DSL (Domain-Specific Language) intermediate state file. In this specification, DSL is a programming language used in the front-end field. It is designed to solve certain front-end field-specific problems. Compared with general programming languages, DSL is more focused on the front-end field, so it is easier to use and understand in the front-end field. DSL can describe problems in the front-end field through syntax, keywords or tags.
[0061] The DSL structure file can represent the data structure information required to generate the front-end code.
[0062] Illustratively, several target node components determined by node component identification processing are used to label all nodes of the aforementioned preliminary node metadata tree with component category labels to complete the update of node component data in the node metadata tree, and then converted into a DSL structure file represented by DSL, usually converted into a DSL structure tree of a DSL specification. The DSL structure tree completely describes the element structure, style structure, and attribute export structure of the tree, and also includes information on resource dependencies and layer dependencies.
[0063] Further, such as Figure 3 As shown, Figure 3 This is a schematic diagram of a front-end code conversion involved in this specification. Figure 3 After the DSL structure file, i.e., the DSL intermediate state, is generated, the information in the DSL structure tree corresponding to the DSL intermediate state can be translated into any target coding language later, which is called DSL compilation, and then converted and compiled into front-end code files, such as CSS / JS / Html / Compose code files and code static resource files.
[0064] For example, you need to extract the color of the rectangle in Figma and convert it into the background color attribute of the button component. Finally, you can generate target language code based on the DSL structure file, such as Jetpack Compose and HTML 5 code.
[0065] S108: Generate a front-end code file in a target coding language based on the DSL structure file.
[0066] The DSL structure file can be used to describe the complete structure tree of the current page / component designed by the UI design draft file, the referenced styles, and business-related data export rules, interaction rules, etc. After conversion to the DSL structure file, it can be easily converted into a machine-executable front-end code file in the corresponding language environment.
[0067] Schematically, the front-end code output process based on the DSL structure file is as follows:
[0068] The goal of this phase is to generate standardized and usable multi-language codes. The complete DSL structure file provides support for generating code languages under different systems. The target code generation method based on component code templates can be used to automatically convert front-end code files based on DSL structure files;
[0069] Furthermore, various commonly used components are pre-organized and component code templates corresponding to the components are configured. According to the DSL structure file, the data information in the DSL is integrated into the corresponding component code template, and the code file can be generated with one click.
[0070] Furthermore, for the standardization of the code, the execution body can support functions such as code indentation and unified naming format under the programming specifications. In terms of code usability, the execution body supports the loop output of the code and the extraction and disassembly of the function. For some loop body components that may be involved in this specification, the system generates sub-item code templates, extracts sub-item variable data and constructs parameter variable arrays, and supports loop code output at the code level. In addition, function extraction is implemented for page function modules. For example, the device screen is split into a top bar, a content body, and a bottom navigation bar. The code of the corresponding function is extracted and combined into a function, which is called in the main structure function. Function disassembly is implemented for long functions, and each layout is logically decoupled to improve the maintainability and utilization of the code.
[0071] In one or more implementations of this specification, a new front-end code generation process is involved, which makes fuller use of metadata and image data in the design UI design draft file to automatically identify node components, and update the DSL structure file with the metadata tree. Based on the DSL structure file, front-end code with accurate style and high usability can be generated, thereby simplifying the front-end development process and greatly improving the efficiency of the front-end development process; and fully combining the design draft images and metadata to complement each other's advantages, thereby improving the usability of generated front-end code.
[0072] Optional, see Figure 4 , Figure 4 1 is a schematic diagram of the node metadata tree determination process involved in the front-end code generation method proposed in this application. Specifically, the steps of determining the node metadata tree based on the node encoding metadata are performed as follows:
[0073] S202: performing data preprocessing on the node encoding metadata to obtain target node encoding metadata after data preprocessing;
[0074] Indicatively, data preprocessing may be performed on the node metadata tree corresponding to the node encoding metadata to achieve a streamlined data structure tree and reduce unnecessary nodes and layer structures as much as possible;
[0075] Data preprocessing is as follows:
[0076] The node encoding metadata exported from the UI design draft file may contain some attributes and redundant nested structures that are useless for code development, which will interfere with the recognition and generation results. The main goal of the data preprocessing stage is to streamline the data structure tree, reduce unnecessary nodes and layer structures as much as possible, and obtain enough clean information for subsequent recognition and data conversion.
[0077] In one possible implementation, please refer to Figure 5 ,like Figure 5 As shown, Figure 5 It is an interpretation diagram of data preprocessing. The data preprocessing stage can be operated from at least one of two directions: (1) node optimization. Node optimization is carried out from three angles: filtering meaningless node attributes, deleting useless nodes, and merging material nodes; (2) layer optimization. Layer optimization involves layer pruning and layout optimization.
[0078] Furthermore, the data is preprocessed as follows:
[0079] A2: determining at least one reference processing node from the node encoding metadata, performing redundant information processing on the reference processing node, and obtaining first node encoding metadata; and / or determining at least one reference processing layer from the node encoding metadata, performing redundant information processing on the reference processing layer, and obtaining second node encoding metadata;
[0080] The reference processing node is a node to be optimized in the data preprocessing stage, and the reference processing layer is a layer to be optimized in the data preprocessing stage;
[0081] Further, the performing redundant information processing on the reference processing node includes:
[0082] Indicatively, for basic nodes, redundant information is removed through node optimization from three perspectives: filtering meaningless node attributes, deleting useless nodes, and merging material nodes.
[0083] 1. If the reference processing node is a first type node, perform node attribute filtering on the reference processing node;
[0084] The first type of nodes, that is, the nodes that are determined to be of a meaningless type, usually require that the node attributes of the first type of nodes be filtered in the node encoding metadata.
[0085] Schematic, filtering of meaningless node attributes. Node encoding metadata generally carries spatial information, style information, operation information and statistical information.
[0086] Operation information records the constraints of the corresponding designer's behavior, such as the locked layer lock (not allowing the designer to drag the layer). Statistical information records the reference and quantity information of some node relationships, such as component prototype pointing, marking which prototype a certain instance points to. These two types of attribute information cannot help code generation and increase the complexity of the node tree.
[0087] Optionally, operation information and statistical information may be obtained to determine the first type of node, and then the attribute information of the first type of node may be filtered during the deep copy process by encapsulating a new node data type.
[0088] 2. If the reference processing node is a second type node, perform node information deletion processing on the reference processing node;
[0089] Please refer to Figure 6 , Figure 6 This is a schematic diagram of a data preprocessing scenario involved in this specification. Figure 6 In the figure, the left picture is the interface design draft image in the UI design draft file;
[0090] The second type of nodes are nodes that are determined to be useless nodes and need to have their node information deleted.
[0091] Indicatively, the second type of nodes (also known as useless nodes) refer to design elements that actually exist in the design draft but have no practical effect on the final presentation of the design draft. The second type of nodes include some system-provided effects or components ( Figure 6 The part indicated by the label A in the middle), as well as the nodes that are blocked by the upper nodes and are invisible, may be omitted during the design process and retained in the node tree ( Figure 6 The part indicated by the reference numeral B);
[0092] Furthermore, in order to avoid the burden of frequent modifications to the design draft, some nodes are hidden instead of being deleted directly; in addition, in order to ensure the integrity and observability of the design image, the designer may draw some system-provided effects or components. The retention of such nodes will have an adverse effect on the code reproduction results. The existence of useless nodes adds a large amount of invalid code, affecting the readability and usability of the code. The reference processing nodes can be processed for node information deletion. Specifically, the node attributes are detected by checking the attributes and fuzzy matching the reference names, and these second type nodes are deleted, thereby modifying the node metadata tree;
[0093] 3. If the reference processing node is a third type node, then obtaining graphic node material information corresponding to at least one of the third type nodes, and performing material merging processing on the graphic node material information;
[0094] The third type of node can be understood as a node that needs to perform material merging in data preprocessing.
[0095] The material information of each of the graphic nodes is processed by material merging as follows:
[0096] The flexibility of design tools such as Figma allows designers to freely draw assembled icons without using external images. This results in the node metadata tree being scattered with a large number of graphic node combinations, which usually need to be modified to use references to the entire image instead of splicing the graphics.
[0097] The material is merged by matching node types based on node encoding metadata, determining the parent container that only contains graphic nodes, and converting the parent container into a graphic node with image filling (such as Figure 6 The dial button part shown by the middle mark C, the dial button graphic composed of three graphic elements is converted into an overall dial button graphic). Subsequently, by identifying the node filling type, the API interface provided by design draft tools such as Figma is used to export the image material and use it as a static resource in code development.
[0098] In a feasible implementation manner, at least one reference processing layer is determined from the node encoding metadata, and redundant information processing is performed on the reference processing layer;
[0099] Specifically, the node encoding metadata such as Figma container nodes reflects the layer structure of the UI design draft to a certain extent.
[0100] A large number of sample design drafts are obtained in advance, and the common design patterns used by designers in using layer containers are configured. In combination with the code implementation strategies for these design drafts, redundant information processing is performed on the reference processing layers. The redundant information processing is based on layer optimization implementation and may include redundant nested filtering and layer layout recognition.
[0101] When performing redundant information processing on the reference processing layer: if the reference processing layer is a first type layer, then obtaining redundant layer nesting information corresponding to the first type layer, and performing redundant nesting filtering processing on the redundant layer nesting information; if the reference processing layer is a second type layer, then obtaining sub-element arrangement mode information corresponding to the second type layer, and performing layout optimization processing on the element arrangement mode information.
[0102] 1. Redundant nested filtering is to optimize the layout structure of the layers.
[0103] Understandably, due to the inconsistency between the designer's layer arrangement thinking and programming habits, there are a large number of redundant layers in the metadata node tree, such as multiple layers with scattered annotations of a set of style information, some layers do not contain any style information, etc., resulting in multi-layer nesting. This manual reduces the nesting relationship and optimizes the node tree structure by dissolving meaningless container nodes, i.e., container nodes corresponding to the first type of layers, merging multi-level styles, etc.
[0104] 2. Layout optimization processing is the layer layout recognition and optimization of the arrangement mode of sub-elements.
[0105] It is understandable that, except for the group node, all container nodes can implement the arrangement constraints of sub-elements by modifying the layout mode to achieve the effect of relative layout and self-adaptation of the page. Most existing products rely on the specific values of the sub-element positions to achieve absolute layout. In this specification, for the container nodes that lack layout information, that is, the container nodes corresponding to the second type of layer, the spacing of the sub-element positions is calculated, the layout information is re-identified, and relative layout constraints are added to further improve the availability and compatibility of the subsequent generated code.
[0106] A4: Determine the target node encoding metadata based on the first node encoding metadata and / or the second node encoding metadata.
[0107] The first node encoding metadata and / or the second node encoding metadata after the above corresponding data preprocessing can obtain the target node encoding metadata.
[0108] S204: Determine a node metadata tree based on the hierarchical structure relationship of the target node encoding metadata.
[0109] After completing data preprocessing, the target node encoding metadata can be obtained, and the node metadata tree can be determined based on the hierarchical structure relationship of the target node encoding metadata.
[0110] In one or more embodiments of this specification, the metadata of the UI design draft can be optimized based on the above method, and nodes without practical significance and redundant node nesting can be filtered out to achieve redundant optimization of node encoding metadata, and useless attributes and redundant nested structures for some code development can be avoided to avoid interference with subsequent recognition and generation results. After data preprocessing, the node metadata tree is streamlined, and unnecessary nodes and layer structures are reduced as much as possible. Based on the node metadata tree, sufficiently clean information can be obtained for subsequent recognition and data conversion.
[0111] Optionally, in the process of front-end code generation, to implement the UI design draft text, it is necessary not only to ensure the accuracy of the style, but also to use the correct components to support subsequent interactive events, based on which accurate component identification processing is required;
[0112] See also Figure 7 , Figure 7 This is a schematic diagram of the component identification process involved in the front-end code generation method proposed in this application. Specifically:
[0113] The component recognition processing stage combines two recognition methods: (1) a component recognition method based on heuristic rules; and (2) an image classification method based on machine learning.
[0114] S302: performing first node component identification processing on the node encoding metadata using a component heuristic rule to obtain at least one first node component;
[0115] The component heuristic rule is used to perform the first node component identification process on the node encoding metadata, as follows:
[0116] In UI design draft tools such as Figma software, UI components are usually designed from basic elements (rectangles, vector graphics, etc.), and each basic element contains semantic information such as names or descriptions, which are helpful to better identify the semantic type of components. The present invention uses Figma's node meta information and can include at least a variety of heuristic recognition methods, such as component recognition based on node annotation information such as Figma, component feature recognition based on the internal structure of Figma nodes, and composite component recognition based on structure tree hierarchical information.
[0117] 1. Component recognition based on node annotation (semantic) information such as Figma.
[0118] Specifically, the node annotation semantic information of the node encoding metadata may be determined, and the node annotation semantic information may be subjected to component keyword recognition using a component keyword matching method to obtain at least one first node component;
[0119] Node annotation semantic information usually comes from node name and description information; a mapping relationship between "keyword-component class" is pre-built (for example, button class), and a component keyword matching method is used to identify components from node annotation semantic information by using string fuzzy matching.
[0120] The mapping relationship between "keyword-component class":
[0121] A series of commonly used keyword lists for components are pre-configured based on the naming habits of a large number of designers. The keyword list and the mapping relationship between keywords and components are highly scalable, and new components that need to support transcoding can be efficiently added based on actual application situations.
[0122] This method allows users to better control the code implementation of UI components. When you want the components in the design draft to be implemented by a specific type of code component, you can achieve control by setting annotation information such as Figma, which has better controllability.
[0123] 2. Component feature recognition based on the internal structure of nodes such as Figma.
[0124] Specifically, determining component composition features of the node encoding metadata, performing component composition identification on the component composition features based on a reference component composition pattern, and obtaining at least one first node component;
[0125] Schematically, component composition recognition is performed based on component composition features using the internal composition of nodes and their visual feature information (including node shape, inclusion relationship, node combination, etc.).
[0126] Furthermore, based on the user design habits of the UI design draft, a series of common composition patterns of UI components, namely reference component composition patterns, are summarized in advance.
[0127] For example, the switch component needs to meet the following conditions at the same time: a) the style (inside the Figma node) is composed of a rounded rectangle and a circle b) the center of the circle is located inside the rounded rectangle. Based on this, the reference component composition mode of the switch component can be configured.
[0128] Composite component identification based on structure tree hierarchical information.
[0129] For some composite components with cyclic characteristics, such as lists and bottom navigation bars, they often contain multiple items with similar structures. From the perspective of code development, the internal sub-items belong to the same template, and the sub-item list can be generated by cyclic parameter passing to avoid a large number of repeated code segments. Based on this, composite components such as loop body components can be identified by determining the structure tree hierarchy information and comparing the structure tree hierarchy information.
[0130] S304: performing second node component identification processing using an image component classification model based on the node encoding metadata and the design draft image to obtain at least one second node component;
[0131] In a feasible implementation, specification position information may be determined based on the node encoding metadata, the design draft image may be cropped based on the specification position information to obtain a local image of a component element, the local image of the component element may be input into an image component classification model, a component category label may be output, and at least one second node component may be determined from the node encoding metadata based on the component category label.
[0132] The image classification method based on the image component classification model of machine learning is as follows:
[0133] In addition to the component identification method based on heuristic rules, the present invention also introduces an image classification method based on machine learning, and pre-creates and trains an image component classification model based on the machine learning model. The image component classification model has better versatility. Even if the nodes in the UI design draft do not contain annotation information with component semantics, or a special method is used to implement the component style, the component type can be identified by this method.
[0134] Specific:
[0135] (1) Determine the exported image data from the UI design draft, i.e., the design draft image;
[0136] (2) Obtaining the specification position information of all nodes from the design draft metadata (i.e., the node encoding metadata), such as the length and width coordinate information, and using the length and width position information to crop the design draft image to obtain the UI element local image corresponding to the corresponding node, the UI element local image is also the aforementioned component element local image;
[0137] (3) Use the image component classification model to classify the cropped component image to obtain the component category label, and the component category label completes the category labeling of the corresponding node;
[0138] (4) Associating the result of model classification - component category label with the corresponding node in the original design data.
[0139] The purpose of step (2) is to obtain a local image of the component from the design draft image for subsequent model classification. This step can be implemented by at least two methods. The first method is to use image segmentation technology in computer vision to automatically detect the position of the component in the design draft image. The methods that can be used include traditional algorithms (for example, canny edge detection algorithm) and deep learning methods (for example, R-CNN model). The second method is to directly use the length, width and position information of all nodes in the original data of the design draft. The original data information (length, width and position) of the node encoding metadata is used to obtain the image of the UI element from the design draft image.
[0140] In step (3), when classifying the image components of the images of UI elements, the machine learning model can use the classic CNN model ResNet-50 in computer vision for prediction.
[0141] Further, the training process of the image component classification model:
[0142] Model creation: Create an initial image component classification model based on the machine learning model;
[0143] Optionally, the machine learning model may be obtained by adopting AIGC (AI-Generated Content) model technology, and the machine learning model may be implemented by fitting one or more models such as Convolutional Neural Network (CNN) model, Deep Neural Network (DNN) model, Recurrent Neural Networks (RNN) model, embedding model, Gradient Boosting Decision Tree (GBDT) model, and Logistic Regression (LR) model.
[0144] Training data preparation: Collect a large amount of data to build a sample data set. Each sample data in the sample data set carries a standard component category label.
[0145] For example, the open source ReDraw and RICO datasets are used as general datasets. These datasets initially have a large number of component images as sample data, and each component image carries a component category label. At the same time, these sample datasets can also be screened to improve data quality. In addition, based on the open source design drafts of the Figma community, custom datasets for specific design draft styles (for example, Material Design 3) can be made to enrich the sample datasets. Custom datasets are usually more relevant to the user's actual design draft style. In order to improve the learning effect of the model on the custom dataset, the custom dataset is enhanced, including image cropping, adjusting image color, contrast, transparency, etc., to obtain a large amount of sample data in a custom style. At the same time, the three datasets of ReDraw, RICO and custom datasets are merged, and the final sample dataset covers a variety of component types such as buttons and switches. During the model training process, the dataset can be divided into training set, validation set, and test set in a ratio of 8:1:1.
[0146] Model training process: In the model forward training process, each sample data in the sample data set is input into the initial image component classification model for at least one round of model training, and the initial image component classification model identifies the sample component category label; in the model reverse training process, the model loss is calculated based on the sample component category label and the standard component category label, and the model parameters of the initial image component classification model are adjusted based on the model loss until the initial image component classification model completes the model end training conditions, thereby obtaining a trained image component classification model;
[0147] Model application: Deploy the trained image component classification model to actual application scenarios for processing.
[0148] S306: Determine at least one target node component based on the first node component and / or the second node component.
[0149] Illustratively, the first node component and / or the second node component is used as the target node component.
[0150] In one or more embodiments of this specification, based on the above method, multiple methods are introduced to identify the component type in the UI design draft file, including semantic understanding based on node annotation information and model recognition based on exported images, so that more accurate UI component type and style reproduction can be achieved. And the UI component type is automatically identified in combination with image data, and the UI component style is subsequently set using metadata, so that the generated code has better component type judgment and component style reproduction.
[0151] Optional, see Figure 8 , Figure 8This is a schematic diagram of the DSL structure file generation process involved in the front-end code generation method proposed in this application. Specifically:
[0152] S402: Marking the node metadata tree with node labels based on the at least one target node component to generate a first node metadata tree;
[0153] Node label marking is to mark the node component category of the corresponding node in the preliminary node metadata tree based on the determined target node component, and the first node metadata tree is obtained after the marking is completed;
[0154] S404: performing label correction processing on the first node metadata tree to generate a second node metadata tree;
[0155] Label correction processing based on label inspection:
[0156] Rule recognition and machine learning-based image classification label all nodes in the node tree. In actual applications, the recognition process may lack the understanding of the hierarchical context for the specified node information and image, and some recognition errors are difficult to find. For this reason, please refer to Fig. 9 , Fig. 9 It is an interpretation diagram of an intelligent recognition scenario involved in this specification. After the aforementioned component heuristic rule recognition and machine learning-based component recognition processing, a method of hierarchical label type checking is performed through the encoding information tree structure, and the component recognition result is corrected based on the label nesting rule.
[0157] For example, two types of error correction may be supported: (i) tag self-detection processing; and (ii) tag level correction processing.
[0158] Illustratively, performing label correction processing on the first node metadata tree to generate a second node metadata tree includes:
[0159] (i) Tag self-detection processing;
[0160] It can be based on the node type attribute information of the first node metadata tree to perform label self-detection processing,
[0161] Indicatively, the label self-check is based on the node type attribute information node of the first node metadata tree as a reference, and the rationality of the label is judged by comparing the node type and node attribute based on its own node type attribute, so as to determine whether the component recognition result is wrong, and then make corrections;
[0162] Usually the encoding information has clear identification for some node types, such as text. Therefore, once a component is recognized as text, it means that the recognition result is wrong.
[0163] (ii) Label level correction processing
[0164] The second node metadata tree may be generated by performing label level correction processing based on the node level information of the first node metadata tree.
[0165] Schematically, the label level correction process is based on the node level information of the first node metadata tree, and uses the node inclusion relationship combined with the hierarchical structure context to correct the node component label through the hierarchical check method. For example, components can be divided into basic components and composite components according to their complexity. Basic components have a single structure, such as checkbox, switch, image, etc., and do not contain other components inside, while cards may contain components such as buttons inside, and have a multi-layer layer structure, so that correction rules can be configured for correction.
[0166] Exemplarily, for a basic component, if other components appear in its child nodes, the label is considered to be wrong and it is modified to a general node. If the parent node and the child node have the same label, the child node label is retained first. From the perspective of the image, the child node lacks some information compared to the parent node, but the same recognition result is obtained, proving that the image of the child node has a core influence on the recognition result. Therefore, the child node label is retained and the parent node is marked as a general node. For text labels, since the text type node is clearly marked in the metadata encoding information, other node types that are mistakenly recognized as text are directly corrected to general nodes.
[0167] S406: Generate a DSL structure file based on the second node metadata tree.
[0168] Intermediate DSL Conversion
[0169] Based on the configuration of intermediate model conversion, a set of intermediate state DSL is designed by introducing intermediate state representation. The DSL data is completed by cross-level attribute extraction and the structure tree is optimized, and finally a set of scalable data structures is generated to meet the needs of different platform languages.
[0170] Specifically, the component common basic attributes of the node component can be determined based on the second node metadata tree, and the composite component-specific attributes for the composite component can be determined, and the DSL nested structure can be determined based on the component common basic attributes of the node component and the composite component-specific attributes, and a DSL structure tree corresponding to the DSL nested structure can be generated;
[0171] Intermediate structure design
[0172] The present invention extracts common attributes of all components, such as width, height, coordinate position, etc., constructs a parent class of common basic attributes, and configures multiple DSL nested structures according to the component tag data and component characteristics of the second node metadata tree node after identification. In specific implementation, each DSL nested structure exists in the form of a single root, maintains its own unique attributes, and inherits the common parent class. For example, Fig.10 As shown, Fig.10 is a schematic diagram of component properties. Fig.10 The code box in the figure shows a common basic property.
[0173] Based on this, the component common basic properties of the node component and the composite component specific properties of the composite component can be determined from the second node metadata tree based on the configured multiple DSL nested structures;
[0174] Furthermore, a set of composite component-specific properties are pre-configured to build the nested structure of composite components and constrain the internal elements. Fig.10 The figure shows the DSL structure design of a button, where the layout attribute marks the constraints on the child elements, including the child element arrangement direction (layoutMode) and the child element's vertical and horizontal alignment modes (horizontalAlign, verticalAlign). The attribute content is a DSL array, which represents the contained elements, and forms a nested structure to generate a DSL tree.
[0175] Furthermore, after the DSL structure tree is generated, a cross-level attribute extraction method is used to extract the internal element attributes of the node in the second node metadata tree to obtain intermediate attributes, and component attribute update processing is performed on the DSL structure tree based on the intermediate attributes to generate a DSL structure file.
[0176] 1) Intermediate state attribute extraction
[0177] The present invention proposes an intermediate attribute extraction method which is implemented through cross-level attributes, and is used to improve the problem of incomplete component attribute conversion caused by non-standard UI design drafts.
[0178] In some design drafts, some components cannot extract complete style information from the current layer component node. Some attributes such as background color, stroke style, etc. are reflected in the internal sub-elements, and these sub-elements have no other meaning except providing style. Therefore, a cross-level attribute extraction method is used to retrieve the node internal elements of the component nodes on the second node metadata tree, and the node internal element attributes are extracted from the node internal elements to obtain the intermediate state attributes for the component nodes.
[0179] Please refer to Fig.11 , Fig.11It is a cross-level attribute extraction method. Fig.11 Taking a button as an example, the method flow of cross-level attribute extraction is shown. The layer node of the button component itself lacks color filling information, but is reflected by a rectangle inside it. Through cross-level extraction, the background color is obtained from the child elements contained in the component and filled into the button's DSL.
[0180] 2) Intermediate structure tree optimization
[0181] Specifically, component attributes of the DSL structure tree are updated based on the intermediate state attributes, so that a DSL structure file can be generated.
[0182] Furthermore, the DSL structure is crucial to front-end code generation. To further improve code availability and accuracy, after the DSL tree is successfully created and the intermediate data with complete information is obtained, the DSL structure tree can be further optimized. On the one hand, nodes with missing attribute meanings are determined and optimized. For nodes with lost meanings due to attribute extraction, a node table is maintained to save nodes with lost meanings. Such nodes are removed from the DSL structure tree during the optimization process. On the other hand, component nodes of the DSL structure with loop bodies are determined. For component nodes of the DSL structure with loop bodies, they are reordered according to the position information of the internal elements.
[0183] In one or more embodiments of the present specification, for the result of component identification, the above method can be used to check the label of the component, and correct some illogical identification results to further improve the availability of the identification results. In addition, the node encoding metadata is converted into a DSL intermediate state, which can represent the key information such as the component type (model identification result), style, page layout, etc. used for converting the UI design draft into code. The DSL structure tree based on the DSL representation uses the target language to implement the UI design draft, and supports the viewing, copying and downloading of the front-end code file export code, which greatly improves the efficiency of front-end code generation and improves the intelligence of the front-end code generation process.
[0184] The following will be combined Fig.12 , the front-end code generation device provided in the embodiment of the present application is introduced in detail. It should be noted that, Fig.12 The front-end code generation device shown is used to execute the application Figures 1 to 11 For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the present application. Figures 1 to 11 The embodiment shown.
[0185] See also Fig.12, which shows a schematic diagram of the structure of the front-end code generation device of an embodiment of the present application. The front-end code generation device 1 can be implemented as all or part of the device through software, hardware or a combination of both. According to some embodiments, the front-end code generation device 1 includes a data processing module 11, a component identification module 12, a structure generation module 13 and a code generation module 14, which are specifically used to:
[0186] The data processing module 11 is used to obtain node coding metadata and a design draft image from the UI design draft file, and determine a node metadata tree based on the node coding metadata;
[0187] A component identification module 12 is used to perform node component identification processing based on the node encoding metadata and the design draft image to obtain at least one target node component;
[0188] A structure generation module 13, configured to update the node component data of the node metadata tree based on the at least one target node component, and generate a DSL structure file;
[0189] The code generation module 14 is used to generate a front-end code file in a target coding language based on the DSL structure file.
[0190] Optionally, the data processing module 11 is used to:
[0191] Performing data preprocessing on the node encoding metadata to obtain target node encoding metadata after data preprocessing;
[0192] A node metadata tree is determined based on the hierarchical structure relationship of the target node encoding metadata.
[0193] Optionally, the data processing module 11 is used to:
[0194] Determine at least one reference processing node from the node encoding metadata, perform redundant information processing on the reference processing node, and obtain first node encoding metadata; and / or determine at least one reference processing layer from the node encoding metadata, perform redundant information processing on the reference processing layer, and obtain second node encoding metadata;
[0195] The target node encoding metadata is determined based on the first node encoding metadata and / or the second node encoding metadata.
[0196] Optionally, the data processing module 11 is used to:
[0197] If the reference processing node is a first type node, node attribute filtering processing is performed on the reference processing node; if the reference processing node is a second type node, node information deletion processing is performed on the reference processing node; if the reference processing node is a third type node, graphic node material information corresponding to at least one of the third type nodes is obtained, and material merging processing is performed on the graphic node material information;
[0198] The performing redundant information processing on the reference processing layer includes:
[0199] If the reference processing layer is a first type layer, the redundant layer nesting information corresponding to the first type layer is obtained, and redundant nesting filtering processing is performed on the redundant layer nesting information; if the reference processing layer is a second type layer, the sub-element arrangement mode information corresponding to the second type layer is obtained, and layout optimization processing is performed on the element arrangement mode information.
[0200] Optionally, the component identification module 12 is used to:
[0201] Using component heuristic rules to perform first node component identification processing on the node encoding metadata to obtain at least one first node component; and / or,
[0202] Based on the node encoding metadata and the design draft image, an image component classification model is used to perform second node component identification processing to obtain at least one second node component;
[0203] At least one target node component is determined based on the first node component and / or the second node component.
[0204] Optionally, the component identification module 12 is used to:
[0205] Determine the node annotation semantic information of the node encoding metadata, perform component keyword recognition on the node annotation semantic information by using a component keyword matching method to obtain at least one first node component, and / or determine the component composition features of the node encoding metadata, perform component composition recognition on the component composition features based on a reference component composition pattern to obtain at least one first node component;
[0206] Furthermore, specification position information is determined based on the node encoding metadata, the design draft image is cropped based on the specification position information to obtain a local image of a component element, the local image of the component element is input into an image component classification model, a component category label is output, and at least one second node component is determined from the node encoding metadata based on the component category label.
[0207] Optionally, the structure generating module 13 is used to:
[0208] Marking the node metadata tree with node labels based on the at least one target node component to generate a first node metadata tree;
[0209] Performing label correction processing on the first node metadata tree to generate a second node metadata tree;
[0210] A DSL structure file is generated based on the second node metadata tree.
[0211] Optionally, the structure generating module 13 is used to:
[0212] A label self-detection process is performed based on the node type attribute information of the first node metadata tree, and / or a label level correction process is performed based on the node level information of the first node metadata tree to generate a second node metadata tree.
[0213] Optionally, the structure generating module 13 is used to:
[0214] Determine the component common basic attributes of the node component based on the second node metadata tree, determine the composite component-specific attributes for the composite component, determine the DSL nested structure based on the component common basic attributes of the node component and the composite component-specific attributes, and generate a DSL structure tree corresponding to the DSL nested structure;
[0215] The second node metadata tree is subjected to node internal element attribute extraction in a cross-level attribute extraction manner to obtain intermediate attributes, and the DSL structure tree is subjected to component attribute update processing based on the intermediate attributes to generate a DSL structure file.
[0216] It should be noted that the front-end code generation device provided in the above embodiment only uses the division of the above functional modules as an example when executing the front-end code generation method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the front-end code generation device provided in the above embodiment and the front-end code generation method embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.
[0217] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0218] The present application also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figures 1 to 11 The front-end code generation method of the embodiment shown in the figure can be found in the specific execution process. Figures 1 to 11 The specific description of the illustrated embodiment will not be repeated here.
[0219] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figures 1 to 11 The front-end code generation method of the embodiment shown in the figure can be found in the specific execution process. Figures 1 to 11 The specific description of the illustrated embodiment will not be repeated here.
[0220] Please refer to Fig.13 , which shows a block diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. The electronic device in the present application may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected via the bus 150.
[0221] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.
[0222] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 120 includes a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The operating system may be an Android system, including a system deeply developed based on the Android system, an IOS system developed by Apple, including a system deeply developed based on the IOS system or other systems. The data storage area may also store data created by the electronic device during use, such as a phone book, audio and video data, chat record data, etc.
[0223] See also Fig.14 As shown, the memory 120 can be divided into an operating system space and a user space. The operating system runs in the operating system space, and native and third-party applications run in the user space. In order to ensure that different third-party applications can achieve good operating results, the operating system allocates corresponding system resources to different third-party applications. However, different application scenarios in the same third-party application also have different requirements for system resources. For example, in the local resource loading scenario, the third-party application has higher requirements for disk reading speed; in the animation rendering scenario, the third-party application has higher requirements for GPU performance. The operating system and third-party applications are independent of each other, and the operating system often cannot perceive the current application scenario of the third-party application in a timely manner, resulting in the operating system being unable to perform targeted system resource adaptation according to the specific application scenario of the third-party application.
[0224] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to open up data communication between third-party applications and the operating system so that the operating system can obtain the current scenario information of third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.
[0225] Taking the Android operating system as an example, the programs and data stored in the memory 120 are as follows: Fig.15As shown, the memory 120 may store a Linux kernel layer 320, a system runtime library layer 340, an application framework layer 360 and an application layer 380, wherein the Linux kernel layer 320, the system runtime library layer 340 and the application framework layer 360 belong to the operating system space, and the application layer 380 belongs to the user space. The Linux kernel layer 320 provides underlying drivers for various hardware of electronic devices, such as display drivers, audio drivers, camera drivers, Bluetooth drivers, Wi-Fi drivers, power management, etc. The system runtime library layer 340 provides the main feature support for the Android system through some C / C++ libraries. For example, the SQLite library provides database support, the OpenGL / ES library provides 3D drawing support, and the Webkit library provides browser kernel support, etc. The Android runtime library (Android runtime) is also provided in the system runtime library layer 340, which mainly provides some core libraries that allow developers to use the Java language to write Android applications. The application framework layer 360 provides various APIs that may be used when building applications. Developers can also use these APIs to build their own applications, such as activity management, window management, view management, notification management, content provider, package management, call management, resource management, and location management. At least one application runs in the application layer 380. These applications can be native applications that come with the operating system, such as contact applications, text messaging applications, clock applications, camera applications, etc.; they can also be third-party applications developed by third-party developers, such as game applications, instant messaging applications, photo beautification applications, etc.
[0226] Taking the operating system as an IOS system as an example, the programs and data stored in the memory 120 are as follows: Fig.16As shown, the IOS system includes: a core operating system layer 420 (Core OS layer), a core service layer 440 (Core Services layer), a media layer 460 (Media layer), and a touchable layer 480 (Cocoa Touch Layer). The core operating system layer 420 includes the operating system kernel, drivers, and underlying program frameworks, which provide functions closer to the hardware for use by the program framework located in the core service layer 440. The core service layer 440 provides system services and / or program frameworks required by the application, such as the foundation framework, account framework, advertising framework, data storage framework, network connection framework, geographic location framework, motion framework, etc. The media layer 460 provides audio-visual interfaces for the application, such as graphics and image related interfaces, audio technology related interfaces, video technology related interfaces, and wireless playback (AirPlay) interfaces for audio and video transmission technologies. The touchable layer 480 provides various commonly used interface-related frameworks for application development, and the touchable layer 480 is responsible for the user's touch interaction operations on the electronic device. For example, local notification service, remote push service, advertising framework, game tool framework, message user interface (UI) framework, user interface UIKit framework, map framework, etc.
[0227] exist Fig.16 Among the frameworks shown, the frameworks related to most applications include but are not limited to: the basic framework in the core service layer 440 and the UIKit framework in the touchable layer 480. The basic framework provides many basic object classes and data types, provides the most basic system services for all applications, and has nothing to do with UI. The classes provided by the UIKit framework are basic UI class libraries for creating touch-based user interfaces. iOS applications can provide UIs based on the UIKit framework, so it provides the basic architecture of applications for building user interfaces, drawing, processing and user interaction events, responding to gestures, etc.
[0228] Among them, the method and principle of implementing data communication between third-party applications and the operating system in the IOS system can be referred to the Android system, and this application will not go into details here.
[0229] Among them, the input device 130 is used to receive input instructions or data, and the input device 130 includes but is not limited to a keyboard, a mouse, a camera, a microphone or a touch device. The output device 140 is used to output instructions or data, and the output device 140 includes but is not limited to a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 are touch screen displays, which are used to receive touch operations on or near the user using any suitable object such as a finger or a touch pen, and to display the user interface of each application. The touch screen display is usually set on the front panel of the electronic device. The touch screen display can be designed as a full screen, a curved screen or a special-shaped screen. The touch screen display can also be designed as a combination of a full screen and a curved screen, or a combination of a special-shaped screen and a curved screen, which is not limited in the embodiments of the present application.
[0230] In addition, those skilled in the art will appreciate that the structure of the electronic device shown in the above-mentioned figures does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange components differently. For example, the electronic device also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, a wireless fidelity (WiFi) module, a power supply, a Bluetooth module, etc., which will not be described in detail herein. In an embodiment of the present application, the execution subject of each step may be the electronic device described above. Optionally, the execution subject of each step is the operating system of the electronic device. The operating system may be an Android system, an IOS system, or other operating systems, which is not limited in the embodiment of the present application.
[0231] The electronic device of the embodiment of the present application may also be equipped with a display device, which may be any device capable of realizing a display function, such as a cathode ray tube display (CR), a light-emitting diode display (LED), an electronic ink screen, a liquid crystal display (LCD), a plasma display panel (PDP), etc. The user may use the display device on the electronic device 101 to view displayed text, images, videos and other information. The electronic device may be a smart phone, a tablet computer, a gaming device, an AR (Augmented Reality) device, a car, a data storage device, an audio playback device, a video playback device, a notebook, a desktop computing device, a wearable device such as an electronic watch, an electronic glasses, an electronic helmet, an electronic bracelet, an electronic necklace, electronic clothing and other devices.
[0232] exist Fig.13 In the electronic device shown, the electronic device may be a terminal, and the processor 110 may be used to call the application stored in the memory 120, and specifically execute the front-end code generation method involved in one or more embodiments of this specification. A person of ordinary skill in the art may understand that all or part of the processes in the above-mentioned embodiment method may be completed by instructing the relevant hardware through a computer program, and the program may be stored in a computer-readable storage medium, which, when executed, may include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium may be a magnetic disk, an optical disk, a read-only storage memory, or a random access memory, etc. What is disclosed above is only a preferred embodiment of the present application, and of course it cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made in accordance with the claims of the present application are still within the scope covered by the present application.
Claims
1. A front-end code generation method, characterized in that: The method comprises: Acquire node encoding metadata and a design draft image from the UI design draft file, and determine a node metadata tree based on the node encoding metadata; Perform node component identification processing based on the node encoding metadata and the design draft image to obtain at least one target node component; Based on the at least one target node component, the node metadata tree is updated with node component data to generate a DSL structure file; A front-end code file in a target coding language is generated based on the DSL structure file.
2. The method according to claim 1, characterized in that The determining of the node metadata tree based on the node encoding metadata comprises: Performing data preprocessing on the node encoding metadata to obtain target node encoding metadata after data preprocessing; A node metadata tree is determined based on the hierarchical structure relationship of the target node encoding metadata.
3. The method according to claim 2, characterized in that The performing data preprocessing on the node encoding metadata to obtain the target node encoding metadata after data preprocessing includes: Determine at least one reference processing node from the node encoding metadata, perform redundant information processing on the reference processing node, and obtain first node encoding metadata; and / or determine at least one reference processing layer from the node encoding metadata, perform redundant information processing on the reference processing layer, and obtain second node encoding metadata; The target node encoding metadata is determined based on the first node encoding metadata and / or the second node encoding metadata.
4. The method according to claim 3, characterized in that The performing redundant information processing on the reference processing node includes: If the reference processing node is a first type node, node attribute filtering processing is performed on the reference processing node; if the reference processing node is a second type node, node information deletion processing is performed on the reference processing node; if the reference processing node is a third type node, graphic node material information corresponding to at least one of the third type nodes is obtained, and material merging processing is performed on the graphic node material information; The performing redundant information processing on the reference processing layer includes: If the reference processing layer is a first type layer, the redundant layer nesting information corresponding to the first type layer is obtained, and redundant nesting filtering processing is performed on the redundant layer nesting information; if the reference processing layer is a second type layer, the sub-element arrangement mode information corresponding to the second type layer is obtained, and layout optimization processing is performed on the element arrangement mode information.
5. The method according to claim 1, characterized in that The node component identification process is performed based on the node encoding metadata and the design draft image to obtain at least one target node component, including: Using component heuristic rules to perform first node component identification processing on the node encoding metadata to obtain at least one first node component; and / or, Based on the node encoding metadata and the design draft image, an image component classification model is used to perform second node component identification processing to obtain at least one second node component; At least one target node component is determined based on the first node component and / or the second node component.
6. The method according to claim 5, characterized in that The adopting component heuristic rules to perform first node component identification processing on the node encoding metadata to obtain at least one first node component includes: Determine the node annotation semantic information of the node encoding metadata, perform component keyword recognition on the node annotation semantic information by using a component keyword matching method to obtain at least one first node component, and / or determine the component composition features of the node encoding metadata, perform component composition recognition on the component composition features based on a reference component composition pattern to obtain at least one first node component; The step of performing second node component identification processing based on the node encoding metadata and the design draft image using an image component classification model to obtain at least one second node component includes: Determine specification position information based on the node encoding metadata, crop the design draft image based on the specification position information to obtain a component element local image, input the component element local image into an image component classification model, output a component category label, and determine at least one second node component from the node encoding metadata based on the component category label.
7. The method according to claim 1, characterized in that The updating of the node component data of the node metadata tree based on the at least one target node component to generate a DSL structure file includes: Marking the node metadata tree with node labels based on the at least one target node component to generate a first node metadata tree; Performing label correction processing on the first node metadata tree to generate a second node metadata tree; A DSL structure file is generated based on the second node metadata tree.
8. The method according to claim 7, characterized in that The step of performing label correction processing on the first node metadata tree to generate a second node metadata tree includes: A label self-detection process is performed based on the node type attribute information of the first node metadata tree, and / or a label level correction process is performed based on the node level information of the first node metadata tree to generate a second node metadata tree.
9. The method according to claim 7, characterized in that: The generating a DSL structure file based on the second node metadata tree includes: Determine the component common basic attributes of the node component based on the second node metadata tree, determine the composite component-specific attributes for the composite component, determine the DSL nested structure based on the component common basic attributes of the node component and the composite component-specific attributes, and generate a DSL structure tree corresponding to the DSL nested structure; The second node metadata tree is subjected to node internal element attribute extraction in a cross-level attribute extraction manner to obtain intermediate attributes, and the DSL structure tree is subjected to component attribute update processing based on the intermediate attributes to generate a DSL structure file.
10. A front-end code generation device, characterized in that: The device comprises: A data processing module, used to obtain node coding metadata and a design draft image from a UI design draft file, and determine a node metadata tree based on the node coding metadata; A component identification module, used to perform node component identification processing based on the node encoding metadata and the design draft image to obtain at least one target node component; A structure generation module, used to update the node component data of the node metadata tree based on the at least one target node component, and generate a DSL structure file; A code generation module is used to generate a front-end code file in a target coding language based on the DSL structure file.
11. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps according to any one of claims 1 to 9.
12. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as claimed in any one of claims 1 to 9.
Citation Information
Cited By
Front-end page building method and related product
CN119917087A
Front-end page building method, device, electronic device, storage medium and product
CN119917087B
