User interface code generation method and system and medium

Abstract

The invention relates to the field of generative artificial intelligence. The invention provides a user interface code generation method and system and a medium. The user interface code generation method comprises the steps of obtaining user interface screenshot data to extract an atomic component set and a text element set; constructing and solving a weighted bipartite graph based on the atomic component set and the text element set so as to merge the successfully matched atomic components and the text into a composite atomic component, and reserving the unsuccessfully matched atomic components and the text as independent atomic components, the weighted bipartite graph realizes semantic matching of the atomic component and the text according to the geometric distance, the alignment direction and the semantic correlation; performing structural analysis on the atomic component, wherein the structural analysis comprises the steps of generating a strong constraint container by adopting a differential aggregation strategy and generating a pre-mask area based on the strong constraint container; constructing a full-page component tree by using an improved recursive projection segmentation algorithm under the protection mechanism of the pre-mask area; and traversing the full-page component tree by adopting a post-order traversal strategy to generate codes in a divide-and-conquer manner.

Description

Technical Field

[0001] This invention relates to the field of generative artificial intelligence, and more particularly to a user interface code generation method, a user interface code generation system, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of internet technology, the demand for front-end development is increasing day by day. The traditional user interface development process usually requires designers to use design tools to create visual drafts, and then front-end engineers to manually write code to reproduce the design. This process is time-consuming and labor-intensive, and is prone to problems such as low visual fidelity and redundant code structure.

[0003] To address this issue, image-to-code technology emerged. Early solutions used end-to-end neural networks to generate code directly from pixels, but limited by context windows and model capabilities, the generated code often lacked readability and struggled to handle complex nested layouts. Existing solutions utilize Multimodal Large Language Models (MLLMs) to understand UI screenshots and generate code. However, existing MLLM-based image-to-code solutions still have many problems.

[0004] Existing technologies suffer from fragmented atomic component recognition and association errors. The results of object detection models and OCR recognition are often independent, or they employ local greedy strategies (such as nearest neighbor) to associate text and components. However, in complex forms or densely formatted layouts, existing technologies are prone to mismatches between labels and input boxes or semantic loss. Furthermore, existing technologies have limitations in layout analysis. Traditional segmentation algorithms cannot handle overlapping layouts or highly coupled components, often over-segmenting such components, resulting in logically complete components being broken into semantically fragmented structures, making code difficult to maintain and causing the model to lose its global layout context. Conversely, without segmentation, directly inputting the entire webpage image into the LLM (Local Modeling Language) results in image compression, making it impossible for the model to discern subtle details. Moreover, existing technologies typically assume the page is a two-dimensional plane; therefore, they cannot effectively handle page elements with Z-axis hierarchical relationships, such as modal pop-ups and floating buttons.

[0005] In order to overcome the above-mentioned defects of the existing technology, there is an urgent need in the field for a user interface code generation technology that can accurately align multi-source recognition results while automatically generating code, intelligently handle complex nested and overlapping layouts, and take into account both global structure and local details. Summary of the Invention

[0006] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0007] To overcome the aforementioned deficiencies in the existing technology, this invention provides a user interface code generation method, a user interface code generation system, and a computer-readable storage medium, which are based on multimodal bipartite graph matching and divide-and-conquer code generation. This method can automatically generate code while accurately aligning multi-source recognition results, intelligently handling complex nested and overlapping layouts, and taking into account both global structure and local details.

[0008] Specifically, the user interface code generation method provided by the first aspect of the present invention includes: acquiring user interface screenshot data to extract an atomic component set and a text element set, wherein the atomic component set includes atomic components and the text element set includes text; constructing and solving a weighted bipartite graph based on the atomic component set and the text element set, so as to merge the successfully matched atomic components and the text into a composite atomic component, and retain the unmatched atomic components and the text as independent atomic components, wherein the weighted bipartite graph realizes semantic matching between the atomic components and the text based on geometric distance, alignment direction and semantic relevance; performing structural analysis on the composite atomic components and the independent atomic components, including: generating a strongly constrained container using a differentiated aggregation strategy and generating a pre-mask region based on the strongly constrained container, wherein the strongly constrained container is composed of the composite atomic components and the independent atomic components that satisfy preset density constraints and substructure features; constructing a full-page component tree using an improved recursive projection segmentation algorithm under the protection mechanism of the pre-mask region; and traversing the full-page component tree using a post-order traversal strategy to generate code in a divide-and-conquer manner.

[0009] Furthermore, in some embodiments of the present invention, the step of constructing and solving the weighted bipartite graph based on the set of atomic components and the set of text elements further includes: defining a weight function for the edge set of the weighted bipartite graph, wherein the weight function calculates the matching cost between the atomic components and the text based on geometric distance, alignment direction and semantic relevance; and using a global optimal matching algorithm to solve the weighted bipartite graph to determine the set of matching pairs.

[0010] Furthermore, in some embodiments of the present invention, the weighting function The calculation formula is: , in, The vertex corresponding to the set of text elements. The vertex corresponding to the set of atomic components. This is a geometric distance penalty term. For alignment deviation penalty, This is a semantic relevance penalty term. , and The learning weight coefficients are determined by performing a grid search on a small sample labeled dataset, with the optimization objective being to maximize the matching F1 score.

[0011] Furthermore, in some embodiments of the present invention, the step of constructing a full-page component tree using an improved recursive projection segmentation algorithm under the protection mechanism of the pre-mask region includes: initializing the root node of the full-page component tree; traversing each node of the full-page component tree; in response to the current node not being the pre-mask region, calculating the pixel projection histogram of the current node's region on the X and Y axes, wherein the pre-mask region within the current node's region is considered as background white space and does not participate in projection accumulation; finding the widest projection blank interval in the pixel projection histogram as a dividing line, and segmenting the current node based on the dividing line; in response to the current node being the pre-mask region, stopping the segmentation and marking the current node as a leaf node of the full-page component tree; and recursively executing the traversal process until segmentation is no longer possible.

[0012] Furthermore, in some embodiments of the present invention, the step of traversing the full-page component tree using a post-order traversal strategy to generate code in a divide-and-conquer manner includes: traversing the nodes of the full-page component tree using a post-order traversal strategy; in response to the node being a leaf node, inputting the original image slice corresponding to the leaf node into a multimodal large language model to generate atomic code; in response to the node not being a leaf node, constructing a contextual hint containing the layout slice of the current node and the code of each child node of the current node, inferring layout parameters by the multimodal large language model and recursively assembling the sub-component code; and in response to the node not being a leaf node and belonging to a weakly constrained container, the multimodal large language model dynamically inferring and generating code with semantic tags based on the layout position of the node and the content and layout position of the child nodes of the node, wherein the weakly constrained container is composed of the composite atomic components and the independent atomic components that do not satisfy the preset density constraints and substructure features.

[0013] Furthermore, in some embodiments of the present invention, the step of performing structural analysis on the composite atomic component and the independent atomic component further includes: determining the view layering result of the Z-axis based on the overlap rate between the composite atomic component and the independent atomic component and the component characteristics of the composite atomic component and the independent atomic component, wherein the view layering result includes a base layer.

[0014] Furthermore, in some embodiments of the present invention, the step of determining the Z-axis view layering result based on the overlap rate between the composite atomic component and the independent atomic component and the component features of the composite atomic component and the independent atomic component includes: determining the front-to-back occlusion relationship of the composite atomic component and the independent atomic component based on the overlap rate between the composite atomic component and the independent atomic component and the component features of the composite atomic component and the independent atomic component; assigning Z-axis values ​​to the composite atomic component and the independent atomic component according to the front-to-back occlusion relationship of the composite atomic component and the independent atomic component; and implementing view layering according to the Z-axis values ​​of the composite atomic component and the independent atomic component.

[0015] Furthermore, in some embodiments of the present invention, the step of initializing the root node of the full-page component tree includes: in response to the view layering result of the Z-axis including a floating layer and a base layer, dividing the root node of the full-page component tree into a floating layer node and a base layer node; the step of traversing the full-page component tree using a post-order traversal strategy to generate code in a divide-and-conquer manner includes: in response to the traversed node being the root node of the full-page component tree, constructing a layout slice containing the root node of the full-page component tree and context hints for the code of the floating layer node and the code of the base layer node, and placing the code of the floating layer node at the end of the code of the base layer node.

[0016] Furthermore, the user interface code generation system provided according to the second aspect of the present invention includes a memory and a processor. The memory stores computer instructions. The processor is connected to the memory and configured to execute the computer instructions stored in the memory to implement the user interface code generation method provided in any of the above embodiments.

[0017] Furthermore, the computer-readable storage medium provided according to the third aspect of the present invention stores computer instructions. When the computer instructions are executed by a processor, the user interface code generation method provided in any of the above embodiments is implemented. Attached Figure Description

[0018] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0019] Figure 1 A schematic diagram of a user interface code generation system provided according to some embodiments of the present invention is shown; Figure 2 A flowchart of a user interface code generation method provided according to some embodiments of the present invention is shown; Figure 3 A schematic diagram illustrating the association matching of atomic components of a weighted bipartite graph with text according to some embodiments of the present invention is shown. Figure 4 This diagram illustrates a logic schematic of primary container differential aggregation and premask generation according to some embodiments of the present invention; Figure 5 A flowchart of an improved recursive projection segmentation algorithm provided according to some embodiments of the present invention is shown; Figure 6 A schematic diagram of a post-order traversal-driven divide-and-conquer code generation process is shown according to some embodiments of the present invention. Figure 7 A block diagram of the module structure of a user interface code generation system according to a preferred embodiment of the present invention is shown.

[0020] Figure label: 100: User interface code generation system; 110: Memory; 111: Computer-readable storage medium; 120: Processor; S1~S6: Steps. Detailed Implementation

[0021] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention is presented in conjunction with preferred embodiments, this does not mean that the features of the invention are limited to these embodiments. On the contrary, the purpose of describing the invention in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of the present invention. To provide a thorough understanding of the invention, many specific details will be included in the following description. The invention may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of the invention, some specific details will be omitted in the description.

[0022] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0023] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as the orientations shown in the relevant paragraphs and accompanying drawings. These relative terms are for illustrative purposes only and do not imply that the described apparatus must be manufactured or operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0024] It is understood that although terms such as "first," "second," and "third" may be used herein to describe various components, regions, layers, and / or parts, these components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers, and / or parts. Therefore, the first components, regions, layers, and / or parts discussed below may be referred to as second components, regions, layers, and / or parts without departing from some embodiments of the present invention.

[0025] As mentioned above, existing technologies suffer from problems such as fragmented atomic component recognition and association errors. The results of object detection models and OCR recognition are often independent, or they employ local greedy strategies (such as nearest neighbor) to associate text and components. However, in complex forms or densely formatted scenarios, existing technologies are prone to mismatches between labels and input boxes or semantic loss. Furthermore, existing technologies have limitations in layout analysis. Traditional segmentation algorithms cannot handle overlapping layouts or highly coupled components, often over-segmenting such components, resulting in logically complete components being cut into semantically fragmented structures, making code difficult to maintain and causing the model to lose its global layout context. However, without segmentation, directly inputting the entire webpage image into the LLM will cause the model to be unable to see subtle details due to image compression. In addition, existing technologies typically assume the page is a two-dimensional plane; therefore, they cannot effectively handle page elements with Z-axis hierarchical relationships, such as modal pop-ups and floating buttons.

[0026] To overcome the aforementioned deficiencies in the existing technology, this invention provides a user interface code generation method, a user interface code generation system, and a computer-readable storage medium, which are based on multimodal bipartite graph matching and divide-and-conquer code generation. This method can automatically generate code while accurately aligning multi-source recognition results, intelligently handling complex nested and overlapping layouts, and taking into account both global structure and local details.

[0027] In some non-limiting embodiments, the user interface code generation method provided in the first aspect of the present invention can be implemented via the user interface code generation system provided in the second aspect of the present invention.

[0028] Please refer to Figure 1 , Figure 1 A schematic diagram of a user interface code generation system provided according to some embodiments of the present invention is shown.

[0029] like Figure 1 As shown, the user interface code generation system 100 may be configured with a memory 110 and a processor 120. The memory 110 includes, but is not limited to, the computer-readable storage medium 111 described in the third aspect of the present invention, which stores computer instructions thereon. The processor 120 is connected to the memory 110 and is configured to execute the computer instructions stored in the memory 110 to implement the user interface code generation method provided in the first aspect of the present invention.

[0030] The working principle of the aforementioned user interface code generation system will first be described in conjunction with some embodiments of user interface code generation methods. Those skilled in the art will understand that these embodiments of user interface code generation methods are merely non-limiting implementations provided by this invention, intended to clearly demonstrate the main concepts of the invention and provide specific solutions convenient for public implementation, rather than limiting all functions or all working methods of the user interface code generation system. Similarly, this user interface code generation system is also only one non-limiting implementation provided by this invention, and does not constitute a limitation on the executing entity and execution order of the steps in these user interface code generation methods.

[0031] Please refer to Figure 2 , Figure 2 A flowchart of a user interface code generation method provided according to some embodiments of the present invention is shown.

[0032] like Figure 2 As shown, the user interface code generation system first performs step S1: multi-granularity visual element extraction and semantic parsing. The user interface code generation system can acquire user interface screenshot data to extract a set of atomic components and a set of text elements. Here, the set of atomic components includes atomic components, and the set of text elements includes text.

[0033] In some embodiments, the user interface code generation system can acquire user interface screenshot data to be processed. Preferably, the user interface code generation system can preprocess the user interface screenshot data, and the preprocessing may include resolution normalization and noise reduction.

[0034] Then, the user interface code generation system can use a fine-tuned deep learning object detection model (such as YOLOv8) to identify sets of atomic components in the image. The detection categories include, but are not limited to, input fields, buttons, icons, and images, thus forming a collection of atomic components. Meanwhile, the user interface code generation system can utilize optical character recognition (OCR) technology to extract a set of text elements from an image. and get the collection of text elements. bounding box coordinates of each text element and confidence level.

[0035] like Figure 2 As shown, the user interface code generation system can perform step S2: association matching between atomic components and text based on a weighted bipartite graph. The user interface code generation system can construct and solve a weighted bipartite graph based on the set of atomic components and the set of text elements, so as to merge successfully matched atomic components and text into composite atomic components, and retain unmatched atomic components and text as independent atomic components.

[0036] Here, the weighted bipartite graph can achieve semantic matching between atomic components and text based on geometric distance, alignment direction, and semantic relevance.

[0037] In some embodiments, the first vertex set of the weighted bipartite graph corresponds to the set of text elements, the second vertex set corresponds to the set of atomic components, and the edge set is a weight function that calculates the matching cost based on normalized geometric distance, alignment direction, and semantic relevance.

[0038] Please refer to Figure 3 , Figure 3 A schematic diagram illustrating the association matching of atomic components of a weighted bipartite graph with text according to some embodiments of the present invention is shown.

[0039] like Figure 3 As shown, the weighted bipartite graph model for: Among them, vertex set Corresponding text element set Vertex set Corresponding atomic component collection .

[0040] To ensure accurate matching of tags and components, the user interface code generation system can define edge sets. Weight function The weighting function The matching cost between atomic components and text is calculated based on geometric distance, alignment direction, and semantic relevance.

[0041] Thus, the user interface code generation system has implemented the construction of a weighted bipartite graph model that semantically binds atomic components to text.

[0042] Then, the user interface code generation system uses a global optimal matching algorithm to solve the weighted bipartite graph to determine the set of optimal matching pairs. Based on this set of optimal matching pairs It merges successfully matched atomic components with text into a composite atomic component, while unmatched atomic components and text remain as independent atomic components.

[0043] Specifically, the above weighting function The calculation formula can be: , in, The vertex set corresponding to the text element set The vertex in The vertex set corresponding to the atomic component set The vertex in This is a geometric distance penalty term. For alignment deviation penalty, This is a semantic relevance penalty term. , and These are the learning weight coefficients.

[0044] In some embodiments, the geometric distance penalty term The calculation method can be the vertex corresponding to the text element set. Vertices corresponding to the atomic component set Shortest boundary distance divided by the length of the image diagonal Specifically, the vertices corresponding to the set of text elements. Vertices corresponding to the atomic component set The shortest boundary Euclidean distance can be , the distance Divide by the length of the page diagonal The geometric distance penalty value can be obtained by normalization. : , Using this geometric distance penalty value The user interface code generation system can eliminate scale differences caused by images of different resolutions.

[0045] Alignment deviation penalty It can be based on the vertices corresponding to the text element set. The vertex corresponding to the atomic component set position vector Does it fall on the vertex corresponding to the atomic component set? The preset area is determined. Preferably, the user interface code generation system can detect vertices. Relative to vertex position vector When the position vector Landing at the top A lower penalty is applied if the area is aligned horizontally to the left or vertically to the top; otherwise, a higher penalty is applied. This alignment area can be defined as a cone-shaped region at a specific angle.

[0046] Semantic relevance penalty term It can be determined based on the semantic inconsistency of the embedding vector.

[0047] Learning weight coefficients of each feature term , and This can be determined by performing a grid search on a small sample labeled dataset, with the optimization objective being to maximize the matching F1 score. Specifically, the weight coefficients... Instead of fixed values, the algorithm uses a pre-built small sample labeled dataset (containing ground truth relationships) to traverse and combine parameters in the [0,1] interval using the grid search algorithm, and uses the parameter combination with the highest F1 score as the system default parameter to ensure the robustness of the algorithm.

[0048] Preferably, when performing bipartite graph matching, the user interface code generation system can also base its analysis on the bounding box coordinates of the text elements output in step S1. The system uses confidence levels to identify the layout position of text elements and performs local matching based on their layout position. For example, when a text element is entirely within a container, the user interface code generation system performs local matching between the text and atomic components only within that container during bipartite graph matching.

[0049] like Figure 2As shown, the user interface code generation system can perform step S3: deep topology analysis and differentiated primary container aggregation. The user interface code generation system can perform structural analysis on composite atomic components and independent atomic components, including: generating strongly constrained containers using a differentiated aggregation strategy and generating pre-masked regions based on the strongly constrained containers. Here, the strongly constrained container consists of composite atomic components and independent atomic components that satisfy preset density constraints and substructure characteristics.

[0050] Please refer to Figure 4 , Figure 4 A schematic diagram of primary container differential aggregation and premask generation provided according to some embodiments of the present invention is shown.

[0051] like Figure 4 As shown, the user interface code generation system can perform bottom-up aggregation of composite atomic components and independent atomic components, employing a differentiated aggregation strategy that categorizes components based on their strength of constraints during the aggregation process. Then, for strongly constrained containers, the user interface code generation system can construct a pre-mask to prevent over-cutting; for weakly constrained containers, the system can employ delayed semantic determination.

[0052] Strongly constrained containers that satisfy preset density constraints and substructure characteristics typically include containers with strong logical coupling characteristics such as form items, cards, list items, and pop-up windows.

[0053] User interface code generation systems can utilize density-based clustering algorithms (such as the DBSCAN algorithm) to group spatially adjacent data with specific substructures (such as...). Figure 4 The system performs density clustering on atomic components ("image + title + price"), and if a cluster satisfies preset density constraints and substructure characteristics, it defines the cluster as a strongly constrained container, thereby aggregating logically tightly coupled components. Once clustering is successful, the user interface code generation system marks the strongly constrained container region as a "pre-mask" region and generates a mask image. Subsequent segmentation steps prohibit segmentation within this strongly constrained container region. This pre-mask generation step ensures that logically inseparable logically coupled components are not fragmented during subsequent segmentation.

[0054] In some embodiments, due to the specific structure of the composite atomic components and the independent atomic components, the composite atomic components and the independent atomic components are identified by the user interface code generation system as indivisible premask regions.

[0055] Weakly constrained containers that do not satisfy preset density constraints and substructure characteristics typically include containers such as footers, navigation bars, and sidebars. These weakly constrained containers usually have semantic over structural characteristics and diverse layout forms. For these weakly constrained containers, the user interface code generation system does not perform forced geometric aggregation or generate premasks, but instead records the topological relationships of the weakly constrained containers.

[0056] In this way, these weakly constrained containers can be segmented in subsequent segmentation steps (for example, the footer can be segmented into link lines and copyright lines), while the semantic recognition task of the weakly constrained containers is offloaded to the divide-and-conquer code generation step, thereby realizing the deferred semantic binding of weakly constrained containers.

[0057] Preferably, after aggregation and topology analysis based on the above-mentioned differentiated strategies, the user interface code generation system can construct an initial component dependency graph to identify and determine parent and child nodes with containment relationships.

[0058] Complex user interface screenshots often contain overlapping elements. In such cases, the user interface code generation system can perform step S4: overlay context resolution and layer decoupling. The user interface code generation system can determine the Z-axis view layering result based on the overlap rate between composite atomic components and independent atomic components, as well as the component characteristics of composite and independent atomic components.

[0059] In some embodiments, the user interface code generation system can perform depth-order parsing on spatially overlapping components to calculate the inter-component overlap (IoU) and construct an occlusion dependency graph. For overlapping regions that are not inclusive, the user interface code generation system can combine component type priors, for example, by identifying a component as a modal component based on component features, thereby determining the front-to-back occlusion relationship between composite atomic components and independent atomic components. More preferably, the user interface code generation system can determine the front-to-back occlusion relationship of components based on pixel-level edge features, shadow detection, and component type priors.

[0060] Next, based on the occlusion relationship between the composite atomic components and the independent atomic components, the user interface code generation system can assign normalized Z-index values ​​to both components. Then, the system can implement view layering based on these Z-index values. Specifically, it can divide the view into an overlay layer and a base layer (serving as the background document flow), separating page elements into these layers. Page elements in the overlay layer will be assigned absolute or fixed positioning properties during subsequent code generation.

[0061] like Figure 2 As shown, the user interface code generation system can perform step S5: layout segmentation based on a premask mechanism. Under the protection mechanism of the premask region, the user interface code generation system can construct a full-page component tree using an improved recursive projection segmentation algorithm.

[0062] Please refer to Figure 5 , Figure 5 A flowchart of an improved recursive projection segmentation algorithm provided according to some embodiments of the present invention is shown.

[0063] like Figure 5 As shown, the user interface code generation system can construct a full-page component tree using an improved recursive projection segmentation algorithm (XY-CUT++) under the protection mechanism of the pre-masked region.

[0064] The improved recursive projection segmentation algorithm (XY-CUT++) ignores the internal pixel distribution of the generated strongly constrained container premask region when calculating the projection histogram, thus preventing over-segmentation of highly cohesive semantic blocks.

[0065] Specifically, the user interface code generation system can first initialize the root node of the full-page component tree.

[0066] Preferably, when initializing the root node of the full-page component tree, the root node can be segmented based on the Z-axis view hierarchy result. The Z-axis view hierarchy result may include at least one floating layer and a base layer, or, in other embodiments, the Z-axis view hierarchy result may only include the base layer. When the Z-axis view hierarchy result includes at least one floating layer and a base layer, the root node can be divided into a floating layer node and a base layer node when initializing the root node of the full-page component tree.

[0067] Then, the system traverses all nodes of the full-page component tree. During this traversal, the user interface code generation system treats the premask area as a "background white space" or an indivisible solid block in the projection calculation.

[0068] In response to the current node not being a pre-masked region, the pixel projection histograms of the current node's region on the X and Y axes are calculated. Sub-regions within the current node's region marked as pre-masked regions can be considered background white space and do not participate in projection accumulation. Then, the widest projection blank interval in the pixel projection histogram is found as a dividing line, and the current node is segmented based on this dividing line. Preferably, if multiple dividing lines exist, the user interface code generation system prioritizes the dividing line that runs through the entire base layer and does not cut any pre-masked regions, thereby protecting strongly constrained containers representing highly cohesive semantic blocks, such as cards and form groups, from being fragmented.

[0069] If the current node is a premasked region, then stop splitting and mark the current node as a leaf node of the full-page component tree.

[0070] The user interface code generation system recursively executes the above traversal process until it cannot be divided further, generating a full-page component tree containing row, column, and leaf nodes.

[0071] Preferably, the user interface code generation system can optimize the structure of the generated full-page component tree by eliminating redundant hierarchical nesting.

[0072] Then, as Figure 2 As shown, the user interface code generation system can execute step S6: post-order traversal-driven divide-and-conquer code generation. The user interface code generation system can use a post-order traversal strategy to traverse the entire page component tree to generate code in a divide-and-conquer manner.

[0073] Please refer to Figure 6 , Figure 6 A schematic diagram of a post-order traversal-driven divide-and-conquer code generation process is shown, according to some embodiments of the present invention.

[0074] Divide and conquer refers to the approach of using different methods to generate user interface code. In this context, it means that the user interface code generation system adopts different code generation strategies based on different nodes of the full-page component tree.

[0075] Please refer to the reference. Figure 6 The user interface code generation system can traverse the entire page component tree using a post-order traversal strategy, that is, traversing the entire page component tree from bottom to top, first child nodes and then parent nodes. Based on the type of the nodes in the current full-page component tree being traversed, the Multimodal Large Language Model (MLLM) is invoked to generate code using different code generation strategies.

[0076] As can be seen from the aforementioned layout segmentation steps, the leaf nodes of the full-page component tree are indivisible pre-mask regions, i.e., child nodes in the full-page component tree. In some embodiments, when the traversed node is a leaf node, the user interface code generation system can crop the original image slice corresponding to the leaf node and input the original image slice corresponding to the leaf node into the multimodal large language model to generate atomic code (HTML / CSS).

[0077] The divide-and-conquer code generation mechanism includes a context-focused approach. The full-page component tree is formed by continuously dividing the root node. Leaf nodes are contained within their parent nodes. When the traversed node is not a leaf node, the user interface code generation system can construct a layout slice containing the current node and context hints for the code of each child node of the current node. Then, the multimodal large language model infers the layout parameters and recursively assembles the child component code.

[0078] When the traversed node is not a leaf node and belongs to a weakly constrained container, the user interface code generation system can leverage the global contextual understanding capability of the multimodal large language model to dynamically infer and generate code with semantic tags based on the node's layout position and the content and layout position of its child nodes.

[0079] In some embodiments, the user interface code generation system can construct a prompt context that includes the layout slice of the current parent node and "code placeholders" for the child nodes. The multimodal large language model infers Flex / Grid layout parameters based on the layout slice and recursively assembles the child component code.

[0080] Specifically, during the recursive generation process, for any parent node... When constructing an input suggestion, the suggestion can include: parent node A cropped screenshot, used as a visual reference; parent node direct child nodes A list of placeholders; each child node A summary or complete code snippet has been generated; and a reference to a global stylesheet, etc.

[0081] Then, the multimodal large language model populates the layout container of the parent node with the child node code, and according to the parent node... Adjust the Flexbox or Grid layout parameters based on the screenshot; also, if the parent node For regions corresponding to weakly constrained containers (such as footers), the multimodal large language model needs to be based on the parent node. Position on the page and parent node The parent node is inferred from the content (e.g., a row containing the text "Copyright") and its layout position. semantic tags (such as)<footer class="...">This allows for differentiated semantic restoration.

[0082] In some embodiments, the Z-axis view layering result may include at least one floating layer and a base layer. In this case, the root node of the full-page component tree is divided into a floating layer node and a base layer node. When the traversed node is the root node of the full-page component tree, the multimodal large language model can construct contextual hints containing the layout slice of the root node of the full-page component tree, as well as the code of the floating layer node and the code of the base layer node, according to the code generation strategy of the parent node. Based on these contextual hints, the code of the floating layer node is placed at the end of the code of the base layer node.

[0083] In summary, the user interface code generation method based on multimodal bipartite graph matching and divide-and-conquer generation provided by this invention has the following beneficial effects: high association accuracy, which significantly reduces the mismatch rate of text labels and components in dense form scenarios by using the introduced normalized distance and parameter-optimized bipartite graph global algorithm to replace the existing local greedy rules; complete component semantics, which avoids the brute-force cutting of logically tightly bound containers (such as cards) and retains the layout flexibility of weakly bound containers (such as footers) through differentiated aggregation strategies and pre-masking mechanisms; high detail fidelity, which ensures that MLLM can slice each atomic component according to the original resolution when generating it; and support for complex stacked layouts, which introduces a Z-axis parsing step to correctly identify and generate page elements such as modal boxes and floating balls that are out of the background document flow.

[0084] The following are two specific, non-limiting preferred embodiments, which are used to further elaborate on the user interface code generation system proposed in this invention.

[0085] Please refer to Figure 7 , Figure 7 A block diagram of the module structure of a user interface code generation system according to a preferred embodiment of the present invention is shown.

[0086] like Figure 7 As shown, in a preferred embodiment, the user interface code generation system may include a visual perception module, a graph theory association module, a structure analysis module, a layout segmentation module, and a generative coding module.

[0087] After acquiring the user interface screenshot data, the visual perception module can be responsible for performing YOLO object detection and OCR text recognition to output a set of atomic components and a set of text elements.

[0088] Then, the set of atomic components and the set of text elements are input into the graph theory association module, which is responsible for constructing and solving the weighted bipartite graph to resolve the semantic binding between atomic components and text. Here, the graph theory association module can achieve semantic binding between atomic components and text based on geometric distance, alignment direction, and semantic relevance. As a result, the graph theory association module can generate composite atomic components.

[0089] Preferably, the graph theory association module may include a parameter tuning unit based on grid search, which can tune the learning weight coefficients in the weight function that calculates the matching cost based on geometric distance, alignment direction and semantic relevance in the edge set of the weighted bipartite graph.

[0090] The composite atomic components generated by the graph theory association module, along with the remaining independent atomic components that failed to match, are input into the structure analysis module. The structure analysis module is responsible for performing differential container aggregation and defining pre-masked regions, and for separating the page into a floating layer and a base layer serving as the background document flow through Z-index hierarchical parsing.

[0091] The analysis results of the page data by the structure analysis module are input into the layout segmentation module. The layout segmentation module can run the improved recursive projection segmentation algorithm (XY-CUT++) to generate a full-page component tree with pre-mask protection.

[0092] The full-page component tree with pre-mask protection is input into the generative coding module. The generative coding module is responsible for interacting with the multimodal large language model to perform post-order traversal, code assembly, and semantic inference of weakly constrained containers. Based on a post-order traversal strategy, the generative coding module recursively calls the multimodal large language model to generate and assemble the user interface code.

[0093] The user interface code generation system can be deployed on a server with high-performance GPUs, receive UI screenshots uploaded by users through a web interface, and output structured front-end engineering code.

[0094] The preferred embodiment takes a typical "mobile e-commerce product details page with marketing pop-ups" as an example to illustrate in detail the complete generation process from image to code of the user interface code generation method provided by the present invention, so as to further verify the effectiveness of the present invention in handling complex interactive interfaces and stacked layouts.

[0095] In the preferred embodiment two, the UI screenshot to be processed contains two layers of content. The bottom layer (base layer) is a product details page, including a product carousel at the top, a "specification selection area" in the middle (containing multiple rows of labels such as "color" and "size" and corresponding option buttons), and a "You May Also Like" product list at the bottom (double-column waterfall layout). Since the page is displaying a promotional activity when the screenshot is taken, the UI screenshot to be processed also includes a top layer (floating layer). A semi-transparent black overlay and a centered "Limited-Time Coupon" pop-up (Modal) are displayed, with a "Close" icon in the upper right corner of the pop-up.

[0096] The user interface code generation system provided by this invention first detects the entire image and extracts the set of atomic components and the set of text elements.

[0097] Then, a weighted bipartite graph is constructed and solved based on the set of atomic components and the set of text elements to achieve the binding of atomic components to text. In this step, the user interface code generation system can use bipartite graph-resistant matching to achieve complex alignment and local matching.

[0098] For example, in the "Specification Selection Area" at the middle of the UI screenshot, there is the text "Color" and a row of options ("Red", "Blue"). Traditional nearest neighbor algorithms tend to incorrectly associate "Color" with the price number in the previous row. However, in this invention, when constructing the bipartite graph, the user interface code generation system calculates the alignment cost between the text "Color" and the "Red" button, i.e., the alignment deviation penalty. Although the text "color" and the "red" button are geometrically far apart, the alignment deviation penalty is negligible because they are on the same horizontal line (close in Y-axis coordinates). Extremely low, and semantically "color" is highly correlated with color-related words, i.e., a semantic relevance penalty. Low. By combining the weight coefficients optimized through grid search, the global matching algorithm successfully bound the "color" to this group of buttons as a composite atomic component, thereby achieving complex alignment of the "specification selection area".

[0099] For example, the "Claim Now" button on the pop-up window in this UI screenshot is physically very close to the "Add to Cart" button in the background document flow layer (Z-axis projection overlap). However, before constructing the bipartite graph, the user interface code generation system uses the confidence and location information of OCR and object detection to identify that the "Claim Now" text is completely within the bounding box of the pop-up container. Therefore, it only performs local matching of text and atomic components within the pop-up container, avoiding erroneous associations across layers, thereby achieving isolation of the content of each container.

[0100] Subsequently, the user interface code generation system provided by this invention can perform spatial structure analysis on UI screenshots. It generates a pre-masked region formed by strongly constrained containers through a differentiated aggregation strategy, and achieves layer stripping through Z-axis analysis.

[0101] Z-axis analysis can specifically include occlusion detection, depth order inference, and layer decoupling.

[0102] The user interface code generation system can perform occlusion detection based on overlap rate. The system identifies that the bounding box of the coupon pop-up component has a large area of ​​IoU overlap (>50%) with several other components (such as product images and specifications areas), thus determining that the coupon pop-up component has a front-to-back occlusion relationship with several other components.

[0103] The user interface code generation system can combine the component types of the coupon pop-up component with those of other components to determine the occlusion relationship between the coupon pop-up component and other components, thereby achieving depth order inference. The system detects that the pop-up component contains a clearly floating "Close" icon and has a semi-transparent dimming layer around it. Based on this, the system determines that the pop-up component area is a floating layer and assigns a Z-axis value to the atomic components within this floating layer, for example, assigning a Z-index of 1000.

[0104] Before proceeding with the next layout segmentation step, the user interface code generation system can remove the floating layer to achieve layer decoupling. The system logically removes the pop-up area from the input data (Temporarily Remove). This means that subsequent layout analysis of the background product page will not be affected by the pop-up obstruction, ensuring the integrity of the underlying layout structure.

[0105] Next, the user interface code generation system can construct a full-page component tree of the base layer using an improved recursive projection segmentation algorithm based on the protection of the premask.

[0106] For example, when processing the "You May Also Like" double-column product list at the bottom of the UI screenshot, the user interface code generation system has already identified the repetitive structures in the list using density clustering during the aforementioned differential aggregation process, and identified each product card (containing image + title + price) as a strongly constrained container and marked it as a pre-mask region. Therefore, when executing the improved recursive projection segmentation algorithm, this improved recursive projection segmentation algorithm is prohibited from segmenting within the cards. The final generated full-page component tree structure accurately reflects: Grid Container -> Product Card N, rather than a jumbled stack of dividers.

[0107] Then, based on the full-page component tree and the view layering results along the Z-axis, the user interface code generation system can use a post-order traversal strategy to traverse the full-page component tree to generate code in a divide-and-conquer manner, and when processing the root node of the full-page component tree, place the code of the floating layer node at the end of the code of the base layer node.

[0108] Specifically, the multimodal large language model receives pop-up slices of floating layer nodes and generates a piece of code with the style "position: fixed;", thereby realizing the generation of code for floating layer nodes.

[0109] For the base layer node, the multimodal large language model can generate code for the specification selection area sub-node and the product list area sub-node within the base layer node, respectively. Additionally, within this base layer node, there is a copyright notice line at the bottom of the page, which is a weakly constrained container. Although this weakly constrained container is not forcibly aggregated during structural analysis, when the layout is segmented into at least one child node and code generation is performed based on this child node, its location at the very bottom of the page infers and wraps it within the weakly constrained container. <footer>The semantic tags are used to generate basic layer code and perform semantic inference on weakly constrained containers. Therefore, multimodal large language models can achieve both basic layer code generation and semantic inference on weakly constrained containers.

[0110] Then, the contextual hints of the root node of the full-page component tree place the code of the floating layer node at the end of the code of the base layer node, thus perfectly restoring the layering effect of the design draft, and making the interaction logic (such as clicking the close button) clearer.

[0111] In summary, this invention effectively solves the problems of image-text mismatch and semantic fragmentation under complex layouts, and significantly improves the structural integrity and layered interaction fidelity of the generated code.

[0112] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0113] Those skilled in the art will understand that information, signals, and data can be represented using any of a variety of different techniques and skills. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0114] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0115] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0116] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0117] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0118] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.< / footer> < / footer>

Claims

1. A method for generating user interface code, characterized in that, include: Obtain user interface screenshot data to extract an atomic component set and a text element set, wherein the atomic component set includes atomic components and the text element set includes text; A weighted bipartite graph is constructed and solved based on the set of atomic components and the set of text elements to merge the successfully matched atomic components and the text into a composite atomic component, while the unmatched atomic components and the text are retained as independent atomic components. The weighted bipartite graph realizes semantic matching between the atomic components and the text based on geometric distance, alignment direction and semantic relevance. Structural analysis of the composite atomic components and the independent atomic components includes: generating a strongly constrained container using a differentiated aggregation strategy and generating a pre-mask region based on the strongly constrained container, wherein the strongly constrained container is composed of the composite atomic components and the independent atomic components that satisfy preset density constraints and substructure characteristics; Under the protection mechanism of the pre-masked region, an improved recursive projection segmentation algorithm is used to construct a full-page component tree; and The full-page component tree is traversed using a post-order traversal strategy to generate code in a divide-and-conquer manner.

2. The user interface code generation method as described in claim 1, characterized in that, The step of constructing and solving the weighted bipartite graph based on the set of atomic components and the set of text elements further includes: Define a weight function for the edge set of the weighted bipartite graph, wherein the weight function calculates the matching cost between the atomic component and the text based on geometric distance, alignment direction, and semantic relevance; and The weighted bipartite graph is solved using a global optimal matching algorithm to determine the set of matching pairs.

3. The user interface code generation method as described in claim 2, characterized in that, The weighting function The calculation formula is: ， in, The vertex corresponding to the set of text elements. The vertex corresponding to the set of atomic components. This is a geometric distance penalty term. For alignment deviation penalty, This is a semantic relevance penalty term. , and The learning weight coefficients are determined by performing a grid search on a small sample labeled dataset, with the optimization objective being to maximize the matching F1 score.

4. The user interface code generation method as described in claim 1, characterized in that, The steps for constructing the full-page component tree using the improved recursive projection segmentation algorithm under the protection mechanism of the pre-mask region include: Initialize the root node of the full-page component tree; Traverse each node of the full-page component tree; In response to the current node not being the premask region, the pixel projection histograms of the current node's region on the X and Y axes are calculated, wherein the premask region within the current node's region is considered as background white space and does not participate in projection accumulation; Find the widest projection blank area in the pixel projection histogram as the dividing line, and divide the current node based on the dividing line; In response to the current node being the premask region, stop segmentation and mark the current node as a leaf node of the full-page component tree; and The process is recursively executed until it can no longer be divided.

5. The user interface code generation method as described in claim 4, characterized in that, The step of using a post-order traversal strategy to traverse the entire page component tree to generate code in a divide-and-conquer manner includes: The nodes of the full-page component tree are traversed using a post-order traversal strategy; In response to the node being a leaf node, the original image slice corresponding to the leaf node is input into the multimodal large language model to generate atomic code; In response to the node not being a leaf node, a context hint is constructed containing a layout slice of the current node and the code of each child node of the current node. Layout parameters are inferred from the multimodal large language model, and the sub-component code is recursively assembled. In response to the node not being a leaf node and belonging to a weakly constrained container, the multimodal large language model dynamically infers and generates code with semantic tags based on the layout position of the node and the content and layout position of the node's child nodes. The weakly constrained container is composed of the composite atomic components and the independent atomic components that do not satisfy the preset density constraints and substructure features.

6. The user interface code generation method as described in claim 5, characterized in that, The step of performing structural analysis on the composite atomic component and the individual atomic component further includes: The view layering result along the Z-axis is determined based on the overlap rate between the composite atomic components and the independent atomic components, as well as the component characteristics of the composite atomic components and the independent atomic components. The view layering result includes a base layer.

7. The user interface code generation method as described in claim 6, characterized in that, The step of determining the Z-axis view layering result based on the overlap rate between the composite atomic component and the independent atomic component, and the component characteristics of the composite atomic component and the independent atomic component, includes: Based on the overlap rate between the composite atomic component and the independent atomic component, as well as the component characteristics of the composite atomic component and the independent atomic component, the front-to-back occlusion relationship of the composite atomic component and the independent atomic component is determined. Assign Z-axis values ​​to the composite atomic component and the independent atomic component based on their front-to-back occlusion relationship; and View layering is achieved based on the Z-axis values ​​of the composite atomic components and the individual atomic components.

8. The user interface code generation method as described in claim 6, characterized in that, The step of initializing the root node of the full-page component tree includes: in response to the view layering result of the Z-axis including a floating layer and a base layer, dividing the root node of the full-page component tree into a floating layer node and a base layer node; The steps of using a post-order traversal strategy to traverse the full-page component tree to generate code in a divide-and-conquer manner include: in response to the traversed node being the root node of the full-page component tree, constructing a context hint containing the layout slice of the root node of the full-page component tree, as well as the code of the floating layer node and the code of the base layer node, and placing the code of the floating layer node at the end of the code of the base layer node.

9. A user interface code generation system, characterized in that, include: Memory, on which computer instructions are stored; as well as A processor, connected to the memory, and configured to execute computer instructions stored on the memory to implement the user interface code generation method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, the user interface code generation method as described in any one of claims 1 to 8 is implemented.

Images